You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2014/05/08 12:21:55 UTC
[JENKINS] Lucene-Solr-4.8-Linux (64bit/jdk1.8.0_05) - Build # 129 -
Still Failing!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.8-Linux/129/
Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseParallelGC
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.DistribCursorPagingTest
Error Message:
ERROR: SolrZkClient opens=12 closes=11
Stack Trace:
java.lang.AssertionError: ERROR: SolrZkClient opens=12 closes=11
at __randomizedtesting.SeedInfo.seed([CA2A8A102C0F8CB3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:434)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:790)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:360)
at java.lang.Thread.run(Thread.java:745)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.DistribCursorPagingTest
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.DistribCursorPagingTest: 1) Thread[id=3037, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-SendThread(127.0.0.1:46594), state=TIMED_WAITING, group=TGRP-DistribCursorPagingTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) 2) Thread[id=3038, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-EventThread, state=WAITING, group=TGRP-DistribCursorPagingTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DistribCursorPagingTest:
1) Thread[id=3037, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-SendThread(127.0.0.1:46594), state=TIMED_WAITING, group=TGRP-DistribCursorPagingTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
2) Thread[id=3038, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-EventThread, state=WAITING, group=TGRP-DistribCursorPagingTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
at __randomizedtesting.SeedInfo.seed([CA2A8A102C0F8CB3]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.DistribCursorPagingTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=3037, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-SendThread(127.0.0.1:46594), state=TIMED_WAITING, group=TGRP-DistribCursorPagingTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=3037, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-SendThread(127.0.0.1:46594), state=TIMED_WAITING, group=TGRP-DistribCursorPagingTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
at __randomizedtesting.SeedInfo.seed([CA2A8A102C0F8CB3]:0)
Build Log:
[...truncated 11079 lines...]
[junit4] Suite: org.apache.solr.cloud.DistribCursorPagingTest
[junit4] 2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/init-core-data-001
[junit4] 2> 676266 T2958 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
[junit4] 2> 676266 T2958 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /q_u/yf
[junit4] 2> 676269 T2958 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4] 2> 676269 T2958 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2> 676269 T2959 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4] 2> 676369 T2958 oasc.ZkTestServer.run start zk server on port:46594
[junit4] 2> 676370 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 676600 T2965 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@77dc7e82 name:ZooKeeperConnection Watcher:127.0.0.1:46594 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 676600 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 676601 T2958 oascc.SolrZkClient.makePath makePath: /solr
[junit4] 2> 676604 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 676605 T2967 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@480931b4 name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 676606 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 676606 T2958 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4] 2> 676607 T2958 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4] 2> 676609 T2958 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4] 2> 676610 T2958 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4] 2> 676611 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-deeppaging.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 676611 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4] 2> 676613 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/schema-sorts.xml to /configs/conf1/schema.xml
[junit4] 2> 676614 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4] 2> 676615 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 676616 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 676620 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 676621 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4] 2> 676623 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 676623 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4] 2> 676625 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 676626 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4] 2> 676627 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 676628 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
[junit4] 2> 676631 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 676631 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4] 2> 676633 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 676634 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 676636 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 676636 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4] 2> 676638 T2958 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 676639 T2958 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4] 2> 676641 T2960 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145db56f2860001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 676916 T2958 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 676921 T2958 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35958
[junit4] 2> 676922 T2958 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 676922 T2958 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 676922 T2958 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002
[junit4] 2> 676922 T2958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/'
[junit4] 2> 676933 T2958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/solr.xml
[junit4] 2> 676940 T2958 oasc.CoreContainer.<init> New CoreContainer 1370555605
[junit4] 2> 676940 T2958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/]
[junit4] 2> 676940 T2958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 676941 T2958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 676941 T2958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 676941 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 676941 T2958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 676941 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 676942 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 676942 T2958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 676942 T2958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 676942 T2958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 676943 T2958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 676943 T2958 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 676943 T2958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46594/solr
[junit4] 2> 676943 T2958 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 676944 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 676946 T2978 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33b4e04a name:ZooKeeperConnection Watcher:127.0.0.1:46594 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 676946 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 676948 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 676949 T2980 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@46c0a83b name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 676949 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 676950 T2958 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4] 2> 676952 T2958 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4] 2> 676954 T2958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
[junit4] 2> 676958 T2958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
[junit4] 2> 676964 T2958 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
[junit4] 2> 676968 T2958 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4] 2> 676970 T2958 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35958_q_u%2Fyf
[junit4] 2> 676971 T2958 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35958_q_u%2Fyf
[junit4] 2> 676972 T2958 oascc.SolrZkClient.makePath makePath: /overseer_elect
[junit4] 2> 676976 T2958 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4] 2> 676979 T2958 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35958_q_u%2Fyf
[junit4] 2> 676980 T2958 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4] 2> 676981 T2958 oasc.Overseer.start Overseer (id=91720533913501699-127.0.0.1:35958_q_u%2Fyf-n_0000000000) starting
[junit4] 2> 676983 T2958 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4] 2> 676988 T2958 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4] 2> 676988 T2982 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4] 2> 676989 T2982 oasc.OverseerCollectionProcessor.prioritizeOverseerNodes prioritizing overseer nodes at 127.0.0.1:35958_q_u%2Fyf
[junit4] 2> 676991 T2958 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4] 2> 676992 T2958 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 676996 T2981 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4] 2> 676998 T2983 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
[junit4] 2> 676998 T2983 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 676999 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 677000 T2983 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 677001 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 677001 T2981 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:35958/q_u/yf",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35958_q_u%2Fyf",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"1",
[junit4] 2> "core_node_name":null}
[junit4] 2> 677002 T2981 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
[junit4] 2> 677002 T2981 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 677003 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 677004 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 678000 T2983 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 678000 T2983 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4] 2> 678001 T2983 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 678001 T2983 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 678002 T2983 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 678002 T2983 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/collection1/'
[junit4] 2> 678003 T2983 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/collection1/lib/.svn/' to classloader
[junit4] 2> 678004 T2983 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/collection1/lib/classes/' to classloader
[junit4] 2> 678005 T2983 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/collection1/lib/README' to classloader
[junit4] 2> 678030 T2983 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 678057 T2983 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 678058 T2983 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 678062 T2983 oass.IndexSchema.readSchema [collection1] Schema name=sortable
[junit4] 2> 678472 T2983 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 678473 T2983 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
[junit4] 2> 678474 T2983 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 678474 T2983 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-002/collection1/, dataDir=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/control/data/
[junit4] 2> 678474 T2983 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 678475 T2983 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/control/data
[junit4] 2> 678475 T2983 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/control/data/index/
[junit4] 2> 678475 T2983 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
[junit4] 2> 678475 T2983 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/control/data/index
[junit4] 2> 678476 T2983 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=27.4814453125, floorSegmentMB=1.48046875, forceMergeDeletesPctAllowed=18.9856854619832, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 678476 T2983 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7a95bcf9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61e14dc9),segFN=segments_1,generation=1}
[junit4] 2> 678477 T2983 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 678477 T2983 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 678477 T2983 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 678477 T2983 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 678477 T2983 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 678478 T2983 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
[junit4] 2> 678478 T2983 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 678482 T2983 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
[junit4] 2> 678483 T2983 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 678484 T2983 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 678485 T2983 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 678485 T2983 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=74.5869140625, floorSegmentMB=0.8916015625, forceMergeDeletesPctAllowed=15.107320202651106, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 678485 T2983 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7a95bcf9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@61e14dc9),segFN=segments_1,generation=1}
[junit4] 2> 678485 T2983 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 678485 T2983 oass.SolrIndexSearcher.<init> Opening Searcher@18029da4[collection1] main
[junit4] 2> 678486 T2983 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
[junit4] 2> 678486 T2983 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 678487 T2983 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 678487 T2983 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 678487 T2983 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 678487 T2983 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 678488 T2983 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 678488 T2983 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 678488 T2983 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 678489 T2983 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 678489 T2983 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 678489 T2984 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18029da4[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 678490 T2987 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35958/q_u/yf collection:control_collection shard:shard1
[junit4] 2> 678490 T2958 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1
[junit4] 2> 678490 T2958 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 678491 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 678492 T2987 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4] 2> 678494 T2989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@751d0f44 name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 678494 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 678495 T2958 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 678497 T2958 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 678501 T2987 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 678502 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 678502 T2987 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 678503 T2987 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> ASYNC NEW_CORE C574 name=collection1 org.apache.solr.core.SolrCore@6a735b5c url=http://127.0.0.1:35958/q_u/yf/collection1 node=127.0.0.1:35958_q_u%2Fyf C574_STATE=coll:control_collection core:collection1 props:{state=down, base_url=http://127.0.0.1:35958/q_u/yf, core=collection1, node_name=127.0.0.1:35958_q_u%2Fyf}
[junit4] 2> 678503 T2987 C574 P35958 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35958/q_u/yf/collection1/
[junit4] 2> 678503 T2987 C574 P35958 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 678503 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 678503 T2987 C574 P35958 oasc.SyncStrategy.syncToMe http://127.0.0.1:35958/q_u/yf/collection1/ has no replicas
[junit4] 2> 678504 T2987 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35958/q_u/yf/collection1/ shard1
[junit4] 2> 678504 T2987 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4] 2> 678505 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 678506 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 678506 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 678509 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 678556 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 678658 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 678658 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 678709 T2987 oasc.ZkController.register We are http://127.0.0.1:35958/q_u/yf/collection1/ and leader is http://127.0.0.1:35958/q_u/yf/collection1/
[junit4] 2> 678709 T2987 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35958/q_u/yf
[junit4] 2> 678709 T2987 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 678709 T2987 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
[junit4] 2> 678709 T2987 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 678710 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 678710 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 678710 T2987 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 678710 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 678711 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 678712 T2981 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:35958/q_u/yf",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:35958_q_u%2Fyf",
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"control_collection",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 678713 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 678751 T2958 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
[junit4] 2> 678751 T2958 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 678753 T2958 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60531
[junit4] 2> 678753 T2958 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 678754 T2958 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 678754 T2958 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003
[junit4] 2> 678754 T2958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/'
[junit4] 2> 678766 T2958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/solr.xml
[junit4] 2> 678773 T2958 oasc.CoreContainer.<init> New CoreContainer 699432553
[junit4] 2> 678773 T2958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/]
[junit4] 2> 678773 T2958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 678774 T2958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 678774 T2958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 678774 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 678774 T2958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 678774 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 678775 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 678775 T2958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 678775 T2958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 678775 T2958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 678776 T2958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 678776 T2958 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 678776 T2958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46594/solr
[junit4] 2> 678776 T2958 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 678777 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 678779 T3000 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@65c61e52 name:ZooKeeperConnection Watcher:127.0.0.1:46594 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 678779 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 678781 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 678782 T3002 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@392eef5d name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 678782 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 678787 T2958 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 678815 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 678815 T3002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 678815 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4] 2> 679789 T2958 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60531_q_u%2Fyf
[junit4] 2> 679791 T2958 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60531_q_u%2Fyf
[junit4] 2> 679794 T2980 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 679794 T3002 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 679794 T2989 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4] 2> 679800 T3003 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 679800 T3003 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 679802 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 679803 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 679803 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 679802 T3003 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 679805 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 679806 T2981 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60531/q_u/yf",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60531_q_u%2Fyf",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 679806 T2981 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
[junit4] 2> 679806 T2981 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4] 2> 679809 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 679913 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 679913 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 679913 T3002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 680804 T3003 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 680804 T3003 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 680805 T3003 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 680805 T3003 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 680807 T3003 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 680807 T3003 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/collection1/'
[junit4] 2> 680808 T3003 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/collection1/lib/.svn/' to classloader
[junit4] 2> 680808 T3003 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/collection1/lib/classes/' to classloader
[junit4] 2> 680809 T3003 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/collection1/lib/README' to classloader
[junit4] 2> 680834 T3003 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 680839 T3003 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 680840 T3003 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 680843 T3003 oass.IndexSchema.readSchema [collection1] Schema name=sortable
[junit4] 2> 680982 T3003 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 680983 T3003 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 680983 T3003 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 680983 T3003 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-003/collection1/, dataDir=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty1/
[junit4] 2> 680984 T3003 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 680984 T3003 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty1
[junit4] 2> 680984 T3003 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty1/index/
[junit4] 2> 680984 T3003 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
[junit4] 2> 680985 T3003 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty1/index
[junit4] 2> 680985 T3003 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=27.4814453125, floorSegmentMB=1.48046875, forceMergeDeletesPctAllowed=18.9856854619832, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 680985 T3003 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7568cfbf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c62070f),segFN=segments_1,generation=1}
[junit4] 2> 680986 T3003 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 680986 T3003 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 680986 T3003 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 680986 T3003 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 680987 T3003 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 680987 T3003 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
[junit4] 2> 680987 T3003 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 680987 T3003 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
[junit4] 2> 680988 T3003 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 680995 T3003 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 680995 T3003 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 680996 T3003 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=74.5869140625, floorSegmentMB=0.8916015625, forceMergeDeletesPctAllowed=15.107320202651106, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 680996 T3003 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@7568cfbf lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c62070f),segFN=segments_1,generation=1}
[junit4] 2> 680996 T3003 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 680997 T3003 oass.SolrIndexSearcher.<init> Opening Searcher@27905e02[collection1] main
[junit4] 2> 680997 T3003 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 680998 T3003 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 680998 T3003 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 680998 T3003 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 680999 T3003 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 680999 T3003 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 680999 T3003 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 680999 T3003 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 681000 T3003 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 681000 T3003 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 681000 T3003 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 681000 T3004 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@27905e02[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 681001 T3007 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60531/q_u/yf collection:collection1 shard:shard2
[junit4] 2> 681001 T2958 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1
[junit4] 2> 681002 T2958 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 681002 T3007 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4] 2> 681006 T3007 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4] 2> 681007 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681007 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681008 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681008 T3007 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 681008 T3007 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 681008 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> ASYNC NEW_CORE C575 name=collection1 org.apache.solr.core.SolrCore@4a238159 url=http://127.0.0.1:60531/q_u/yf/collection1 node=127.0.0.1:60531_q_u%2Fyf C575_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:60531/q_u/yf, core=collection1, node_name=127.0.0.1:60531_q_u%2Fyf}
[junit4] 2> 681008 T3007 C575 P60531 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60531/q_u/yf/collection1/
[junit4] 2> 681009 T3007 C575 P60531 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 681009 T3007 C575 P60531 oasc.SyncStrategy.syncToMe http://127.0.0.1:60531/q_u/yf/collection1/ has no replicas
[junit4] 2> 681009 T3007 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60531/q_u/yf/collection1/ shard2
[junit4] 2> 681009 T3007 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4] 2> 681010 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681012 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681013 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681118 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 681118 T3002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 681118 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 681162 T3007 oasc.ZkController.register We are http://127.0.0.1:60531/q_u/yf/collection1/ and leader is http://127.0.0.1:60531/q_u/yf/collection1/
[junit4] 2> 681163 T3007 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60531/q_u/yf
[junit4] 2> 681163 T3007 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4] 2> 681163 T3007 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
[junit4] 2> 681163 T3007 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 681164 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681164 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681164 T3007 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 681164 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681165 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 681165 T2981 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"active",
[junit4] 2> "base_url":"http://127.0.0.1:60531/q_u/yf",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:60531_q_u%2Fyf",
[junit4] 2> "shard":"shard2",
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":"core_node1"}
[junit4] 2> 681167 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 681225 T2958 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
[junit4] 2> 681226 T2958 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 681227 T2958 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:59619
[junit4] 2> 681229 T2958 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 681230 T2958 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 681230 T2958 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004
[junit4] 2> 681230 T2958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/'
[junit4] 2> 681240 T2958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/solr.xml
[junit4] 2> 681246 T2958 oasc.CoreContainer.<init> New CoreContainer 1250248907
[junit4] 2> 681246 T2958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/]
[junit4] 2> 681247 T2958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 681247 T2958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 681247 T2958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 681247 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 681247 T2958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 681248 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 681248 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 681248 T2958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 681248 T2958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 681249 T2958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 681249 T2958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 681249 T2958 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 681249 T2958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46594/solr
[junit4] 2> 681250 T2958 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 681250 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 681252 T3018 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45a4be2c name:ZooKeeperConnection Watcher:127.0.0.1:46594 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 681252 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 681255 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 681256 T3020 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@698e1d72 name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 681256 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 681260 T2958 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4] 2> 681268 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 681268 T3020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 681268 T3002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 681268 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4] 2> 682262 T2958 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59619_q_u%2Fyf
[junit4] 2> 682263 T2958 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:59619_q_u%2Fyf
[junit4] 2> 682265 T3002 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 682265 T2980 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 682265 T3020 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 682265 T2989 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4] 2> 682269 T3021 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
[junit4] 2> 682269 T3021 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4] 2> 682270 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 682270 T3021 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4] 2> 682270 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 682271 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 682271 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 682271 T2981 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4] 2> "operation":"state",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:59619/q_u/yf",
[junit4] 2> "core":"collection1",
[junit4] 2> "roles":null,
[junit4] 2> "node_name":"127.0.0.1:59619_q_u%2Fyf",
[junit4] 2> "shard":null,
[junit4] 2> "collection":"collection1",
[junit4] 2> "numShards":"2",
[junit4] 2> "core_node_name":null}
[junit4] 2> 682272 T2981 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4] 2> 682272 T2981 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4] 2> 682273 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 683811 T2962 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 1435ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 683812 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 683812 T3020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 683812 T3002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 683812 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 684271 T3021 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4] 2> 684271 T3021 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4] 2> 684272 T3021 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4] 2> 684272 T3021 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 684273 T3021 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 684274 T3021 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: './solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/collection1/'
[junit4] 2> 684274 T3021 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/collection1/lib/.svn/' to classloader
[junit4] 2> 684275 T3021 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/collection1/lib/classes/' to classloader
[junit4] 2> 684275 T3021 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/collection1/lib/README' to classloader
[junit4] 2> 684297 T3021 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_48
[junit4] 2> 684303 T3021 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4] 2> 684303 T3021 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4] 2> 684306 T3021 oass.IndexSchema.readSchema [collection1] Schema name=sortable
[junit4] 2> 684453 T3021 oass.IndexSchema.readSchema unique key field: id
[junit4] 2> 684454 T3021 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
[junit4] 2> 684454 T3021 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4] 2> 684454 T3021 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-004/collection1/, dataDir=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty2/
[junit4] 2> 684454 T3021 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
[junit4] 2> 684454 T3021 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty2
[junit4] 2> 684455 T3021 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty2/index/
[junit4] 2> 684455 T3021 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
[junit4] 2> 684455 T3021 oasc.CachingDirectoryFactory.get return new directory for ./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-001/jetty2/index
[junit4] 2> 684455 T3021 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=18, maxMergedSegmentMB=27.4814453125, floorSegmentMB=1.48046875, forceMergeDeletesPctAllowed=18.9856854619832, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 684456 T3021 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@16b9d618 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30bed86b),segFN=segments_1,generation=1}
[junit4] 2> 684456 T3021 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 684457 T3021 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
[junit4] 2> 684457 T3021 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4] 2> 684457 T3021 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4] 2> 684457 T3021 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4] 2> 684457 T3021 oasc.RequestHandlers.initHandlersFromConfig created /select: solr.SearchHandler
[junit4] 2> 684457 T3021 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4] 2> 684457 T3021 oasc.RequestHandlers.initHandlersFromConfig created /admin/: solr.admin.AdminHandlers
[junit4] 2> 684459 T3021 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4] 2> 684460 T3021 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4] 2> 684460 T3021 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4] 2> 684460 T3021 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=11, maxMergedSegmentMB=74.5869140625, floorSegmentMB=0.8916015625, forceMergeDeletesPctAllowed=15.107320202651106, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 684460 T3021 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4] 2> commit{dir=MockDirectoryWrapper(RAMDirectory@16b9d618 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@30bed86b),segFN=segments_1,generation=1}
[junit4] 2> 684461 T3021 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4] 2> 684461 T3021 oass.SolrIndexSearcher.<init> Opening Searcher@7058c376[collection1] main
[junit4] 2> 684461 T3021 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
[junit4] 2> 684462 T3021 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
[junit4] 2> 684462 T3021 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
[junit4] 2> 684462 T3021 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 684462 T3021 oasr.RestManager.init Initializing RestManager with initArgs: {}
[junit4] 2> 684462 T3021 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 684463 T3021 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
[junit4] 2> 684463 T3021 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 684463 T3021 oasr.ManagedResource.notifyObserversDuringInit WARN No registered observers for /rest/managed
[junit4] 2> 684463 T3021 oasr.RestManager.init Initializing 0 registered ManagedResources
[junit4] 2> 684464 T3022 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7058c376[collection1] main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4] 2> 684464 T3021 oasc.CoreContainer.registerCore registering core: collection1
[junit4] 2> 684464 T3025 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:59619/q_u/yf collection:collection1 shard:shard1
[junit4] 2> 684464 T2958 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1
[junit4] 2> 684465 T2958 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4] 2> 684465 T3025 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4] 2> 684542 T3025 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4] 2> 684582 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 684582 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 684582 T3025 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4] 2> 684582 T3025 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4] 2> 684582 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> ASYNC NEW_CORE C576 name=collection1 org.apache.solr.core.SolrCore@556c9a3f url=http://127.0.0.1:59619/q_u/yf/collection1 node=127.0.0.1:59619_q_u%2Fyf C576_STATE=coll:collection1 core:collection1 props:{state=down, base_url=http://127.0.0.1:59619/q_u/yf, core=collection1, node_name=127.0.0.1:59619_q_u%2Fyf}
[junit4] 2> 684582 T3025 C576 P59619 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:59619/q_u/yf/collection1/
[junit4] 2> 684583 T3025 C576 P59619 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4] 2> 684583 T3025 C576 P59619 oasc.SyncStrategy.syncToMe http://127.0.0.1:59619/q_u/yf/collection1/ has no replicas
[junit4] 2> 684583 T3025 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:59619/q_u/yf/collection1/ shard1
[junit4] 2> 684583 T3025 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4] 2> 684584 T2981 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4] 2> 684642 T2980 oasc.DistributedQueue$LatchChildWatcher.process LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type NodeChildrenChanged
[junit4] 2> 684682 T2989 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 684682 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 684682 T3002 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 684682 T3020 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4] 2> 684685 T2958 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
[junit4] 2> 684686 T2958 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4] 2> 684688 T2958 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50214
[junit4] 2> 684689 T2958 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4] 2> 684689 T2958 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4] 2> 684689 T2958 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-005
[junit4] 2> 684689 T2958 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-005/'
[junit4] 2> 684699 T2958 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-005/solr.xml
[junit4] 2> 684704 T2958 oasc.CoreContainer.<init> New CoreContainer 1008089750
[junit4] 2> 684705 T2958 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build/solr-core/test/J1/./solr.cloud.DistribCursorPagingTest-CA2A8A102C0F8CB3-001/tempDir-005/]
[junit4] 2> 684705 T2958 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
[junit4] 2> 684705 T2958 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to:
[junit4] 2> 684706 T2958 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4] 2> 684706 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4] 2> 684706 T2958 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4] 2> 684706 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4] 2> 684706 T2958 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4] 2> 684706 T2958 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4] 2> 684707 T2958 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4] 2> 684707 T2958 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
[junit4] 2> 684707 T2958 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
[junit4] 2> 684707 T2958 oasc.CoreContainer.load Host Name: 127.0.0.1
[junit4] 2> 684708 T2958 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:46594/solr
[junit4] 2> 684708 T2958 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4] 2> 684709 T2958 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4] 2> 691352 T2960 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145db56f2860004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 691452 T2989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@751d0f44 name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 691452 T2989 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 692613 T2989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@751d0f44 name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 698021 T2960 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145db56f2860008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 698021 T2960 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145db56f2860003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 698022 T2960 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145db56f2860006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 699283 T2960 oazs.NIOServerCnxn.doIO WARN caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x145db56f2860004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
[junit4] 2> at java.lang.Thread.run(Thread.java:745)
[junit4] 2>
[junit4] 2> 699383 T2989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@751d0f44 name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:Disconnected type:None path:null path:null type:None
[junit4] 2> 699383 T2989 oascc.ConnectionManager.process zkClient has disconnected
[junit4] 2> 700953 T2989 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@751d0f44 name:ZooKeeperConnection Watcher:127.0.0.1:46594/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 706658 T2962 oazsp.FileTxnLog.commit WARN fsync-ing the write ahead log in SyncThread:0 took 21975ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide
[junit4] 2> 706659 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 706659 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:169)
[junit4] 2>
[junit4] 2> 706661 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 706662 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 706663 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 706663 T3036 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76a4ea60 name:ZooKeeperConnection Watcher:127.0.0.1:46594 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4] 2> 706663 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 706664 T2958 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4] 2> 706664 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 706666 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
[junit4] 2> at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:170)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
[junit4] 2> at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
[junit4] 2>
[junit4] 2> 706666 T2962 oazs.NIOServerCnxn.sendBuffer ERROR Unexpected Exception: java.nio.channels.CancelledKeyException
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java
[...truncated too long message...]
ngsFormat(name=Asserting), currency____currency=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), date_first=Lucene41(blocksize=128), legacy_sortable_double=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), legacy_date_last=Lucene41(blocksize=128), bool=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), currency_first____amount_raw=PostingsFormat(name=Memory doPackFST= false), currency_first____currency=PostingsFormat(name=Memory doPackFST= false), double_last=PostingsFormat(name=Asserting), int_last=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), long=PostingsFormat(name=Asserting), enum_first=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), bcd_int_last=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), double_first=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), legacy_sortable_double_first=PostingsFormat(name=Memory doPackFST= false), icu_collation_en_primary_first=PostingsFormat(name=Memory doPackFST= false), float_first=Lucene41(blocksize=128), legacy_sortable_int=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), currency____amount_raw=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), long_first=Lucene41(blocksize=128), currency_last____currency=Lucene41(blocksize=128), bcd_long_first=Lucene41(blocksize=128), bool_first=PostingsFormat(name=Memory doPackFST= false), legacy_sortable_long_last=PostingsFormat(name=Asserting), long_last=PostingsFormat(name=Memory doPackFST= false), bcd_int_first=PostingsFormat(name=Asserting), legacy_sortable_long=PostingsFormat(name=Memory doPackFST= false), bcd_long_last=PostingsFormat(name=Memory doPackFST= false), int=Lucene41(blocksize=128), legacy_int=PostingsFormat(name=Memory doPackFST= false), int_first=PostingsFormat(name=Asserting), legacy_long=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), enum_last=PostingsFormat(name=Asserting), _version_=Pulsing41(freqCutoff=1 minBlockSize=94 maxBlockSize=252), bcd_str=Lucene41(blocksize=128), currency_last____amount_raw=PostingsFormat(name=Memory doPackFST= false), float_last=Lucene41(blocksize=128), legacy_date_first=PostingsFormat(name=Memory doPackFST= false), legacy_sortable_int_last=Lucene41(blocksize=128)}, docValues:{uuid_dv=DocValuesFormat(name=SimpleText), collation_en_primary_dv_first=DocValuesFormat(name=Lucene45), icu_collation_en_primary_dv_first=DocValuesFormat(name=Lucene45), date_dv=DocValuesFormat(name=Disk), collation_en_primary_dv=DocValuesFormat(name=Asserting), float_dv_first=DocValuesFormat(name=SimpleText), double_dv_last=DocValuesFormat(name=Disk), uuid_dv_last=DocValuesFormat(name=Asserting), date_dv_last=DocValuesFormat(name=SimpleText), date_dv_first=DocValuesFormat(name=Lucene45), str_dv_last=DocValuesFormat(name=Disk), icu_collation_en_primary_dv_last=DocValuesFormat(name=SimpleText), int_dv=DocValuesFormat(name=SimpleText), str_dv_first=DocValuesFormat(name=Asserting), icu_collation_en_primary_dv=DocValuesFormat(name=Asserting), float_dv=DocValuesFormat(name=Asserting), long_dv_last=DocValuesFormat(name=Lucene45), int_dv_first=DocValuesFormat(name=Disk), collation_en_primary_dv_last=DocValuesFormat(name=SimpleText), long_dv_first=DocValuesFormat(name=SimpleText), double_dv_first=DocValuesFormat(name=Disk), float_dv_last=DocValuesFormat(name=Lucene45), bin_dv_last=DocValuesFormat(name=Asserting), uuid_dv_first=DocValuesFormat(name=Asserting), double_dv=DocValuesFormat(name=Lucene45), str_dv=DocValuesFormat(name=Lucene45), int_dv_last=DocValuesFormat(name=Disk), bin_dv_first=DocValuesFormat(name=Asserting), long_dv=DocValuesFormat(name=Asserting), bin_dv=DocValuesFormat(name=SimpleText)}, sim=DefaultSimilarity, locale=ar_SA, timezone=Canada/Atlantic
[junit4] 2> NOTE: Linux 3.8.0-39-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=2,free=103180440,total=278921216
[junit4] 2> NOTE: All tests run in this JVM: [TestFoldingMultitermQuery, TestManagedSynonymFilterFactory, TestStandardQParsers, TestStressReorder, XsltUpdateRequestHandlerTest, TestRangeQuery, HdfsWriteToMultipleCollectionsTest, DistributedSuggestComponentTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, TestSchemaSimilarityResource, TestCodecSupport, TestFastOutputStream, TestInfoStreamLogging, DeleteInactiveReplicaTest, TestRecovery, NotRequiredUniqueKeyTest, BasicDistributedZk2Test, FieldAnalysisRequestHandlerTest, TestUtils, TestRandomFaceting, TestLazyCores, TestIBSimilarityFactory, MinimalSchemaTest, PrimUtilsTest, SSLMigrationTest, TestComponentsName, TestIndexSearcher, TestCSVLoader, QueryResultKeyTest, TestStressRecovery, TestHighFrequencyDictionaryFactory, BadIndexSchemaTest, TestAnalyzedSuggestions, TestSchemaResource, TestJmxIntegration, SpellCheckCollatorTest, IndexBasedSpellCheckerTest, TestDefaultSimilarityFactory, TimeZoneUtilsTest, DirectSolrSpellCheckerTest, TestBinaryField, AlternateDirectoryTest, CurrencyFieldXmlFileTest, TestQuerySenderListener, TestQuerySenderNoQuery, MigrateRouteKeyTest, TestNumberUtils, FileUtilsTest, HighlighterConfigTest, TestDistribDocBasedVersion, ParsingFieldUpdateProcessorsTest, TestDynamicFieldCollectionResource, CachingDirectoryFactoryTest, TestHashPartitioner, TestExpandComponent, TestFileDictionaryLookup, TestCloudManagedSchema, TestSolrCoreProperties, DistributedExpandComponentTest, ChaosMonkeySafeLeaderTest, AliasIntegrationTest, TestInitQParser, BlockDirectoryTest, TestSolrQueryParserDefaultOperatorResource, ChaosMonkeyNothingIsSafeTest, FullSolrCloudDistribCmdsTest, HighlighterTest, HdfsDirectoryTest, RequestHandlersTest, LeaderElectionIntegrationTest, CoreContainerCoreInitFailuresTest, TestSolrXml, DocValuesMultiTest, MultiTermTest, DistanceFunctionTest, ZkSolrClientTest, TestDistributedGrouping, DistribCursorPagingTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=DistribCursorPagingTest -Dtests.seed=CA2A8A102C0F8CB3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SA -Dtests.timezone=Canada/Atlantic -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J1 | DistribCursorPagingTest (suite) <<<
[junit4] > Throwable #1: java.lang.AssertionError: ERROR: SolrZkClient opens=12 closes=11
[junit4] > at __randomizedtesting.SeedInfo.seed([CA2A8A102C0F8CB3]:0)
[junit4] > at org.apache.solr.SolrTestCaseJ4.endTrackingZkClients(SolrTestCaseJ4.java:434)
[junit4] > at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:180)
[junit4] > at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.DistribCursorPagingTest:
[junit4] > 1) Thread[id=3037, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-SendThread(127.0.0.1:46594), state=TIMED_WAITING, group=TGRP-DistribCursorPagingTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
[junit4] > 2) Thread[id=3038, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-EventThread, state=WAITING, group=TGRP-DistribCursorPagingTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
[junit4] > at __randomizedtesting.SeedInfo.seed([CA2A8A102C0F8CB3]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=3037, name=TEST-DistribCursorPagingTest.testDistribSearch-seed#[CA2A8A102C0F8CB3]-SendThread(127.0.0.1:46594), state=TIMED_WAITING, group=TGRP-DistribCursorPagingTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
[junit4] > at __randomizedtesting.SeedInfo.seed([CA2A8A102C0F8CB3]:0)
[junit4] Completed on J1 in 102.39s, 1 test, 1 failure, 2 errors <<< FAILURES!
[...truncated 699 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:447: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/build.xml:45: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/extra-targets.xml:37: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/build.xml:189: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/solr/common-build.xml:496: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:1293: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.8-Linux/lucene/common-build.xml:919: There were test failures: 387 suites, 1624 tests, 3 suite-level errors, 813 ignored (9 assumptions)
Total time: 48 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.8.0_05 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure