You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/12/15 16:48:33 UTC

[JENKINS] Lucene-Solr-trunk-Solaris (64bit/jdk1.8.0) - Build # 261 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Solaris/261/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreDiscovery

Error Message:
ERROR: SolrIndexSearcher opens=27 closes=26

Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=27 closes=26
	at __randomizedtesting.SeedInfo.seed([45F43F257D825341]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:453)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:225)
	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreDiscovery

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestCoreDiscovery:     1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestCoreDiscovery: 
   1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([45F43F257D825341]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestCoreDiscovery

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([45F43F257D825341]:0)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Tue Dec 15 18:41:37 MSK 2015

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Tue Dec 15 18:41:37 MSK 2015
	at __randomizedtesting.SeedInfo.seed([45F43F257D825341:9E5F3FE378AA3AF2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1419)
	at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:771)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10215 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/init-core-data-001
   [junit4]   2> 1864473 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEmptyCommits
   [junit4]   2> 1864474 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1
   [junit4]   2> 1864509 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1864510 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@774784f0{/solr,null,AVAILABLE}
   [junit4]   2> 1864529 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.ServerConnector Started ServerConnector@35cee0d2{HTTP/1.1,[http/1.1]}{127.0.0.1:49643}
   [junit4]   2> 1864529 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.Server Started @1872509ms
   [junit4]   2> 1864529 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=49643}
   [junit4]   2> 1864529 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@659e0bfd
   [junit4]   2> 1864529 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001'
   [junit4]   2> 1864530 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1864530 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1864530 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/solr.xml
   [junit4]   2> 1864543 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/.
   [junit4]   2> 1864543 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer New CoreContainer 814676950
   [junit4]   2> 1864543 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001]
   [junit4]   2> 1864543 WARN  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/lib
   [junit4]   2> 1864544 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1864545 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1864545 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1864545 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1864546 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1864546 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1864546 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/.
   [junit4]   2> 1864547 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 1864548 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1
   [junit4]   2> 1864548 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1864549 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 1864549 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1864550 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1'
   [junit4]   2> 1864550 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1864550 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1864557 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1864559 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1864563 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1864566 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1864568 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1864569 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1'
   [junit4]   2> 1864570 INFO  (coreLoadExecutor-5255-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1
   [junit4]   2> 1864570 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1864570 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 1864570 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1864570 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1/data
   [junit4]   2> 1864570 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1/data/index/
   [junit4]   2> 1864570 WARN  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1864570 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1/data/index
   [junit4]   2> 1864571 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=35.31640625, floorSegmentMB=1.3564453125, forceMergeDeletesPctAllowed=11.3188745513442, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1864571 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_1,generation=1}
   [junit4]   2> 1864571 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1864572 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/./collection1/data/
   [junit4]   2> 1864595 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1864598 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1864605 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1864605 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1864606 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1864608 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1864608 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,lazy,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,defaults,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1864608 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1864609 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1864609 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1864609 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=613440346, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1864609 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_1,generation=1}
   [junit4]   2> 1864609 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1864609 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@63728d0[collection1] main
   [junit4]   2> 1864610 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1/conf
   [junit4]   2> 1864610 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1/conf}
   [junit4]   2> 1864610 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-001/collection1/conf
   [junit4]   2> 1864610 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1864610 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 1864610 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1864610 INFO  (coreLoadExecutor-5255-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1864610 INFO  (searcherExecutor-5256-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@63728d0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1865054 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1
   [junit4]   2> 1865057 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.Server jetty-9.3.6.v20151106
   [junit4]   2> 1865061 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@f484285{/solr,null,AVAILABLE}
   [junit4]   2> 1865062 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.ServerConnector Started ServerConnector@7c54739e{HTTP/1.1,[http/1.1]}{127.0.0.1:51195}
   [junit4]   2> 1865062 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.e.j.s.Server Started @1873042ms
   [junit4]   2> 1865062 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=51195}
   [junit4]   2> 1865062 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@659e0bfd
   [junit4]   2> 1865062 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002'
   [junit4]   2> 1865062 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1865062 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1865063 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/solr.xml
   [junit4]   2> 1865066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/.
   [junit4]   2> 1865066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer New CoreContainer 1163977227
   [junit4]   2> 1865066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002]
   [junit4]   2> 1865066 WARN  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer Couldn't add files from /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/lib to classpath: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/lib
   [junit4]   2> 1865066 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1865067 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1865068 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1865068 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1865068 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1865068 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1865069 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/.
   [junit4]   2> 1865070 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 1865070 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1
   [junit4]   2> 1865070 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1865071 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1
   [junit4]   2> 1865071 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1865073 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1'
   [junit4]   2> 1865074 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1865074 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1865078 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1865080 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1865084 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1865086 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1865088 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1865089 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1'
   [junit4]   2> 1865089 INFO  (coreLoadExecutor-5265-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1
   [junit4]   2> 1865089 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1865089 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 1865089 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 1865090 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data
   [junit4]   2> 1865090 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index/
   [junit4]   2> 1865090 WARN  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1865090 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 1865090 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=22, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=35.31640625, floorSegmentMB=1.3564453125, forceMergeDeletesPctAllowed=11.3188745513442, segmentsPerTier=27.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1865091 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_1,generation=1}
   [junit4]   2> 1865091 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1865091 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/
   [junit4]   2> 1865093 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1865094 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1865095 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1865096 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1865097 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1865098 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 1865098 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,lazy,/update/csv,/update/json/docs,/admin/luke,/admin/segments,/get,/admin/system,/replication,/admin/properties,defaults,/config,/schema,/admin/plugins,/admin/logging,/update/json,/admin/threads,/admin/ping,/update,/admin/file
   [junit4]   2> 1865098 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1865098 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1865098 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1865099 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=613440346, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1865099 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_1,generation=1}
   [junit4]   2> 1865100 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1865100 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@3d079b9e[collection1] main
   [junit4]   2> 1865100 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1/conf
   [junit4]   2> 1865100 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1/conf}
   [junit4]   2> 1865100 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/collection1/conf
   [junit4]   2> 1865100 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1865101 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 1865101 INFO  (coreLoadExecutor-5265-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1865102 INFO  (searcherExecutor-5266-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3d079b9e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1865576 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 1865579 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 0
   [junit4]   2> 1865581 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1865583 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_2,generation=2}
   [junit4]   2> 1865583 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 1865585 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@48a44ca7[collection1] main
   [junit4]   2> 1865585 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1865585 INFO  (searcherExecutor-5256-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@48a44ca7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1)))}
   [junit4]   2> 1865585 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={commit=true&wt=javabin&version=2}{commit=} 0 3
   [junit4]   2> 1865594 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1865596 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865596 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 1865597 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 1865597 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1865598 WARN  (qtp1300959840-12692) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _0_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865601 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=3
   [junit4]   2> 1865602 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 11
   [junit4]   2> 1865602 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131452
   [junit4]   2> 1865602 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 1865602 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5999591403033244]
   [junit4]   2> 1865603 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_1,generation=1}
   [junit4]   2> 1865604 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1865604 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1865604 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_1,generation=1}
   [junit4]   2> 1865604 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1865604 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 1865604 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@7a19b45e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11382706)
   [junit4]   2> 1865606 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865613 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_MockRandom_0.tix&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865616 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_MockRandom_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865619 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865621 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_MockRandom_0.tbk&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865623 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_MockRandom_0.sd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865626 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865629 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865631 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865633 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865635 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865636 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1594) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@7a19b45e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11382706)
   [junit4]   2> 1865638 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=31.6005859375, floorSegmentMB=1.2138671875, forceMergeDeletesPctAllowed=6.390960140636153, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1865638 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_2,generation=2}
   [junit4]   2> 1865639 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 1865639 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1865639 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@6075da2d[collection1] main
   [junit4]   2> 1865640 INFO  (searcherExecutor-5266-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6075da2d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1)))}
   [junit4]   2> 1865640 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131452 [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131452;done=true>>]
   [junit4]   2> 1865640 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131452
   [junit4]   2> 1865640 INFO  (qtp2125443004-12712) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:49643/solr/collection1/replication&command=fetchindex} status=0 QTime=49
   [junit4]   2> 1865646 INFO  (qtp2125443004-12714) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=name:empty1&sort=id+desc&wt=javabin&version=2} hits=1 status=0 QTime=3
   [junit4]   2> 1865646 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 1 docs
   [junit4]   2> 1865648 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1865651 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1865658 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1865659 INFO  (qtp2125443004-12716) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=5
   [junit4]   2> 1865661 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865664 INFO  (qtp2125443004-12711) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865666 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1865666 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 1865667 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.SolrCore SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 1865667 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1865667 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={commit=true&wt=javabin&version=2}{commit=} 0 1
   [junit4]   2> 1865669 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1865671 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865671 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave in sync with master.
   [junit4]   2> 1865672 INFO  (qtp2125443004-12715) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:49643/solr/collection1/replication&command=fetchindex} status=0 QTime=4
   [junit4]   2> 1865676 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[2]} 0 0
   [junit4]   2> 1865678 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1865683 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865684 INFO  (indexFetcher-5270-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave in sync with master.
   [junit4]   2> 1865684 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_2,generation=2}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_3,generation=3}
   [junit4]   2> 1865685 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 3
   [junit4]   2> 1865685 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@33d8f46d[collection1] main
   [junit4]   2> 1865685 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1865685 INFO  (searcherExecutor-5256-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@33d8f46d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_1(6.0.0):C1)))}
   [junit4]   2> 1865685 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={commit=true&wt=javabin&version=2}{commit=} 0 7
   [junit4]   2> 1865687 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1865690 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865691 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 3
   [junit4]   2> 1865691 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 2
   [junit4]   2> 1865691 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1865693 WARN  (qtp1300959840-12691) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _0_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865693 WARN  (qtp1300959840-12691) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _1_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865693 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=1
   [junit4]   2> 1865694 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 20
   [junit4]   2> 1865695 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131545
   [junit4]   2> 1865695 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 1865695 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5999591403033244]
   [junit4]   2> 1865696 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_2,generation=2}
   [junit4]   2> 1865696 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_2,generation=2}
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@6dda9a5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6b2ad8bc)
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.si because it already exists
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0_MockRandom_0.tix because it already exists
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0_MockRandom_0.doc because it already exists
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.nvd because it already exists
   [junit4]   2> 1865697 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0_MockRandom_0.tbk because it already exists
   [junit4]   2> 1865699 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_MockRandom_0.sd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865700 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.fdx because it already exists
   [junit4]   2> 1865700 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.fdt because it already exists
   [junit4]   2> 1865701 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.fnm because it already exists
   [junit4]   2> 1865701 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Skipping download for _0.nvm because it already exists
   [junit4]   2> 1865703 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_MockRandom_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865705 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_MockRandom_0.sd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865709 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865712 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865715 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865717 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865719 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865722 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_MockRandom_0.tfp&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865725 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865729 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=segments_3&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865730 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1571) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@6dda9a5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6b2ad8bc)
   [junit4]   2> 1865731 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Cannot complete replication attempt because file already exists:_0_MockRandom_0.sd
   [junit4]   2> 1865731 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=23, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=31.6005859375, floorSegmentMB=1.2138671875, forceMergeDeletesPctAllowed=6.390960140636153, segmentsPerTier=32.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1865731 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567),segFN=segments_2,generation=2}
   [junit4]   2> 1865731 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 1865733 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1865733 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131545 [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131545;done=true>>]
   [junit4]   2> 1865733 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131545
   [junit4]   2> 1865733 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Replication attempt was not successful - trying a full index replication reloadCore=false
   [junit4]   2> 1865735 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865735 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 3
   [junit4]   2> 1865735 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 2
   [junit4]   2> 1865735 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1865743 WARN  (qtp1300959840-12692) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _0_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865744 WARN  (qtp1300959840-12692) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _1_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865744 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=1
   [junit4]   2> 1865745 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 20
   [junit4]   2> 1865745 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131595
   [junit4]   2> 1865745 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=true) to MockDirectoryWrapper(RAMDirectory@658b3481 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c5308db)
   [junit4]   2> 1865747 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865750 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_MockRandom_0.tix&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865753 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_MockRandom_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865757 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865775 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_MockRandom_0.tbk&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865778 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0_MockRandom_0.sd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865781 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865784 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865787 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865789 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865793 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_MockRandom_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865796 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_MockRandom_0.sd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865799 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865803 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865805 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865808 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865811 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865814 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1_MockRandom_0.tfp&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865816 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=_1.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865819 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=3&qt=/replication&file=segments_3&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 1865819 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=true,bytesDownloaded=2991) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@658b3481 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c5308db)
   [junit4]   2> 1865820 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher New index installed. Updating index properties... index=index.20151215184131595
   [junit4]   2> 1865820 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher removing old index directory MockDirectoryWrapper(RAMDirectory@3cf8626a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7385f567)
   [junit4]   2> 1865820 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 1865821 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index/ new=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index.20151215184131595
   [junit4]   2> 1865821 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=37, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=57.8359375, floorSegmentMB=0.7958984375, forceMergeDeletesPctAllowed=10.603727199166821, segmentsPerTier=43.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1865821 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@658b3481 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1c5308db),segFN=segments_3,generation=3}
   [junit4]   2> 1865821 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 3
   [junit4]   2> 1865821 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 1865822 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@7143a26e[collection1] main
   [junit4]   2> 1865822 INFO  (searcherExecutor-5266-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7143a26e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C1) Uninverting(_1(6.0.0):C1)))}
   [junit4]   2> 1865823 INFO  (qtp2125443004-12709) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:49643/solr/collection1/replication&command=fetchindex} status=0 QTime=136
   [junit4]   2> 1865823 INFO  (qtp2125443004-12709) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index [CachedDir<<refCount=0;path=/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index;done=true>>]
   [junit4]   2> 1865823 INFO  (qtp2125443004-12709) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_45F43F257D825341-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 1865825 INFO  (qtp2125443004-12712) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=name:empty2&sort=id+desc&wt=javabin&version=2} hits=1 status=0 QTime=0
   [junit4]   2> 1865825 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[45F43F257D825341]) [    ] o.a.s.h.TestReplicationHandler Waited for 0ms and found 1 docs
   [junit4]   2> 1865827 INFO  (qtp1300959840-12692) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1865830 INFO  (qtp1300959840-12691) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1865834 INFO  (qtp1300959840-12694) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 1865835 INFO  (qtp2125443004-12714) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=2
   [junit4]   2> 1865836 INFO  (qtp1300959840-12696) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865838 INFO  (qtp2125443004-12716) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865841 INFO  (qtp1300959840-12695) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[3]} 0 0
   [junit4]   2> 1865843 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 1865845 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=3
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_2,generation=2}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_3,generation=3}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@68dd2dcc lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5bd9491),segFN=segments_4,generation=4}
   [junit4]   2> 1865845 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 4
   [junit4]   2> 1865845 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening Searcher@37f7c654[collection1] realtime
   [junit4]   2> 1865845 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 1865845 INFO  (qtp1300959840-12698) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={openSearcher=false&commit=true&wt=javabin&version=2}{commit=} 0 2
   [junit4]   2> 1865847 WARN  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher 'masterUrl' must be specified without the /replication suffix
   [junit4]   2> 1865850 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 1865850 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 4
   [junit4]   2> 1865850 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 3
   [junit4]   2> 1865850 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 1865852 WARN  (qtp1300959840-12697) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _0_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865852 WARN  (qtp1300959840-12697) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _1_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865852 WARN  (qtp1300959840-12697) [    x:collection1] o.a.s.h.ReplicationHandler Could not read checksum from index file: _2_MockRandom_0.sd
   [junit4]   2> java.lang.ArrayIndexOutOfBoundsException
   [junit4]   2> 1865853 INFO  (qtp1300959840-12697) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=4&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=1
   [junit4]   2> 1865853 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 30
   [junit4]   2> 1865853 INFO  (explicit-fetchindex-cmd) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build/solr-

[...truncated too long message...]

ing.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]   2> Dec 15, 2015 3:47:25 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=en_SG, timezone=America/Guadeloupe
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_66 (64-bit)/cpus=3,threads=2,free=154661776,total=407138304
   [junit4]   2> NOTE: All tests run in this JVM: [TestDefaultStatsCache, SegmentsInfoRequestHandlerTest, SuggestComponentTest, TestBlendedInfixSuggestions, RequestHandlersTest, CachingDirectoryFactoryTest, HighlighterConfigTest, DistributedQueueTest, BufferStoreTest, DistribDocExpirationUpdateProcessorTest, SpatialHeatmapFacetsTest, FileBasedSpellCheckerTest, TestFastWriter, SortByFunctionTest, SimpleMLTQParserTest, TestCollationField, TestNonDefinedSimilarityFactory, TestConfigSetProperties, TestCustomSort, OutputWriterTest, TestDistribDocBasedVersion, TestJoin, SearchHandlerTest, FullSolrCloudDistribCmdsTest, HighlighterMaxOffsetTest, SpellCheckCollatorTest, TestScoreJoinQPNoScore, TestSolrQueryResponse, TermVectorComponentDistributedTest, TermVectorComponentTest, TestRandomFaceting, TestRTimerTree, SpellCheckComponentTest, TestSolrQueryParser, TestFoldingMultitermQuery, TestLRUStatsCache, CopyFieldTest, TestJsonFacets, TestPerFieldSimilarityWithDefaultOverride, SolrInfoMBeanTest, FullHLLTest, StatsComponentTest, HdfsCollectionsAPIDistributedZkTest, TestFiltering, DirectSolrConnectionTest, SSLMigrationTest, TestUtils, TestReRankQParserPlugin, TestLMJelinekMercerSimilarityFactory, TestNRTOpen, CollectionsAPIDistributedZkTest, LeaderFailoverAfterPartitionTest, TestRawResponseWriter, TestReload, RequestLoggingTest, BinaryUpdateRequestHandlerTest, TestRestManager, DistributedIntervalFacetingTest, MultiTermTest, TestValueSourceCache, TestCrossCoreJoin, ResponseLogComponentTest, SolrIndexSplitterTest, TestConfigSetsAPI, CurrencyFieldOpenExchangeTest, RestartWhileUpdatingTest, TestSQLHandler, AtomicUpdatesTest, TestInfoStreamLogging, LeaderInitiatedRecoveryOnShardRestartTest, TestIndexingPerformance, RuleEngineTest, HdfsChaosMonkeyNothingIsSafeTest, TestSolrQueryParserDefaultOperatorResource, CollectionsAPIAsyncDistributedZkTest, TestMiniSolrCloudClusterBase, InfoHandlerTest, TestSolrCoreProperties, TestRebalanceLeaders, NotRequiredUniqueKeyTest, ZkStateReaderTest, TestQueryUtils, TestSimpleTrackingShardHandler, OverseerRolesTest, SolrCmdDistributorTest, TestSchemaResource, NoCacheHeaderTest, EchoParamsTest, DeleteShardTest, SolrCloudExampleTest, AsyncMigrateRouteKeyTest, TestChildDocTransformer, OverriddenZkACLAndCredentialsProvidersTest, TestDefaultSearchFieldResource, HdfsSyncSliceTest, OverseerTaskQueueTest, TestRTGBase, TestShardHandlerFactory, QueryParsingTest, BJQParserTest, HdfsUnloadDistributedZkTest, TestSystemIdResolver, BlockCacheTest, TestStressRecovery, CacheHeaderTest, CdcrReplicationDistributedZkTest, TestPHPSerializedResponseWriter, TestCharFilters, ShowFileRequestHandlerTest, TestClassNameShortening, TestRandomMergePolicy, TestAuthenticationFramework, TestRequestForwarding, TestManagedResource, TestConfigReload, HdfsLockFactoryTest, TestExceedMaxTermLength, SuggesterTest, PKIAuthenticationIntegrationTest, DistanceUnitsTest, SpatialRPTFieldTypeTest, DistributedExpandComponentTest, ShardSplitTest, SpellingQueryConverterTest, ParsingFieldUpdateProcessorsTest, TestReversedWildcardFilterFactory, RemoteQueryErrorTest, TestStressUserVersions, TestPKIAuthenticationPlugin, AnalysisErrorHandlingTest, TestExpandComponent, TestReqParamsAPI, TestMiniSolrCloudClusterSSL, DistanceFunctionTest, DistributedSuggestComponentTest, DataDrivenBlockJoinTest, DistributedMLTComponentTest, TestShortCircuitedRequests, TestFuzzyAnalyzedSuggestions, ShardRoutingCustomTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, BasicZkTest, TestReplicationHandler, ClusterStateUpdateTest, ZkSolrClientTest, TestZkChroot, ZkCLITest, AutoCommitTest, TestDistributedGrouping, TestHashPartitioner, TestRangeQuery, TestCoreContainer, TestSolr4Spatial, TestGroupingSearch, QueryElevationComponentTest, BasicFunctionalityTest, DirectUpdateHandlerTest, TestIndexSearcher, TestCoreDiscovery]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCoreDiscovery -Dtests.seed=45F43F257D825341 -Dtests.slow=true -Dtests.locale=en_SG -Dtests.timezone=America/Guadeloupe -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestCoreDiscovery (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ERROR: SolrIndexSearcher opens=27 closes=26
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([45F43F257D825341]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:453)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:225)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestCoreDiscovery: 
   [junit4]    >    1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([45F43F257D825341]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=14668, name=searcherExecutor-6412-thread-1, state=WAITING, group=TGRP-TestCoreDiscovery]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([45F43F257D825341]:0)
   [junit4] Completed [404/566 (2!)] on J1 in 145.86s, 12 tests, 1 failure, 2 errors <<< FAILURES!

[...truncated 502 lines...]
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/build.xml:784: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/build.xml:728: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/build.xml:59: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/build.xml:233: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/solr/common-build.xml:516: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/lucene/common-build.xml:1436: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-trunk-Solaris/lucene/common-build.xml:991: There were test failures: 566 suites (8 ignored), 2253 tests, 3 suite-level errors, 1 failure, 526 ignored (65 assumptions) [seed: 45F43F257D825341]

Total time: 60 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any