You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2015/01/16 12:39:49 UTC

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b44) - Build # 11615 - Still Failing!

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11615/
Java: 64bit/jdk1.9.0-ea-b44 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.HttpPartitionTest.testDistribSearch

Error Message:
Captured an uncaught exception in thread: Thread[id=9172, name=OverseerStateUpdate-93153374336712716-127.0.0.1:43758_-n_0000000004, state=RUNNABLE, group=Overseer state updater.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=9172, name=OverseerStateUpdate-93153374336712716-127.0.0.1:43758_-n_0000000004, state=RUNNABLE, group=Overseer state updater.]
Caused by: java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([9A8FDA743E8D7463]:0)
	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:534)
	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:258)
	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9692 lines...]
   [junit4] Suite: org.apache.solr.cloud.HttpPartitionTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/init-core-data-001
   [junit4]   2> 1417871 T8730 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 1417873 T8730 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 1417874 T8730 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1417874 T8731 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 1417974 T8730 oasc.ZkTestServer.run start zk server on port:50688
   [junit4]   2> 1417975 T8730 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1417975 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1417977 T8738 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3d13fb06 name:ZooKeeperConnection Watcher:127.0.0.1:50688 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1417978 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1417978 T8730 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1417978 T8730 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 1417980 T8730 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1417981 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1417981 T8741 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9542a54 name:ZooKeeperConnection Watcher:127.0.0.1:50688/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1417982 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1417982 T8730 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1417982 T8730 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 1417984 T8730 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 1417985 T8730 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 1417986 T8730 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 1417986 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1417987 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1417988 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1417989 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 1417990 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1417990 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1417992 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1417993 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1417994 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1417994 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1417995 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1417996 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 1417997 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1417998 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1418000 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1418000 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1418002 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1418002 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1418003 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1418003 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1418005 T8730 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1418005 T8730 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1418253 T8730 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 1418255 T8730 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7174ad95{/,null,AVAILABLE}
   [junit4]   2> 1418255 T8730 oejs.AbstractConnector.doStart Started ServerConnector@643daf2a{HTTP/1.1}{127.0.0.1:57210}
   [junit4]   2> 1418256 T8730 oejs.Server.doStart Started @1419297ms
   [junit4]   2> 1418256 T8730 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@58d25a40
   [junit4]   2> 1418257 T8730 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1418257 T8730 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-002
   [junit4]   2> 1418257 T8730 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-002/'
   [junit4]   2> 1418271 T8730 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-002/solr.xml
   [junit4]   2> 1418278 T8730 oasc.CoreContainer.<init> New CoreContainer 1111087727
   [junit4]   2> 1418278 T8730 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-002/]
   [junit4]   2> 1418278 T8730 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1418279 T8730 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1418279 T8730 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1418279 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1418279 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1418279 T8730 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1418280 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1418280 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1418280 T8730 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1418280 T8730 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1418280 T8730 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 1418281 T8730 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1418281 T8730 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1418281 T8730 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1418282 T8730 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1418282 T8730 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50688/solr
   [junit4]   2> 1418282 T8730 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1418282 T8730 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1418283 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1418284 T8755 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@48055513 name:ZooKeeperConnection Watcher:127.0.0.1:50688 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1418284 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1418284 T8730 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1418285 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1418286 T8758 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@282d06d6 name:ZooKeeperConnection Watcher:127.0.0.1:50688/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1418286 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1418287 T8730 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 1418289 T8730 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 1418290 T8730 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 1418291 T8730 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 1418293 T8730 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 1418294 T8730 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 1418295 T8730 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39412_
   [junit4]   2> 1418295 T8730 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39412_
   [junit4]   2> 1418296 T8730 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 1418297 T8730 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 1418298 T8730 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1418298 T8730 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39412_
   [junit4]   2> 1418299 T8730 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 1418300 T8730 oasc.Overseer.start Overseer (id=93153374336712707-127.0.0.1:39412_-n_0000000000) starting
   [junit4]   2> 1418301 T8730 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 1418304 T8730 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1418304 T8760 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 1418305 T8730 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 1418306 T8730 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 1418306 T8730 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1418308 T8759 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 1418309 T8762 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 1418309 T8762 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1418310 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1418310 T8762 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1418312 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39412",
   [junit4]   2> 	  "node_name":"127.0.0.1:39412_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 1418312 T8759 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39412",
   [junit4]   2> 	  "node_name":"127.0.0.1:39412_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1418312 T8759 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 1418312 T8759 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1418313 T8758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1419311 T8762 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1419311 T8762 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 1419311 T8762 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1419312 T8762 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1419313 T8762 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1419313 T8762 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-002/collection1/'
   [junit4]   2> 1419313 T8762 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 1419313 T8762 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 1419314 T8762 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 1419323 T8762 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1419325 T8762 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1419329 T8762 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1419338 T8762 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1419348 T8762 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1419349 T8762 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1419353 T8762 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1419437 T8762 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1419440 T8762 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1419441 T8762 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1419446 T8762 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1419448 T8762 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1419449 T8762 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1419450 T8762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1419450 T8762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1419450 T8762 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1419451 T8762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1419451 T8762 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1419451 T8762 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1419451 T8762 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1419451 T8762 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/control/data/
   [junit4]   2> 1419452 T8762 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f34c716
   [junit4]   2> 1419452 T8762 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/control/data
   [junit4]   2> 1419452 T8762 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/control/data/index/
   [junit4]   2> 1419452 T8762 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1419452 T8762 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/control/data/index
   [junit4]   2> 1419453 T8762 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1419453 T8762 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4ad8063c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@626624f7),segFN=segments_1,generation=1}
   [junit4]   2> 1419453 T8762 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1419455 T8762 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1419455 T8762 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1419455 T8762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1419455 T8762 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1419455 T8762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1419456 T8762 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1419456 T8762 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1419456 T8762 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1419456 T8762 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1419456 T8762 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1419457 T8762 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1419457 T8762 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1419457 T8762 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1419457 T8762 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1419457 T8762 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1419458 T8762 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1419458 T8762 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1419458 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1419458 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1419459 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1419459 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1419459 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1419459 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1419460 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1419460 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1419460 T8762 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1419460 T8762 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1419462 T8762 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1419462 T8762 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1419463 T8762 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1419463 T8762 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1419464 T8762 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1419464 T8762 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1419465 T8762 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1419465 T8762 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=18.9052734375, floorSegmentMB=1.9033203125, forceMergeDeletesPctAllowed=18.081659211030434, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8116281600196091
   [junit4]   2> 1419465 T8762 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4ad8063c lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@626624f7),segFN=segments_1,generation=1}
   [junit4]   2> 1419465 T8762 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1419466 T8762 oass.SolrIndexSearcher.<init> Opening Searcher@55a6f749[collection1] main
   [junit4]   2> 1419466 T8762 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 1419467 T8762 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1419467 T8762 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1419467 T8762 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1419467 T8762 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1419467 T8762 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1419468 T8762 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1419468 T8762 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1419468 T8762 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1419468 T8762 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1419468 T8762 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1419469 T8763 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@55a6f749[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1419469 T8766 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39412 collection:control_collection shard:shard1
   [junit4]   2> 1419469 T8730 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1419469 T8730 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1419470 T8766 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1419471 T8730 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1419472 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1419472 T8766 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1419472 T8770 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@584a1499 name:ZooKeeperConnection Watcher:127.0.0.1:50688/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1419473 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1419473 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1419473 T8730 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1419473 T8766 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1419473 T8766 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 1419473 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 1419474 T8730 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> ASYNC  NEW_CORE C4643 name=collection1 org.apache.solr.core.SolrCore@49f4794a url=http://127.0.0.1:39412/collection1 node=127.0.0.1:39412_ C4643_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:39412, node_name=127.0.0.1:39412_, state=down}
   [junit4]   2> 1419474 T8766 C4643 P39412 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39412/collection1/
   [junit4]   2> 1419474 T8766 C4643 P39412 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1419474 T8766 C4643 P39412 oasc.SyncStrategy.syncToMe http://127.0.0.1:39412/collection1/ has no replicas
   [junit4]   2> 1419475 T8766 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39412/collection1/ shard1
   [junit4]   2> 1419475 T8766 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1419475 T8730 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1419476 T8730 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 1419476 T8730 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1419476 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1419477 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1419477 T8773 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7aa23e69 name:ZooKeeperConnection Watcher:127.0.0.1:50688/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1419477 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1419477 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39412",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 1419477 T8730 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1419482 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 1
   [junit4]   2> 1419482 T8759 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 1419483 T8758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1419483 T8770 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1419485 T8774 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1419485 T8759 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 1419486 T8770 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1419527 T8766 oasc.ZkController.register We are http://127.0.0.1:39412/collection1/ and leader is http://127.0.0.1:39412/collection1/
   [junit4]   2> 1419528 T8766 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39412
   [junit4]   2> 1419528 T8766 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1419528 T8766 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 1419528 T8766 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1419529 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1419530 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39412",
   [junit4]   2> 	  "node_name":"127.0.0.1:39412_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 1419530 T8759 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39412",
   [junit4]   2> 	  "node_name":"127.0.0.1:39412_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1419632 T8758 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1419632 T8770 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 1419726 T8730 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 1419727 T8730 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 1419729 T8730 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@6f7c9f26{/,null,AVAILABLE}
   [junit4]   2> 1419729 T8730 oejs.AbstractConnector.doStart Started ServerConnector@642e49f1{HTTP/1.1}{127.0.0.1:53458}
   [junit4]   2> 1419729 T8730 oejs.Server.doStart Started @1420771ms
   [junit4]   2> 1419730 T8730 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@58d25a40
   [junit4]   2> 1419730 T8730 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1419730 T8730 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-003
   [junit4]   2> 1419730 T8730 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-003/'
   [junit4]   2> 1419741 T8730 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-003/solr.xml
   [junit4]   2> 1419747 T8730 oasc.CoreContainer.<init> New CoreContainer 1208983481
   [junit4]   2> 1419747 T8730 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-003/]
   [junit4]   2> 1419748 T8730 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1419748 T8730 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1419748 T8730 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1419748 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1419749 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1419749 T8730 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1419749 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1419749 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1419749 T8730 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1419750 T8730 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1419750 T8730 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 1419750 T8730 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1419751 T8730 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1419751 T8730 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1419751 T8730 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1419751 T8730 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50688/solr
   [junit4]   2> 1419751 T8730 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1419752 T8730 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1419752 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1419753 T8788 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54112f7e name:ZooKeeperConnection Watcher:127.0.0.1:50688 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1419753 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1419754 T8730 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1419755 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1419756 T8791 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@36c7b07a name:ZooKeeperConnection Watcher:127.0.0.1:50688/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1419756 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1419758 T8730 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1420761 T8730 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40892_
   [junit4]   2> 1420762 T8730 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40892_
   [junit4]   2> 1420764 T8730 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1420767 T8792 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1420767 T8792 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1420768 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1420768 T8792 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1420768 T8792 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1420768 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40892",
   [junit4]   2> 	  "node_name":"127.0.0.1:40892_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 1420768 T8792 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1420769 T8759 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40892",
   [junit4]   2> 	  "node_name":"127.0.0.1:40892_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1420769 T8759 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 1420769 T8759 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 1420770 T8792 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 1420770 T8792 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1420872 T8759 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 1420873 T8791 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1420874 T8791 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 1421771 T8792 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1421771 T8792 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1421771 T8792 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1421772 T8792 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1421772 T8792 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1421772 T8792 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-003/collection1/'
   [junit4]   2> 1421773 T8792 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 1421774 T8792 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 1421774 T8792 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 1421788 T8792 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1421789 T8792 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1421794 T8792 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1421804 T8792 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1421812 T8792 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1421814 T8792 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1421818 T8792 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1421905 T8792 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1421908 T8792 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1421909 T8792 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1421917 T8792 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1421919 T8792 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1421920 T8792 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1421920 T8792 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1421921 T8792 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1421921 T8792 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1421921 T8792 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1421922 T8792 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1421922 T8792 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1421922 T8792 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1421922 T8792 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty1/
   [junit4]   2> 1421922 T8792 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f34c716
   [junit4]   2> 1421923 T8792 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty1
   [junit4]   2> 1421923 T8792 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty1/index/
   [junit4]   2> 1421923 T8792 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 1421923 T8792 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty1/index
   [junit4]   2> 1421923 T8792 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1421924 T8792 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@118b918a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f5abf08),segFN=segments_1,generation=1}
   [junit4]   2> 1421924 T8792 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1421926 T8792 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1421926 T8792 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1421926 T8792 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1421926 T8792 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1421926 T8792 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1421926 T8792 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1421927 T8792 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1421927 T8792 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1421927 T8792 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1421927 T8792 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1421928 T8792 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1421928 T8792 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1421928 T8792 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1421928 T8792 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1421929 T8792 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1421929 T8792 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1421929 T8792 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1421929 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1421930 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1421930 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1421931 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1421931 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1421931 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1421931 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1421932 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1421932 T8792 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1421932 T8792 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1421934 T8792 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1421934 T8792 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1421935 T8792 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1421935 T8792 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1421936 T8792 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1421936 T8792 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1421936 T8792 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1421937 T8792 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=18.9052734375, floorSegmentMB=1.9033203125, forceMergeDeletesPctAllowed=18.081659211030434, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8116281600196091
   [junit4]   2> 1421937 T8792 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@118b918a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f5abf08),segFN=segments_1,generation=1}
   [junit4]   2> 1421937 T8792 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1421937 T8792 oass.SolrIndexSearcher.<init> Opening Searcher@731241da[collection1] main
   [junit4]   2> 1421937 T8792 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1421938 T8792 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1421938 T8792 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1421939 T8792 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1421939 T8792 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1421939 T8792 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1421939 T8792 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1421939 T8792 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1421940 T8792 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1421940 T8792 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1421940 T8793 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@731241da[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1421940 T8792 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1421941 T8796 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40892 collection:collection1 shard:shard2
   [junit4]   2> 1421941 T8730 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1421941 T8730 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1421942 T8796 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 1421944 T8796 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 1421945 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1421945 T8796 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1421945 T8796 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4644 name=collection1 org.apache.solr.core.SolrCore@108630bc url=http://127.0.0.1:40892/collection1 node=127.0.0.1:40892_ C4644_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:40892, node_name=127.0.0.1:40892_, state=down}
   [junit4]   2> 1421946 T8796 C4644 P40892 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40892/collection1/
   [junit4]   2> 1421946 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 1421946 T8796 C4644 P40892 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1421946 T8796 C4644 P40892 oasc.SyncStrategy.syncToMe http://127.0.0.1:40892/collection1/ has no replicas
   [junit4]   2> 1421946 T8796 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40892/collection1/ shard2
   [junit4]   2> 1421947 T8759 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 1421947 T8796 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 1421947 T8791 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1421947 T8791 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 1421949 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1421950 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40892",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 1421950 T8759 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
   [junit4]   2> 1421950 T8791 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1421951 T8791 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1422000 T8796 oasc.ZkController.register We are http://127.0.0.1:40892/collection1/ and leader is http://127.0.0.1:40892/collection1/
   [junit4]   2> 1422000 T8796 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40892
   [junit4]   2> 1422000 T8796 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 1422000 T8796 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 1422000 T8796 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1422001 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1422002 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40892",
   [junit4]   2> 	  "node_name":"127.0.0.1:40892_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 1422002 T8759 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:40892",
   [junit4]   2> 	  "node_name":"127.0.0.1:40892_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1422003 T8759 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
   [junit4]   2> 1422003 T8796 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 1422003 T8791 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1422004 T8791 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1422188 T8730 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 1422188 T8730 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 1422190 T8730 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@28803350{/,null,AVAILABLE}
   [junit4]   2> 1422190 T8730 oejs.AbstractConnector.doStart Started ServerConnector@20be93e8{HTTP/1.1}{127.0.0.1:35351}
   [junit4]   2> 1422191 T8730 oejs.Server.doStart Started @1423232ms
   [junit4]   2> 1422191 T8730 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@58d25a40
   [junit4]   2> 1422191 T8730 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1422192 T8730 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-004
   [junit4]   2> 1422192 T8730 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-004/'
   [junit4]   2> 1422202 T8730 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-004/solr.xml
   [junit4]   2> 1422208 T8730 oasc.CoreContainer.<init> New CoreContainer 1667030443
   [junit4]   2> 1422209 T8730 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-004/]
   [junit4]   2> 1422209 T8730 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 1422209 T8730 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 1422209 T8730 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 1422210 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 1422210 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 1422210 T8730 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 1422210 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 1422211 T8730 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 1422211 T8730 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 1422211 T8730 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 1422211 T8730 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 1422212 T8730 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1422212 T8730 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1422212 T8730 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1422212 T8730 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 1422213 T8730 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:50688/solr
   [junit4]   2> 1422213 T8730 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 1422213 T8730 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 1422214 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1422215 T8811 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c7c5def name:ZooKeeperConnection Watcher:127.0.0.1:50688 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1422215 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1422215 T8730 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 1422216 T8730 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 1422217 T8814 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ac518be name:ZooKeeperConnection Watcher:127.0.0.1:50688/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1422217 T8730 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 1422222 T8730 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 1423226 T8730 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44076_
   [junit4]   2> 1423226 T8730 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44076_
   [junit4]   2> 1423228 T8730 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 1423231 T8815 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 1423231 T8815 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 1423232 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1423232 T8815 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 1423232 T8815 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 1423232 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44076",
   [junit4]   2> 	  "node_name":"127.0.0.1:44076_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 1423232 T8815 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 1423233 T8759 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44076",
   [junit4]   2> 	  "node_name":"127.0.0.1:44076_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 1423233 T8759 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 1423233 T8759 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 1423233 T8815 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 1423233 T8815 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 1423234 T8759 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
   [junit4]   2> 1423235 T8814 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1423235 T8814 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 1423236 T8791 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1423236 T8791 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 1424233 T8815 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1424234 T8815 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 1424234 T8815 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 1424234 T8815 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1424235 T8815 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1424235 T8815 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-004/collection1/'
   [junit4]   2> 1424236 T8815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 1424236 T8815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 1424236 T8815 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest%209A8FDA743E8D7463-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 1424248 T8815 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 1424249 T8815 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 1424264 T8815 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 1424269 T8815 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 1424280 T8815 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1424281 T8815 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 1424286 T8815 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 1424406 T8815 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1424410 T8815 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 1424412 T8815 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 1424418 T8815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1424420 T8815 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 1424421 T8815 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1424422 T8815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1424422 T8815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1424422 T8815 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1424423 T8815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 1424423 T8815 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 1424423 T8815 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1424423 T8815 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1424424 T8815 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty2/
   [junit4]   2> 1424424 T8815 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5f34c716
   [junit4]   2> 1424424 T8815 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty2
   [junit4]   2> 1424424 T8815 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty2/index/
   [junit4]   2> 1424425 T8815 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 1424425 T8815 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.HttpPartitionTest 9A8FDA743E8D7463-001/tempDir-001/jetty2/index
   [junit4]   2> 1424425 T8815 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=23, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 1424426 T8815 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@75af6d25 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@41cc7257),segFN=segments_1,generation=1}
   [junit4]   2> 1424426 T8815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1424428 T8815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1424428 T8815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1424428 T8815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1424429 T8815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1424429 T8815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1424429 T8815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1424429 T8815 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1424429 T8815 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1424430 T8815 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1424430 T8815 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1424430 T8815 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1424431 T8815 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1424431 T8815 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 1424431 T8815 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 1424431 T8815 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 1424432 T8815 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 1424432 T8815 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 1424432 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 1424433 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 1424434 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 1424434 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 1424434 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 1424435 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 1424435 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 1424435 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 1424436 T8815 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 1424436 T8815 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 1424438 T8815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1424438 T8815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1424439 T8815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1424440 T8815 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 1424440 T8815 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1424441 T8815 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 1424441 T8815 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 1424441 T8815 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=43, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=18.9052734375, floorSegmentMB=1.9033203125, forceMergeDeletesPctAllowed=18.081659211030434, segmentsPerTier=15.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8116281600196091
   [junit4]   2> 1424442 T8815 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@75af6d25 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@41cc7257),segFN=segments_1,generation=1}
   [junit4]   2> 1424442 T8815 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 1424442 T8815 oass.SolrIndexSearcher.<init> Opening Searcher@5ac0312f[collection1] main
   [junit4]   2> 1424442 T8815 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 1424443 T8815 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 1424443 T8815 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1424443 T8815 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1424443 T8815 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 1424444 T8815 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1424444 T8815 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1424444 T8815 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1424444 T8815 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 1424444 T8815 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 1424445 T8816 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5ac0312f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1424445 T8815 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 1424446 T8819 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44076 collection:collection1 shard:shard1
   [junit4]   2> 1424446 T8730 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 1424446 T8730 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 1424446 T8819 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 1424449 T8819 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 1424450 T8758 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 1424450 T8819 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 1424450 T8819 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C4645 name=collection1 org.apache.solr.core.SolrCore@5fecee81 url=http://127.0.0.1:44076/collection1 node=127.0.0.1:44076_ C4645_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:44076, node_name=127.0.0.1:44076_, state=down}
   [junit4]   2> 1424450 T8819 C4645 P44076 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44076/collection1/
   [junit4]   2> 1424450 T8819 C4645 P44076 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 1424451 T8819 C4645 P44076 oasc.SyncStrategy.syncToMe http://127.0.0.1:44076/collection1/ has no replicas
   [junit4]   2> 1424451 T8819 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44076/collection1/ shard1
   [junit4]   2> 1424451 T8819 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 1424451 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 1424452 T8759 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 5
   [junit4]   2> 1424452 T8791 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1424453 T8814 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 1424453 T8791 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 1424453 T8814 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 6 
   [junit4]   2> 1424459 T8759 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44076",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 1424460 T8759 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json 

[...truncated too long message...]

FDA743E8D7463]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:534)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:258)
   [junit4]    > 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #31: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=9292, name=OverseerStateUpdate-93153374336712716-127.0.0.1:43758_-n_0000000034, state=RUNNABLE, group=Overseer state updater.]
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9A8FDA743E8D7463]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:534)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:258)
   [junit4]    > 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #32: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=9296, name=OverseerStateUpdate-93153374336712716-127.0.0.1:43758_-n_0000000035, state=RUNNABLE, group=Overseer state updater.]
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9A8FDA743E8D7463]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:534)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:258)
   [junit4]    > 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #33: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=9300, name=OverseerStateUpdate-93153374336712716-127.0.0.1:43758_-n_0000000036, state=RUNNABLE, group=Overseer state updater.]
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9A8FDA743E8D7463]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:534)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:258)
   [junit4]    > 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #34: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=9305, name=OverseerStateUpdate-93153374336712716-127.0.0.1:43758_-n_0000000037, state=RUNNABLE, group=Overseer state updater.]
   [junit4]    > Caused by: java.lang.AssertionError
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9A8FDA743E8D7463]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:534)
   [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.updateClusterState(ZkStateReader.java:258)
   [junit4]    > 	at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:233)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1523246 T8730 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {_version_=PostingsFormat(name=Memory doPackFST= false), multiDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_t=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=LuceneVarGapFixedInterval), text=PostingsFormat(name=Direct), range_facet_l=PostingsFormat(name=Memory doPackFST= false), timestamp=PostingsFormat(name=Memory doPackFST= false)}, docValues:{timestamp=DocValuesFormat(name=Asserting)}, sim=RandomSimilarityProvider(queryNorm=true,coord=yes): {}, locale=en_US, timezone=Europe/Warsaw
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.9.0-ea (64-bit)/cpus=8,threads=1,free=97142696,total=259731456
   [junit4]   2> NOTE: All tests run in this JVM: [TestLeaderElectionZkExpiry, RemoteQueryErrorTest, AnalysisAfterCoreReloadTest, TestInitParams, TestDFRSimilarityFactory, MinimalSchemaTest, TestIBSimilarityFactory, SolrIndexSplitterTest, DOMUtilTest, AddBlockUpdateTest, PreAnalyzedFieldTest, TestComplexPhraseQParserPlugin, OpenExchangeRatesOrgProviderTest, TestSearcherReuse, SimpleCollectionCreateDeleteTest, ZkControllerTest, TestNonDefinedSimilarityFactory, TestSimpleQParserPlugin, TestCoreContainer, SuggesterWFSTTest, TestInfoStreamLogging, RequestHandlersTest, TestUpdate, TestSolrConfigHandlerCloud, TestFastOutputStream, DefaultValueUpdateProcessorTest, TestSolrQueryParserDefaultOperatorResource, TestSerializedLuceneMatchVersion, TestQuerySenderListener, TestCodecSupport, LegacyHTMLStripCharFilterTest, TestShortCircuitedRequests, TestQuerySenderNoQuery, HdfsChaosMonkeySafeLeaderTest, DocExpirationUpdateProcessorFactoryTest, NotRequiredUniqueKeyTest, QueryElevationComponentTest, TestSortingResponseWriter, TestObjectReleaseTracker, TestSolrQueryParserResource, CoreContainerCoreInitFailuresTest, SolrPluginUtilsTest, HdfsRecoveryZkTest, TestArbitraryIndexDir, FieldAnalysisRequestHandlerTest, TestSurroundQueryParser, ExitableDirectoryReaderTest, TestStressUserVersions, AsyncMigrateRouteKeyTest, SolrTestCaseJ4Test, HdfsSyncSliceTest, NoCacheHeaderTest, HdfsDirectoryFactoryTest, TestStressReorder, HardAutoCommitTest, SyncSliceTest, TestInitQParser, TestSchemaVersionResource, TestFaceting, DeleteInactiveReplicaTest, DistanceFunctionTest, TestOmitPositions, TestLMDirichletSimilarityFactory, DirectUpdateHandlerTest, CoreAdminCreateDiscoverTest, TestFastWriter, TestSweetSpotSimilarityFactory, TestFieldTypeResource, DistributedFacetPivotSmallTest, DistributedFacetPivotLongTailTest, JSONWriterTest, TestBinaryResponseWriter, ClusterStateTest, CloudExitableDirectoryReaderTest, AlternateDirectoryTest, TestClassNameShortening, OverseerTest, BasicDistributedZkTest, SolrCoreTest, MultiTermTest, TestStressRecovery, DistributedDebugComponentTest, TestExtendedDismaxParser, TestCollapseQParserPlugin, TestPartialUpdateDeduplication, TestUniqueKeyFieldResource, TestImplicitCoreProperties, TestStressLucene, CircularListTest, CacheHeaderTest, TestSchemaSimilarityResource, TestCustomSort, ShardRoutingTest, TestLazyCores, ZkCLITest, TestGroupingSearch, HdfsWriteToMultipleCollectionsTest, DistributedFacetPivotSmallAdvancedTest, TestAnalyzedSuggestions, SliceStateTest, DistributedSpellCheckComponentTest, TestCursorMarkWithoutUniqueKey, DocValuesMissingTest, TestCollationField, TestClusterStateMutator, LeaderElectionTest, HdfsUnloadDistributedZkTest, DateMathParserTest, TestMissingGroups, RecoveryZkTest, TestManagedResourceStorage, IndexSchemaRuntimeFieldTest, BlockCacheTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, FullSolrCloudDistribCmdsTest, TestReplicationHandler, ClusterStateUpdateTest, TestRandomFaceting, TestRandomDVFaceting, AutoCommitTest, TestRealTimeGet, TestReload, TestStressVersions, PeerSyncTest, BadIndexSchemaTest, TestFunctionQuery, TestIndexSearcher, SoftAutoCommitTest, SuggesterFSTTest, SolrRequestParserTest, DocValuesTest, SuggesterTest, SpatialFilterTest, TestCSVLoader, SolrCoreCheckLockOnStartupTest, FieldMutatingUpdateProcessorTest, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, SortByFunctionTest, DocValuesMultiTest, LukeRequestHandlerTest, TestQueryUtils, TestWriterPerf, TestQueryTypes, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, RequiredFieldsTest, RegexBoostProcessorTest, TestCSVResponseWriter, UniqFieldsUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, PingRequestHandlerTest, TestComponentsName, HighlighterConfigTest, TestSolrIndexConfig, ResponseLogComponentTest, SolrIndexConfigTest, TestMergePolicyConfig, TestBinaryField, TestConfig, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, TestPhraseSuggestions, TestXIncludeConfig, TestJmxMonitoredMap, TimeZoneUtilsTest, PluginInfoTest, ChaosMonkeyNothingIsSafeTest, TestDistributedMissingSort, TestHighlightDedupGrouping, TestTolerantSearch, AliasIntegrationTest, ConnectionManagerTest, CustomCollectionTest, HttpPartitionTest]
   [junit4] Completed on J0 in 105.72s, 1 test, 1 error <<< FAILURES!

[...truncated 289 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 463 suites, 1871 tests, 1 error, 47 ignored (21 assumptions)

Total time: 62 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.9.0-ea-b44 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b20) - Build # 11628 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11628/
Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA]:0)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.testDistribSearch

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA]:0)


FAILED:  org.apache.solr.cloud.OverseerStatusTest.testDistribSearch

Error Message:
Error from server at http://127.0.0.1:51111: reload the collection time out:180s

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51111: reload the collection time out:180s
	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA:8E16D25740B1D086]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.invokeCollectionApi(AbstractFullDistribZkTestBase.java:1870)
	at org.apache.solr.cloud.OverseerStatusTest.doTest(OverseerStatusTest.java:103)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9115 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerStatusTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/init-core-data-001
   [junit4]   2> 667781 T2952 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 667783 T2952 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 667783 T2952 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 667784 T2953 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 667884 T2952 oasc.ZkTestServer.run start zk server on port:39574
   [junit4]   2> 667884 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 667885 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 667890 T2960 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ed65f1c name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 667890 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 667890 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 667891 T2952 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 667893 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 667894 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 667898 T2963 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@411d26bb name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 667899 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 667899 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 667899 T2952 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 667901 T2952 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 667902 T2952 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 667903 T2952 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 667904 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 667904 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 667906 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 667906 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 667907 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 667908 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 667909 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 667909 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 667910 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 667911 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 667912 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 667912 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 667913 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 667914 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 667915 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 667915 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 667916 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 667916 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 667918 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 667918 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 667920 T2952 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 667920 T2952 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 668259 T2952 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 668260 T2952 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@259cd35a{/,null,AVAILABLE}
   [junit4]   2> 668269 T2952 oejs.AbstractConnector.doStart Started ServerConnector@6fa1c9b7{HTTP/1.1}{127.0.0.1:39621}
   [junit4]   2> 668270 T2952 oejs.Server.doStart Started @669634ms
   [junit4]   2> 668270 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 668270 T2952 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 668271 T2952 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002
   [junit4]   2> 668271 T2952 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/'
   [junit4]   2> 668286 T2952 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/solr.xml
   [junit4]   2> 668298 T2952 oasc.CoreContainer.<init> New CoreContainer 648521723
   [junit4]   2> 668298 T2952 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/]
   [junit4]   2> 668299 T2952 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 668299 T2952 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 668300 T2952 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 668300 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 668300 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 668301 T2952 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 668302 T2952 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 668302 T2952 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 668303 T2952 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 668303 T2952 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 668303 T2952 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 668304 T2952 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 668304 T2952 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39574/solr
   [junit4]   2> 668304 T2952 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 668305 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 668307 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 668308 T2977 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2e23bd52 name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 668309 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 668310 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 668312 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 668313 T2980 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30765cf0 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 668313 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 668315 T2952 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 668317 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 668318 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 668320 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 668322 T2952 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 668323 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 668324 T2952 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39621_
   [junit4]   2> 668325 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39621_
   [junit4]   2> 668327 T2952 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 668328 T2952 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 668329 T2952 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 668330 T2952 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:39621_
   [junit4]   2> 668330 T2952 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 668331 T2952 oasc.Overseer.start Overseer (id=93160157159030787-127.0.0.1:39621_-n_0000000000) starting
   [junit4]   2> 668333 T2952 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 668337 T2952 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 668337 T2982 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 668338 T2952 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 668340 T2952 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 668341 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 668345 T2981 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 668349 T2984 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 668349 T2984 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 668350 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 668350 T2984 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 668352 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 668352 T2981 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 668352 T2981 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 668353 T2981 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 668354 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669350 T2984 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 669351 T2984 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 669351 T2984 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 669351 T2984 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 669352 T2984 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 669352 T2984 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/collection1/'
   [junit4]   2> 669353 T2984 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 669353 T2984 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 669353 T2984 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 669365 T2984 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 669366 T2984 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 669370 T2984 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 669384 T2984 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 669393 T2984 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 669394 T2984 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 669398 T2984 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 669493 T2984 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 669494 T2984 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 669495 T2984 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 669497 T2984 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 669507 T2984 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 669507 T2984 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 669507 T2984 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/
   [junit4]   2> 669508 T2984 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3334a526
   [junit4]   2> 669508 T2984 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data
   [junit4]   2> 669508 T2984 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/index/
   [junit4]   2> 669508 T2984 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 669508 T2984 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/control/data/index
   [junit4]   2> 669509 T2984 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42767485660393223]
   [junit4]   2> 669510 T2984 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10eaef17 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@672875cd),segFN=segments_1,generation=1}
   [junit4]   2> 669510 T2984 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 669511 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 669511 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 669512 T2984 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 669513 T2984 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669513 T2984 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 669514 T2984 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 669515 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 669516 T2984 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 669517 T2984 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 669525 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669526 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669527 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669527 T2984 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 669528 T2984 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 669529 T2984 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 669529 T2984 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 669529 T2984 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7196545575110694]
   [junit4]   2> 669530 T2984 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@10eaef17 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@672875cd),segFN=segments_1,generation=1}
   [junit4]   2> 669530 T2984 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 669530 T2984 oass.SolrIndexSearcher.<init> Opening Searcher@376810cc[collection1] main
   [junit4]   2> 669530 T2984 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 669531 T2984 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 669531 T2984 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 669532 T2984 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 669532 T2984 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 669532 T2984 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 669533 T2984 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 669533 T2984 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 669533 T2984 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 669533 T2984 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 669534 T2984 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 669534 T2985 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@376810cc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 669534 T2988 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39621 collection:control_collection shard:shard1
   [junit4]   2> 669534 T2952 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 669535 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 669535 T2988 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 669535 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 669536 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669538 T2991 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@635190b8 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669538 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669538 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 669539 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 669539 T2988 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 669540 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669541 T2988 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 669541 T2988 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2990 name=collection1 org.apache.solr.core.SolrCore@2b4d1ed8 url=http://127.0.0.1:39621/collection1 node=127.0.0.1:39621_ C2990_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:39621, node_name=127.0.0.1:39621_, state=down}
   [junit4]   2> 669541 T2988 C2990 P39621 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39621/collection1/
   [junit4]   2> 669541 T2988 C2990 P39621 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 669541 T2988 C2990 P39621 oasc.SyncStrategy.syncToMe http://127.0.0.1:39621/collection1/ has no replicas
   [junit4]   2> 669541 T2988 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39621/collection1/ shard1
   [junit4]   2> 669541 T2988 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 669542 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 669542 T2952 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 669542 T2952 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 669542 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 669543 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669544 T2994 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@329c1998 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669545 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669545 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 669546 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669546 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 669548 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 1
   [junit4]   2> 669548 T2981 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 669549 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669549 T2991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669554 T2981 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 669554 T2995 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669555 T2991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669596 T2988 oasc.ZkController.register We are http://127.0.0.1:39621/collection1/ and leader is http://127.0.0.1:39621/collection1/
   [junit4]   2> 669597 T2988 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39621
   [junit4]   2> 669597 T2988 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 669597 T2988 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 669597 T2988 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 669598 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 669599 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 669600 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39621",
   [junit4]   2> 	  "node_name":"127.0.0.1:39621_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 669702 T2980 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669702 T2991 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 669791 T2952 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 669792 T2952 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 669793 T2952 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@73b406e4{/,null,AVAILABLE}
   [junit4]   2> 669794 T2952 oejs.AbstractConnector.doStart Started ServerConnector@59727343{HTTP/1.1}{127.0.0.1:45877}
   [junit4]   2> 669794 T2952 oejs.Server.doStart Started @671158ms
   [junit4]   2> 669794 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 669795 T2952 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 669795 T2952 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003
   [junit4]   2> 669795 T2952 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/'
   [junit4]   2> 669806 T2952 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/solr.xml
   [junit4]   2> 669815 T2952 oasc.CoreContainer.<init> New CoreContainer 1588537754
   [junit4]   2> 669815 T2952 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/]
   [junit4]   2> 669815 T2952 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 669816 T2952 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 669817 T2952 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 669818 T2952 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 669818 T2952 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 669818 T2952 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 669818 T2952 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 669819 T2952 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39574/solr
   [junit4]   2> 669819 T2952 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 669819 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 669820 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669822 T3009 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8e77fc name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669822 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669823 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 669826 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 669827 T3012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6ada0a70 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 669827 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 669834 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 670838 T2952 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45877_
   [junit4]   2> 670840 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45877_
   [junit4]   2> 670845 T2952 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 670853 T3013 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 670853 T3013 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 670855 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 670856 T3013 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 670857 T3013 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 670857 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 670857 T3013 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 670858 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 670858 T2981 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 670858 T2981 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 670859 T3013 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 670859 T3013 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 670962 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 670963 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 670964 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 671860 T3013 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 671860 T3013 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 671861 T3013 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 671861 T3013 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 671863 T3013 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 671863 T3013 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/collection1/'
   [junit4]   2> 671864 T3013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 671865 T3013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 671865 T3013 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 671884 T3013 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 671887 T3013 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 671891 T3013 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 671909 T3013 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 671924 T3013 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 671926 T3013 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 671929 T3013 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 672047 T3013 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 672049 T3013 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 672049 T3013 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 672052 T3013 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 672061 T3013 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 672061 T3013 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 672061 T3013 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/
   [junit4]   2> 672061 T3013 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3334a526
   [junit4]   2> 672062 T3013 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1
   [junit4]   2> 672062 T3013 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/index/
   [junit4]   2> 672062 T3013 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 672063 T3013 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty1/index
   [junit4]   2> 672063 T3013 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42767485660393223]
   [junit4]   2> 672063 T3013 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@24384936 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@270bb23e),segFN=segments_1,generation=1}
   [junit4]   2> 672064 T3013 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 672066 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 672067 T3013 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 672067 T3013 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 672067 T3013 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 672067 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 672068 T3013 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 672069 T3013 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 672069 T3013 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 672070 T3013 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 672070 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 672070 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 672071 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 672071 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 672071 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 672072 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 672072 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 672073 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 672073 T3013 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 672074 T3013 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 672076 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672076 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672077 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672078 T3013 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 672079 T3013 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 672080 T3013 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 672080 T3013 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 672081 T3013 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7196545575110694]
   [junit4]   2> 672082 T3013 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@24384936 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@270bb23e),segFN=segments_1,generation=1}
   [junit4]   2> 672082 T3013 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 672082 T3013 oass.SolrIndexSearcher.<init> Opening Searcher@2110a1d2[collection1] main
   [junit4]   2> 672082 T3013 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 672084 T3013 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 672084 T3013 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 672084 T3013 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 672085 T3013 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 672085 T3013 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 672085 T3013 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 672086 T3013 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 672086 T3013 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 672086 T3013 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 672094 T3014 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2110a1d2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 672095 T3013 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 672096 T3017 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45877 collection:collection1 shard:shard2
   [junit4]   2> 672096 T2952 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 672097 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 672098 T3017 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 672101 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 672102 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 672102 T3017 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 672103 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C2991 name=collection1 org.apache.solr.core.SolrCore@3d512af1 url=http://127.0.0.1:45877/collection1 node=127.0.0.1:45877_ C2991_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:45877, node_name=127.0.0.1:45877_, state=down}
   [junit4]   2> 672103 T3017 C2991 P45877 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45877/collection1/
   [junit4]   2> 672103 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> 672103 T3017 C2991 P45877 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 672103 T3017 C2991 P45877 oasc.SyncStrategy.syncToMe http://127.0.0.1:45877/collection1/ has no replicas
   [junit4]   2> 672103 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45877/collection1/ shard2
   [junit4]   2> 672104 T3017 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 672104 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 672104 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 672105 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 672106 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 672106 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 672107 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
   [junit4]   2> 672107 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 672108 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 672156 T3017 oasc.ZkController.register We are http://127.0.0.1:45877/collection1/ and leader is http://127.0.0.1:45877/collection1/
   [junit4]   2> 672157 T3017 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45877
   [junit4]   2> 672157 T3017 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 672157 T3017 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 672157 T3017 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 672158 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 672159 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 672159 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:45877",
   [junit4]   2> 	  "node_name":"127.0.0.1:45877_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 672160 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
   [junit4]   2> 672160 T3017 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 672160 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 672161 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 672333 T2952 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 672334 T2952 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 672336 T2952 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@14316aed{/,null,AVAILABLE}
   [junit4]   2> 672336 T2952 oejs.AbstractConnector.doStart Started ServerConnector@79500b40{HTTP/1.1}{127.0.0.1:51111}
   [junit4]   2> 672336 T2952 oejs.Server.doStart Started @673701ms
   [junit4]   2> 672337 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 672337 T2952 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 672337 T2952 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004
   [junit4]   2> 672337 T2952 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/'
   [junit4]   2> 672348 T2952 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/solr.xml
   [junit4]   2> 672357 T2952 oasc.CoreContainer.<init> New CoreContainer 316805325
   [junit4]   2> 672357 T2952 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/]
   [junit4]   2> 672358 T2952 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 672358 T2952 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 672358 T2952 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 672359 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 672359 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 672359 T2952 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 672360 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 672360 T2952 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 672360 T2952 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 672361 T2952 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 672361 T2952 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 672361 T2952 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 672362 T2952 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 672362 T2952 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 672362 T2952 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 672363 T2952 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39574/solr
   [junit4]   2> 672363 T2952 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 672363 T2952 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 672364 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672366 T3031 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@33f998cb name:ZooKeeperConnection Watcher:127.0.0.1:39574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672366 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672367 T2952 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 672369 T2952 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 672371 T3034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7ff885a4 name:ZooKeeperConnection Watcher:127.0.0.1:39574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 672371 T2952 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 672375 T2952 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 673379 T2952 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51111_
   [junit4]   2> 673380 T2952 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:51111_
   [junit4]   2> 673384 T2952 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 673392 T3035 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 673392 T3035 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 673394 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 673394 T3035 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 673395 T3035 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 673395 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51111",
   [junit4]   2> 	  "node_name":"127.0.0.1:51111_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 673395 T3035 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 673395 T2981 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:51111",
   [junit4]   2> 	  "node_name":"127.0.0.1:51111_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 673396 T2981 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 673396 T2981 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 673396 T3035 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 673396 T3035 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 673397 T2981 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
   [junit4]   2> 673398 T3012 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 673398 T3034 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 673398 T3012 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 673399 T3034 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 674397 T3035 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 674397 T3035 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 674398 T3035 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 674398 T3035 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 674400 T3035 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 674400 T3035 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/collection1/'
   [junit4]   2> 674401 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 674402 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 674402 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest%20FF05C4F37EEB0BA-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 674421 T3035 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 674424 T3035 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 674430 T3035 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 674450 T3035 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 674465 T3035 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 674466 T3035 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 674469 T3035 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 674564 T3035 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 674565 T3035 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 674566 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 674568 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 674579 T3035 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 674579 T3035 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 674579 T3035 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/
   [junit4]   2> 674579 T3035 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3334a526
   [junit4]   2> 674580 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2
   [junit4]   2> 674580 T3035 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/index/
   [junit4]   2> 674581 T3035 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 674581 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerStatusTest FF05C4F37EEB0BA-001/tempDir-001/jetty2/index
   [junit4]   2> 674581 T3035 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.42767485660393223]
   [junit4]   2> 674582 T3035 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3bfc58a2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7dea9369),segFN=segments_1,generation=1}
   [junit4]   2> 674582 T3035 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 674584 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 674585 T3035 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 674585 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 674585 T3035 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 674585 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 674586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 674587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 674587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 674588 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 674589 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 674589 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 674590 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 674590 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 674591 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 674591 T3035 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 674592 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 674594 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674596 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674597 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674598 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 674599 T3035 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 674601 T3035 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 674601 T3035 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 674602 T3035 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7196545575110694]
   [junit4]   2> 674603 T3035 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@3bfc58a2 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7dea9369),segFN=segments_1,generation=1}
   [junit4]   2> 674603 T3035 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 674604 T3035 oass.SolrIndexSearcher.<init> Opening Searcher@6dfc8efa[collection1] main
   [junit4]   2> 674604 T3035 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 674605 T3035 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 674606 T3035 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 674606 T3035 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 674606 T3035 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 674606 T3035 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 674607 T3035 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 674607 T3035 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 674608 T3035 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 674608 T3035 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 674608 T3036 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6dfc8efa[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 674609 T3035 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 674610 T3039 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:51111 collection:collection1 shard:shard1
   [junit4]   2> 674610 T2952 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 674610 T2952 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 674611 T3039 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 674614 T3039 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 674615 T2980 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 674615 T3039 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 674615 T3039 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 674615 T2981 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> ASYNC  NEW_CORE C2992 name=collection1 org.apache.solr.core.SolrCore@26096e79 url=http://127.0.0.1:51111/collection1 node=127.0.0.1:51111_ C2992_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:51111, node_name=127.0.0.1:51111_, state=down}
   [junit4]   2> 674615 T3039 C2992 P51111 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:51111/collection1/
   [junit4]   2> 674615 T3039 C2992 P51111 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 674615 T3039 C2992 P51111 oasc.SyncStrate

[...truncated too long message...]

dPoolExecutor.java:1066)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  21) Thread[id=7536, name=zkCallback-869-thread-115, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]   2> 	        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  22) Thread[id=6789, name=qtp901606055-6789-selector-ServerConnectorManager@6e891376/1, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]   2> 	        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2> 	        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2> 	        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596)
   [junit4]   2> 	        at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
   [junit4]   2> 	        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  23) Thread[id=6797, name=org.eclipse.jetty.server.session.HashSessionManager@57d04011Timer, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 	  24) Thread[id=6816, name=Scheduler-1401580675, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
   [junit4]   2> 	        at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]   2> 	        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2> 	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	        at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {rnd_b=PostingsFormat(name=LuceneFixedGap), _version_=FSTOrd50, a_t=FSTOrd50, a_i=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, sim=DefaultSimilarity, locale=fr_LU, timezone=America/Grenada
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=34,free=61724600,total=285704192
   [junit4]   2> NOTE: All tests run in this JVM: [TestLMDirichletSimilarityFactory, TestOverriddenPrefixQueryForCustomFieldType, TestSchemaSimilarityResource, PathHierarchyTokenizerFactoryTest, SliceStateTest, AlternateDirectoryTest, TestSolrConfigHandlerConcurrent, TestSurroundQueryParser, TestBinaryField, TestSearcherReuse, TestSort, TestShardHandlerFactory, PeerSyncTest, NotRequiredUniqueKeyTest, DeleteInactiveReplicaTest, DistributedFacetPivotWhiteBoxTest, TestXIncludeConfig, TestReRankQParserPlugin, TestHashPartitioner, RankQueryTest, TestStressRecovery, PrimUtilsTest, TestFieldTypeResource, CustomCollectionTest, HighlighterConfigTest, TestRequestStatusCollectionAPI, TestWordDelimiterFilterFactory, TestSolrQueryParserDefaultOperatorResource, RequiredFieldsTest, SolrTestCaseJ4Test, RollingRestartTest, ConnectionManagerTest, TestCollectionAPI, TestSimpleQParserPlugin, TestManagedResourceStorage, QueryElevationComponentTest, DocValuesTest, TestQuerySenderListener, TestFastWriter, CoreAdminCreateDiscoverTest, DocExpirationUpdateProcessorFactoryTest, HdfsRecoveryZkTest, DirectUpdateHandlerTest, ExternalCollectionsTest, RecoveryZkTest, TestSolrConfigHandlerCloud, TestBulkSchemaConcurrent, PreAnalyzedFieldTest, TestStressReorder, OutputWriterTest, TestNonDefinedSimilarityFactory, ShowFileRequestHandlerTest, UnloadDistributedZkTest, TriLevelCompositeIdRoutingTest, CoreAdminHandlerTest, SolrCoreCheckLockOnStartupTest, DateMathParserTest, TestShortCircuitedRequests, TestObjectReleaseTracker, TestRandomMergePolicy, ClusterStateUpdateTest, RemoteQueryErrorTest, HdfsChaosMonkeySafeLeaderTest, TestIndexSearcher, SaslZkACLProviderTest, SortByFunctionTest, TestComponentsName, TestFunctionQuery, SolrXmlInZkTest, LegacyHTMLStripCharFilterTest, TestReplicationHandlerBackup, CircularListTest, DistributedFacetPivotLongTailTest, TestRandomDVFaceting, NumericFieldsTest, LeaderInitiatedRecoveryOnCommitTest, TestCloudPivotFacet, SolrIndexConfigTest, FacetPivotSmallTest, DocValuesMissingTest, TestCSVResponseWriter, TestOmitPositions, DefaultValueUpdateProcessorTest, UniqFieldsUpdateProcessorFactoryTest, SystemInfoHandlerTest, TestPivotHelperCode, SyncSliceTest, UUIDFieldTest, AddBlockUpdateTest, DateRangeFieldTest, LeaderElectionTest, BasicDistributedZkTest, TestStressLucene, TestRestManager, SimpleCollectionCreateDeleteTest, TestSweetSpotSimilarityFactory, TestImplicitCoreProperties, SpatialRPTFieldTypeTest, TestDocumentBuilder, TestCursorMarkWithoutUniqueKey, TestAnalyzeInfixSuggestions, TestLeaderElectionZkExpiry, TestGroupingSearch, CursorPagingTest, ZkCLITest, TestInitParams, DeleteReplicaTest, TestIndexingPerformance, TestMissingGroups, SolrInfoMBeanTest, NoCacheHeaderTest, JSONWriterTest, AssignTest, ChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=FF05C4F37EEB0BA -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=fr_LU -Dtests.timezone=America/Grenada -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | ChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FF05C4F37EEB0BA]:0)
   [junit4] Completed on J0 in 7224.00s, 1 test, 2 errors <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 463 suites, 1871 tests, 1 suite-level error, 2 errors, 47 ignored (21 assumptions)

Total time: 167 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build # 11627 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11627/
Java: 32bit/jdk1.8.0_25 -client -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
GC overhead limit exceeded

Stack Trace:
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([9E0EABB85A598E76:C65DCA9D477C414C]:0)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:280)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)




Build Log:
[...truncated 13425 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001
   [junit4]   2> 19647 T81 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 19673 T81 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 19673 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/'
   [junit4]   2> 19710 T81 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 19722 T81 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 19737 T81 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 19737 T81 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 19745 T81 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 19758 T81 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 19759 T81 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 19759 T81 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 19760 T81 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 19760 T81 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001
   [junit4]   2> 19760 T81 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/'
   [junit4]   2> 19794 T81 oasc.CoreContainer.<init> New CoreContainer 4179699
   [junit4]   2> 19795 T81 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/]
   [junit4]   2> 19795 T81 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 19796 T81 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 19797 T81 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 19798 T81 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 19798 T81 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 19798 T81 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 19799 T81 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 19799 T81 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 19801 T82 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/'
   [junit4]   2> 19827 T82 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 19835 T82 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 19854 T82 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 19855 T82 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 19862 T82 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 19878 T82 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 19879 T82 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 19879 T82 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 19880 T82 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/
   [junit4]   2> 19880 T82 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 19880 T82 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/
   [junit4]   2> 19881 T82 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 19881 T82 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 19882 T82 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 19897 T82 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001
   [junit4]   2> 19898 T82 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index/
   [junit4]   2> 19899 T82 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 19899 T82 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index
   [junit4]   2> 19900 T82 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_1,generation=1}
   [junit4]   2> 19901 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 19902 T82 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 19903 T82 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 19903 T82 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 19903 T82 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 19904 T82 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 19904 T82 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 19904 T82 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 19905 T82 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 19905 T82 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 19906 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 19906 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 19907 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 19907 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 19907 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 19908 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 19908 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 19909 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 19909 T82 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 19909 T82 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 19910 T82 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 19910 T82 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 19910 T82 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 19913 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19914 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19916 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19917 T82 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 19918 T82 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 19919 T82 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 19919 T82 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 19921 T82 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_1,generation=1}
   [junit4]   2> 19921 T82 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 19921 T82 oass.SolrIndexSearcher.<init> Opening Searcher@1415085[collection1] main
   [junit4]   2> 19922 T82 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf
   [junit4]   2> 19922 T82 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf}
   [junit4]   2> 19923 T82 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/core-home-001/collection1/conf
   [junit4]   2> 19923 T82 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 19923 T82 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 19924 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@1415085[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 19924 T82 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 19924 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 19925 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1415085[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 19926 T81 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 19928 T81 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C9 name=collection1 org.apache.solr.core.SolrCore@b6ae2e
   [junit4]   2> 19931 T81 C9 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 19932 T81 C9 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 19932 T81 C9 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 19938 T81 C9 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 19947 T81 C9 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 19947 T81 C9 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 19972 T81 C9 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1badd62 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@192f06c),segFN=segments_2,generation=2}
   [junit4]   2> 19972 T81 C9 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 19982 T81 C9 oass.SolrIndexSearcher.<init> Opening Searcher@1b22fb2[collection1] main
   [junit4]   2> 19983 T81 C9 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 19983 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@1b22fb2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C30)))}
   [junit4]   2> 19986 T83 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=2 
   [junit4]   2> 19987 T83 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 19988 T83 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 19988 T83 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 19989 T83 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b22fb2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C30)))}
   [junit4]   2> 19989 T81 C9 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 19990 T81 C9 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 19990 T81 C9 oashd.DocBuilder.execute Time taken = 0:0:0.52
   [junit4]   2> 19991 T81 C9 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[2, 1, 3],commit=} 0 62
   [junit4]   2> 19994 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=30 status=0 QTime=1 
   [junit4]   2> 20002 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 20004 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=12 status=0 QTime=1 
   [junit4]   2> 20006 T81 C9 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=15 status=0 QTime=1 
   [junit4]   2> 20022 T81 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 20025 T81 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 51284 T81 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=9E0EABB85A598E76 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=in_ID -Dtests.timezone=Europe/Vilnius -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   31.4s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9E0EABB85A598E76:C65DCA9D477C414C]:0)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:280)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 51379 T81 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 51380 T81 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=4179699
   [junit4]   2> 51380 T81 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@b6ae2e
   [junit4]   2> 51380 T81 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 51381 T81 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 51381 T81 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 51381 T81 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 51382 T81 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 51382 T81 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 51382 T81 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 51383 T81 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001/index
   [junit4]   2> 51383 T81 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001;done=false>>]
   [junit4]   2> 51383 T81 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 9E0EABB85A598E76-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=PostingsFormat(name=LuceneFixedGap), _root_=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), desc=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), timestamp=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, sim=DefaultSimilarity, locale=in_ID, timezone=Europe/Vilnius
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=285039384,total=302776320
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestWriterImpl, TestSqlEntityProcessorDelta, TestSqlEntityProcessor, TestDocBuilder2, TestXPathEntityProcessor, TestNumberFormatTransformer, TestPlainTextEntityProcessor, TestLineEntityProcessor, TestURLDataSource, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 31.77s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_141649_716.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid25119.hprof ...
   [junit4] Heap dump file created [478614752 bytes in 7.029 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 68 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -client -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_25) - Build # 11626 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11626/
Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([2E73AA5B93046FDE:7620CB7E8E21A0E4]:0)
	at java.util.HashMap.resize(HashMap.java:703)
	at java.util.HashMap.putVal(HashMap.java:628)
	at java.util.HashMap.put(HashMap.java:611)
	at org.apache.solr.handler.dataimport.AbstractDataImportHandlerTestCase.createMap(AbstractDataImportHandlerTestCase.java:163)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)




Build Log:
[...truncated 13430 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001
   [junit4]   2> 7550 T43 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 7584 T43 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 7585 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/'
   [junit4]   2> 7627 T43 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 7672 T43 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 7690 T43 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 7691 T43 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 7707 T43 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 7778 T43 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 7779 T43 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 7780 T43 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 7781 T43 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 7781 T43 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001
   [junit4]   2> 7781 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/'
   [junit4]   2> 7808 T43 oasc.CoreContainer.<init> New CoreContainer 1665522085
   [junit4]   2> 7808 T43 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/]
   [junit4]   2> 7809 T43 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 7809 T43 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 7809 T43 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 7810 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 7810 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 7810 T43 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 7811 T43 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 7812 T43 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 7838 T43 oejul.Log.initialized Logging initialized @9160ms
   [junit4]   2> 7849 T43 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 7851 T43 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 7852 T43 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 7852 T43 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 7855 T44 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/'
   [junit4]   2> 7915 T44 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 7934 T44 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 7953 T44 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 7954 T44 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 7969 T44 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 8007 T44 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 8008 T44 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 8009 T44 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 8010 T44 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/
   [junit4]   2> 8011 T44 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 8012 T44 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/
   [junit4]   2> 8013 T44 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 8022 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 8023 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 8024 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001
   [junit4]   2> 8024 T44 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index/
   [junit4]   2> 8025 T44 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 8026 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index
   [junit4]   2> 8028 T44 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_1,generation=1}
   [junit4]   2> 8029 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 8031 T44 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 8032 T44 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 8034 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8039 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8039 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 8040 T44 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 8040 T44 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 8041 T44 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 8043 T44 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 8044 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 8045 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 8045 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 8059 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 8069 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 8069 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 8070 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 8070 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 8071 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 8071 T44 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 8072 T44 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 8072 T44 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 8072 T44 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 8079 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8084 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8087 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8089 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 8091 T44 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 8091 T44 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 8092 T44 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 8093 T44 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_1,generation=1}
   [junit4]   2> 8094 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 8095 T44 oass.SolrIndexSearcher.<init> Opening Searcher@6070b61d[collection1] main
   [junit4]   2> 8096 T44 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf
   [junit4]   2> 8105 T44 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf}
   [junit4]   2> 8106 T44 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/core-home-001/collection1/conf
   [junit4]   2> 8106 T44 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 8107 T44 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 8108 T44 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 8108 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@6070b61d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 8109 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 8110 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6070b61d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 8110 T43 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 8116 T43 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@be0b0df
   [junit4]   2> 8121 T43 C2 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 8121 T43 C2 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 8152 T43 C2 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 8162 T43 C2 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 8187 T43 C2 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 8188 T43 C2 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 8234 T43 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(NRTCachingDirectory(RAMDirectory@2ed2ebda lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4b0d3336; maxCacheMB=0.458984375 maxMergeSizeMB=0.5302734375)),segFN=segments_2,generation=2}
   [junit4]   2> 8235 T43 C2 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 8270 T43 C2 oass.SolrIndexSearcher.<init> Opening Searcher@2e2a214e[collection1] main
   [junit4]   2> 8272 T43 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 8272 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@2e2a214e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c27)))}
   [junit4]   2> 8286 T45 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=13 
   [junit4]   2> 8288 T45 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 8293 T45 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=4 
   [junit4]   2> 8293 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 8293 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2e2a214e[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c27)))}
   [junit4]   2> 8295 T43 C2 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 8296 T43 C2 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 8297 T43 C2 oashd.DocBuilder.execute Time taken = 0:0:0.144
   [junit4]   2> 8298 T43 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[2, 3, 1],commit=} 0 180
   [junit4]   2> 8307 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=27 status=0 QTime=8 
   [junit4]   2> 8335 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 8348 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=9 status=0 QTime=1 
   [junit4]   2> 8357 T43 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=15 status=0 QTime=1 
   [junit4]   2> 8407 T43 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 8421 T43 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 54500 T43 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=2E73AA5B93046FDE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=vi -Dtests.timezone=Asia/Karachi -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   46.1s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([2E73AA5B93046FDE:7620CB7E8E21A0E4]:0)
   [junit4]    > 	at java.util.HashMap.resize(HashMap.java:703)
   [junit4]    > 	at java.util.HashMap.putVal(HashMap.java:628)
   [junit4]    > 	at java.util.HashMap.put(HashMap.java:611)
   [junit4]    > 	at org.apache.solr.handler.dataimport.AbstractDataImportHandlerTestCase.createMap(AbstractDataImportHandlerTestCase.java:163)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 54542 T43 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 54542 T43 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1665522085
   [junit4]   2> 54543 T43 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@be0b0df
   [junit4]   2> 54543 T43 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 54544 T43 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 54544 T43 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 54544 T43 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 54545 T43 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 54546 T43 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 54547 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 54547 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001/index
   [junit4]   2> 54548 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001;done=false>>]
   [junit4]   2> 54548 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 2E73AA5B93046FDE-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=Lucene50(blocksize=128), _root_=Lucene50(blocksize=128), id=PostingsFormat(name=Direct), desc=PostingsFormat(name=LuceneVarGapFixedInterval), timestamp=Lucene50(blocksize=128)}, docValues:{}, sim=DefaultSimilarity, locale=vi, timezone=Asia/Karachi
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=53073904,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 47.07s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_120345_310.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid13598.hprof ...
   [junit4] Heap dump file created [797826427 bytes in 7.268 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 66 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) - Build # 11625 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11625/
Java: 32bit/jdk1.8.0_40-ea-b20 -client -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([3F11AE594FB86051:6742CF7C529DAF6B]:0)
	at java.util.Arrays.copyOf(Arrays.java:3210)
	at java.util.Arrays.copyOf(Arrays.java:3181)
	at java.util.ArrayList.grow(ArrayList.java:261)
	at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
	at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
	at java.util.ArrayList.addAll(ArrayList.java:579)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:299)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)




Build Log:
[...truncated 13443 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001
   [junit4]   2> 25407 T190 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 25444 T190 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 25444 T190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/'
   [junit4]   2> 25498 T190 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 25516 T190 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 25541 T190 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 25542 T190 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 25552 T190 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 25596 T190 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 25598 T190 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 25598 T190 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 25599 T190 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 25599 T190 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001
   [junit4]   2> 25599 T190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/'
   [junit4]   2> 25659 T190 oasc.CoreContainer.<init> New CoreContainer 12284623
   [junit4]   2> 25659 T190 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/]
   [junit4]   2> 25660 T190 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 25664 T190 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 25665 T190 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 25665 T190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 25666 T190 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 25666 T190 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 25666 T190 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 25667 T190 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 25667 T190 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 25667 T190 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 25668 T190 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 25669 T190 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 25669 T190 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 25669 T190 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 25670 T190 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 25672 T191 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/'
   [junit4]   2> 25712 T191 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 25721 T191 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 25773 T191 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 25776 T191 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 25789 T191 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 25819 T191 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 25820 T191 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 25821 T191 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 25822 T191 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/
   [junit4]   2> 25823 T191 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 25824 T191 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/
   [junit4]   2> 25825 T191 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 25825 T191 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 25826 T191 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 25827 T191 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001
   [junit4]   2> 25828 T191 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index/
   [junit4]   2> 25828 T191 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 25829 T191 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index
   [junit4]   2> 25830 T191 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_1,generation=1}
   [junit4]   2> 25831 T191 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 25833 T191 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 25844 T191 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 25845 T191 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 25846 T191 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 25846 T191 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 25847 T191 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 25847 T191 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 25848 T191 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 25848 T191 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 25849 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 25849 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 25850 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 25851 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 25851 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 25852 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 25852 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 25853 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 25853 T191 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 25854 T191 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 25854 T191 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 25855 T191 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 25855 T191 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 25859 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25861 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25863 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25864 T191 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 25865 T191 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 25866 T191 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 25866 T191 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 25867 T191 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_1,generation=1}
   [junit4]   2> 25867 T191 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 25867 T191 oass.SolrIndexSearcher.<init> Opening Searcher@19db2ae[collection1] main
   [junit4]   2> 25868 T191 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf
   [junit4]   2> 25868 T191 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf}
   [junit4]   2> 25869 T191 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/core-home-001/collection1/conf
   [junit4]   2> 25869 T191 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 25869 T191 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 25870 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@19db2ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 25870 T191 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 25870 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 25870 T192 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19db2ae[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 25871 T190 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 25874 T190 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C68 name=collection1 org.apache.solr.core.SolrCore@809c76
   [junit4]   2> 25877 T190 C68 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 25878 T190 C68 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 25879 T190 C68 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 25879 T190 C68 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 25890 T190 C68 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 25891 T190 C68 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> ASYNC  NEW_CORE C69 name=collection1 org.apache.solr.core.SolrCore@809c76
   [junit4]   2> 25915 T190 C69 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@9f5939 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@eb15b4),segFN=segments_2,generation=2}
   [junit4]   2> 25916 T190 C69 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 25920 T190 C69 oass.SolrIndexSearcher.<init> Opening Searcher@1c7e5f2[collection1] main
   [junit4]   2> 25920 T190 C69 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 25921 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@1c7e5f2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C25)))}
   [junit4]   2> 25922 T192 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 25923 T192 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 25924 T192 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 25924 T192 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 25924 T192 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c7e5f2[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C25)))}
   [junit4]   2> 25925 T190 C69 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 25926 T190 C69 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 25926 T190 C69 oashd.DocBuilder.execute Time taken = 0:0:0.46
   [junit4]   2> 25926 T190 C69 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[1, 2, 3],commit=} 0 51
   [junit4]   2> 25928 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=25 status=0 QTime=1 
   [junit4]   2> 25937 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=1 
   [junit4]   2> 25940 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=10 status=0 QTime=1 
   [junit4]   2> 25943 T190 C69 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=12 status=0 QTime=0 
   [junit4]   2> 25958 T190 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 25962 T190 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 56720 T190 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=3F11AE594FB86051 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt_PT -Dtests.timezone=Australia/NSW -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   31.9s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3F11AE594FB86051:6742CF7C529DAF6B]:0)
   [junit4]    > 	at java.util.Arrays.copyOf(Arrays.java:3210)
   [junit4]    > 	at java.util.Arrays.copyOf(Arrays.java:3181)
   [junit4]    > 	at java.util.ArrayList.grow(ArrayList.java:261)
   [junit4]    > 	at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
   [junit4]    > 	at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
   [junit4]    > 	at java.util.ArrayList.addAll(ArrayList.java:579)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:299)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 57823 T190 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 57823 T190 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=12284623
   [junit4]   2> 57824 T190 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@809c76
   [junit4]   2> 57824 T190 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 57825 T190 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 57825 T190 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 57826 T190 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 57828 T190 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 57829 T190 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 57829 T190 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 57830 T190 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001/index
   [junit4]   2> 57831 T190 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001;done=false>>]
   [junit4]   2> 57831 T190 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 3F11AE594FB86051-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=PostingsFormat(name=LuceneVarGapDocFreqInterval), _root_=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=PostingsFormat(name=Memory doPackFST= false), desc=FSTOrd50, timestamp=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=pt_PT, timezone=Australia/NSW
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=495680048,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestWriterImpl, TestSolrEntityProcessorEndToEnd, TestSqlEntityProcessorDelta, TestSimplePropertiesWriter, TestXPathEntityProcessor, TestBuiltInEvaluators, TestXPathRecordReader, TestNumberFormatTransformer, TestDateFormatTransformer, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 32.47s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_094442_528.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid30912.hprof ...
   [junit4] Heap dump file created [592115515 bytes in 8.211 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 80 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b20 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) - Build # 11624 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11624/
Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([A5A0DD89F042C7CC:FDF3BCACED6708F6]:0)
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
	at java.lang.StringBuilder.append(StringBuilder.java:136)
	at java.lang.StringBuilder.append(StringBuilder.java:76)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457)
	at java.lang.StringBuilder.append(StringBuilder.java:166)
	at java.lang.StringBuilder.append(StringBuilder.java:76)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2913)
	at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
	at java.util.Formatter.format(Formatter.java:2520)
	at java.util.Formatter.format(Formatter.java:2455)
	at java.lang.String.format(String.java:2969)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:296)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)




Build Log:
[...truncated 13377 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001
   [junit4]   2> 6139 T43 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 6192 T43 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 6193 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/'
   [junit4]   2> 6253 T43 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 6268 T43 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 6302 T43 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 6304 T43 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 6321 T43 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 6358 T43 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 6359 T43 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 6360 T43 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 6361 T43 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 6361 T43 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001
   [junit4]   2> 6362 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/'
   [junit4]   2> 6437 T43 oasc.CoreContainer.<init> New CoreContainer 17131600
   [junit4]   2> 6438 T43 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/]
   [junit4]   2> 6439 T43 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 6440 T43 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 6440 T43 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 6440 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 6441 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 6441 T43 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 6442 T43 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 6442 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 6442 T43 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 6443 T43 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 6444 T43 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 6446 T43 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 6449 T43 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 6449 T43 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 6449 T43 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 6453 T44 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/'
   [junit4]   2> 6492 T44 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 6501 T44 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 6530 T44 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 6531 T44 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 6542 T44 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 6561 T44 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 6561 T44 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 6562 T44 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 6562 T44 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/
   [junit4]   2> 6563 T44 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 6563 T44 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/
   [junit4]   2> 6564 T44 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 6564 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 6565 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 6565 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001
   [junit4]   2> 6566 T44 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index/
   [junit4]   2> 6566 T44 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 6566 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index
   [junit4]   2> 6567 T44 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_1,generation=1}
   [junit4]   2> 6568 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6569 T44 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 6570 T44 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 6570 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6571 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6571 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 6572 T44 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 6572 T44 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 6572 T44 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 6573 T44 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 6573 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 6574 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 6574 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 6575 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 6575 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 6575 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 6576 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 6576 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 6576 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 6577 T44 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 6577 T44 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 6577 T44 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 6578 T44 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 6580 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6582 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6588 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6590 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 6591 T44 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 6591 T44 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 6592 T44 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 6593 T44 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_1,generation=1}
   [junit4]   2> 6593 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 6594 T44 oass.SolrIndexSearcher.<init> Opening Searcher@c2c128[collection1] main
   [junit4]   2> 6595 T44 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf
   [junit4]   2> 6596 T44 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf}
   [junit4]   2> 6596 T44 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/core-home-001/collection1/conf
   [junit4]   2> 6597 T44 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 6597 T44 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 6598 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@c2c128[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6598 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 6598 T44 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 6599 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c2c128[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 6600 T43 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 6604 T43 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 54876 T43 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=A5A0DD89F042C7CC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_KW -Dtests.timezone=Etc/GMT-13 -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   48.3s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A5A0DD89F042C7CC:FDF3BCACED6708F6]:0)
   [junit4]    > 	at java.util.Arrays.copyOf(Arrays.java:3332)
   [junit4]    > 	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
   [junit4]    > 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
   [junit4]    > 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:136)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:76)
   [junit4]    > 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:457)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:166)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:76)
   [junit4]    > 	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2913)
   [junit4]    > 	at java.util.Formatter$FormatSpecifier.printString(Formatter.java:2886)
   [junit4]    > 	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2763)
   [junit4]    > 	at java.util.Formatter.format(Formatter.java:2520)
   [junit4]    > 	at java.util.Formatter.format(Formatter.java:2455)
   [junit4]    > 	at java.lang.String.format(String.java:2969)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:296)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 54936 T43 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@1dbd1a7
   [junit4]   2> 54942 T43 C3 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 54943 T43 C3 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 54956 T43 C3 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 54957 T43 C3 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 54968 T43 C3 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 54969 T43 C3 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 54978 T43 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@165f9e0 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@112f1ce),segFN=segments_2,generation=2}
   [junit4]   2> 54978 T43 C3 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 54980 T43 C3 oass.SolrIndexSearcher.<init> Opening Searcher@16fdc04[collection1] main
   [junit4]   2> 54981 T43 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 54981 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@16fdc04[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C26)))}
   [junit4]   2> 54991 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=9 
   [junit4]   2> 54992 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 54994 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=2 
   [junit4]   2> 54994 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 54994 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@16fdc04[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C26)))}
   [junit4]   2> 54995 T43 C3 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 54997 T43 C3 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 54997 T43 C3 oashd.DocBuilder.execute Time taken = 0:0:0.40
   [junit4]   2> 54997 T43 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[1, 2, 3],commit=} 0 59
   [junit4]   2> 54999 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=26 status=0 QTime=1 
   [junit4]   2> 55013 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 55016 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=10 status=0 QTime=0 
   [junit4]   2> 55021 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=13 status=0 QTime=1 
   [junit4]   2> 55049 T43 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 55051 T43 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 55052 T43 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=17131600
   [junit4]   2> 55052 T43 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1dbd1a7
   [junit4]   2> 55053 T43 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 55053 T43 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 55053 T43 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 55054 T43 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 55054 T43 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 55055 T43 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 55055 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 55056 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001/index
   [junit4]   2> 55056 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001;done=false>>]
   [junit4]   2> 55057 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A5A0DD89F042C7CC-001
   [junit4]   2> NOTE: test params are: codec=Lucene50, sim=DefaultSimilarity, locale=ar_KW, timezone=Etc/GMT-13
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=51361120,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 48.96s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_071446_963.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid14852.hprof ...
   [junit4] Heap dump file created [592095532 bytes in 8.216 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 64 minutes 2 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b20 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build # 11623 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11623/
Java: 32bit/jdk1.8.0_25 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.DeleteReplicaTest.testDistribSearch

Error Message:
Should have had a good message here

Stack Trace:
java.lang.AssertionError: Should have had a good message here
	at __randomizedtesting.SeedInfo.seed([409FEEBDB49636E3:C17960A5C3C956DF]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:138)
	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 8752 lines...]
   [junit4] Suite: org.apache.solr.cloud.DeleteReplicaTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/init-core-data-001
   [junit4]   2> 135483 T1460 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 135484 T1460 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 135487 T1460 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 135487 T1460 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 135488 T1461 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 135588 T1460 oasc.ZkTestServer.run start zk server on port:39666
   [junit4]   2> 135588 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 135589 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135592 T1468 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d243d9 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135593 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135593 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 135593 T1460 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 135596 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 135596 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135598 T1471 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9c1241 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135598 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135598 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 135598 T1460 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 135600 T1460 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 135601 T1460 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 135602 T1460 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 135604 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 135604 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 135606 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 135606 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 135607 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 135607 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 135609 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 135609 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 135610 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 135611 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 135612 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 135612 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 135613 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 135614 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 135615 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 135615 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 135616 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 135617 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 135618 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 135618 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 135619 T1460 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 135619 T1460 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 135872 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 135874 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@1f9b78e{/,null,AVAILABLE}
   [junit4]   2> 135880 T1460 oejs.AbstractConnector.doStart Started ServerConnector@f250e4{HTTP/1.1}{127.0.0.1:36085}
   [junit4]   2> 135880 T1460 oejs.Server.doStart Started @137156ms
   [junit4]   2> 135880 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 135881 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 135881 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002
   [junit4]   2> 135881 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/'
   [junit4]   2> 135892 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/solr.xml
   [junit4]   2> 135905 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002
   [junit4]   2> 135906 T1460 oasc.CoreContainer.<init> New CoreContainer 21422377
   [junit4]   2> 135906 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/]
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 135907 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 135908 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 135909 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 135909 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 135909 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 135909 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 135910 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 135910 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 135910 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 135910 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 135911 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 135911 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 135912 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135913 T1485 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17bb112 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135913 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135914 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 135916 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 135917 T1488 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1790f4c name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 135917 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 135919 T1460 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 135921 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 135922 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 135924 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 135925 T1460 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 135927 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 135928 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36085_
   [junit4]   2> 135928 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36085_
   [junit4]   2> 135930 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 135931 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 135932 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 135933 T1460 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:36085_
   [junit4]   2> 135933 T1460 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 135934 T1460 oasc.Overseer.start Overseer (id=93157454132019203-127.0.0.1:36085_-n_0000000000) starting
   [junit4]   2> 135937 T1460 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 135943 T1460 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 135944 T1490 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 135945 T1460 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 135946 T1460 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 135948 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 135950 T1489 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 135951 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002
   [junit4]   2> 135958 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/conf/
   [junit4]   2> 136076 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 136077 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 136077 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 136077 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 136078 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136079 T1495 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18c6501 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136079 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136080 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 136081 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 136083 T1460 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 136084 T1460 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 136084 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 136084 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136085 T1498 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@161dc0f name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136086 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136086 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 136088 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 136090 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 0
   [junit4]   2> 136090 T1489 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 136093 T1489 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 136094 T1495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 136095 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 136333 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 136334 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 136336 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7f3edf{/,null,AVAILABLE}
   [junit4]   2> 136337 T1460 oejs.AbstractConnector.doStart Started ServerConnector@15db3d1{HTTP/1.1}{127.0.0.1:52980}
   [junit4]   2> 136337 T1460 oejs.Server.doStart Started @137613ms
   [junit4]   2> 136337 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 136338 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 136338 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003
   [junit4]   2> 136338 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/'
   [junit4]   2> 136348 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/solr.xml
   [junit4]   2> 136354 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003
   [junit4]   2> 136354 T1460 oasc.CoreContainer.<init> New CoreContainer 18741746
   [junit4]   2> 136355 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/]
   [junit4]   2> 136355 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 136355 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 136356 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 136357 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 136358 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 136358 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 136358 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 136359 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 136359 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 136359 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 136359 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 136360 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 136360 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136362 T1512 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f8a8bb name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136362 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136363 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 136365 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 136366 T1515 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b8c6fd name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 136367 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 136371 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 137375 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52980_
   [junit4]   2> 137377 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:52980_
   [junit4]   2> 137381 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 137385 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003
   [junit4]   2> 137391 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-003/conf/
   [junit4]   2> 137511 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 137512 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 137512 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 137763 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 137764 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 137765 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@160f77c{/,null,AVAILABLE}
   [junit4]   2> 137766 T1460 oejs.AbstractConnector.doStart Started ServerConnector@bd8155{HTTP/1.1}{127.0.0.1:48955}
   [junit4]   2> 137766 T1460 oejs.Server.doStart Started @139043ms
   [junit4]   2> 137767 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 137767 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 137767 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004
   [junit4]   2> 137768 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/'
   [junit4]   2> 137779 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/solr.xml
   [junit4]   2> 137784 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004
   [junit4]   2> 137785 T1460 oasc.CoreContainer.<init> New CoreContainer 3577483
   [junit4]   2> 137785 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/]
   [junit4]   2> 137785 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 137786 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 137787 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 137788 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 137788 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 137788 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 137789 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 137789 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 137789 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 137789 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 137790 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 137790 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 137791 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 137792 T1530 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cc3021 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 137792 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 137793 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 137795 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 137796 T1533 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e74805 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 137796 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 137800 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 138803 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48955_
   [junit4]   2> 138804 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48955_
   [junit4]   2> 138809 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 138813 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004
   [junit4]   2> 138822 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/conf/
   [junit4]   2> 138976 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 138977 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 138977 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 139243 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 3
   [junit4]   2> 139244 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 139245 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@87aa5f{/,null,AVAILABLE}
   [junit4]   2> 139246 T1460 oejs.AbstractConnector.doStart Started ServerConnector@1fe29e8{HTTP/1.1}{127.0.0.1:39630}
   [junit4]   2> 139246 T1460 oejs.Server.doStart Started @140522ms
   [junit4]   2> 139247 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 139247 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 139247 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005
   [junit4]   2> 139248 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/'
   [junit4]   2> 139259 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/solr.xml
   [junit4]   2> 139264 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005
   [junit4]   2> 139264 T1460 oasc.CoreContainer.<init> New CoreContainer 3704360
   [junit4]   2> 139265 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/]
   [junit4]   2> 139265 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 139265 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 139266 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 139267 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 139268 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 139268 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 139268 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 139269 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 139269 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 139269 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 139269 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 139270 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 139270 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 139272 T1548 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e3f7dc name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 139273 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 139273 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 139276 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 139277 T1551 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11455c4 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 139277 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 139282 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 140285 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39630_
   [junit4]   2> 140286 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39630_
   [junit4]   2> 140289 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 140292 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005
   [junit4]   2> 140299 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/conf/
   [junit4]   2> 140451 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 140452 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 140452 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 140730 T1460 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 4
   [junit4]   2> 140731 T1460 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 140733 T1460 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@f87fa2{/,null,AVAILABLE}
   [junit4]   2> 140733 T1460 oejs.AbstractConnector.doStart Started ServerConnector@3c9fd6{HTTP/1.1}{127.0.0.1:39298}
   [junit4]   2> 140734 T1460 oejs.Server.doStart Started @142010ms
   [junit4]   2> 140734 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 140735 T1460 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 140735 T1460 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006
   [junit4]   2> 140735 T1460 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/'
   [junit4]   2> 140746 T1460 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/solr.xml
   [junit4]   2> 140751 T1460 oasc.CorePropertiesLocator.<init> Config-defined core root directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006
   [junit4]   2> 140752 T1460 oasc.CoreContainer.<init> New CoreContainer 27949791
   [junit4]   2> 140752 T1460 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/]
   [junit4]   2> 140752 T1460 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 140753 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 140754 T1460 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 140755 T1460 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 140755 T1460 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 140755 T1460 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 140756 T1460 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 140756 T1460 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 140756 T1460 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 140756 T1460 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:39666/solr
   [junit4]   2> 140757 T1460 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 140757 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 140758 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 140759 T1566 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4ea416 name:ZooKeeperConnection Watcher:127.0.0.1:39666 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 140759 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 140760 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 140762 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 140764 T1569 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11eb2be name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 140764 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 140768 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 141771 T1460 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39298_
   [junit4]   2> 141772 T1460 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39298_
   [junit4]   2> 141775 T1460 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 141777 T1460 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006
   [junit4]   2> 141783 T1460 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/conf/
   [junit4]   2> 141932 T1460 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 141933 T1460 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 141933 T1460 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 141935 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 141935 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 141937 T1573 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@164e654 name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 141937 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 141937 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 141939 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 141940 T1460 oasc.AbstractFullDistribZkTestBase.createCollection Creating collection with stateFormat=2: delLiveColl
   [junit4]   2> 141941 T1460 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 141941 T1460 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 141942 T1576 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d675ec name:ZooKeeperConnection Watcher:127.0.0.1:39666/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 141943 T1460 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 141943 T1460 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 141944 T1460 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 141951 T1523 oasha.CollectionsHandler.handleCreateAction Creating Collection : replicationFactor=2&maxShardsPerNode=2&name=delLiveColl&action=CREATE&numShards=2&stateFormat=2&wt=javabin&version=2
   [junit4]   2> 141952 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/collection-queue-work state SyncConnected
   [junit4]   2> 141953 T1490 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 141955 T1578 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : create , {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"}
   [junit4]   2> 141956 T1578 oasc.OverseerCollectionProcessor.createConfNode Only one config set found in zk - using it:conf1
   [junit4]   2> 141956 T1578 oasc.OverseerCollectionProcessor.createConfNode creating collections conf node /collections/delLiveColl 
   [junit4]   2> 141957 T1578 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl
   [junit4]   2> 141959 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 141960 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "fromApi":"true",
   [junit4]   2> 	  "name":"delLiveColl",
   [junit4]   2> 	  "replicationFactor":"2",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "maxShardsPerNode":"2",
   [junit4]   2> 	  "stateFormat":"2"} current state version: 1
   [junit4]   2> 141960 T1489 oasco.ClusterStateMutator.createCollection building a new cName: delLiveColl
   [junit4]   2> 141961 T1489 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/delLiveColl/state.json
   [junit4]   2> 141962 T1495 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141962 T1551 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1569 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1533 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141962 T1576 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1488 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141963 T1515 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 141962 T1573 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 142160 T1578 oasc.OverseerCollectionProcessor.createCollection Creating SolrCores for new collection delLiveColl, shardNames [shard1, shard2] , replicationFactor : 2
   [junit4]   2> 142160 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica1 as part of slice shard1 of collection delLiveColl on 127.0.0.1:39298_
   [junit4]   2> 142160 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard1_replica2 as part of slice shard1 of collection delLiveColl on 127.0.0.1:36085_
   [junit4]   2> 142163 T1559 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard1_replica1&action=CREATE&numShards=2&collection=delLiveColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 142169 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica1 as part of slice shard2 of collection delLiveColl on 127.0.0.1:48955_
   [junit4]   2> 142169 T1578 oasc.OverseerCollectionProcessor.createCollection Creating shard delLiveColl_shard2_replica2 as part of slice shard2 of collection delLiveColl on 127.0.0.1:39630_
   [junit4]   2> 142169 T1559 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica1 state=down collection=delLiveColl
   [junit4]   2> 142171 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142173 T1559 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142173 T1559 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142173 T1559 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142173 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142174 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142175 T1559 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 142175 T1559 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142182 T1541 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard2_replica2&action=CREATE&numShards=2&collection=delLiveColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 142182 T1524 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard2_replica1&action=CREATE&numShards=2&collection=delLiveColl&shard=shard2&wt=javabin&version=2
   [junit4]   2> 142182 T1478 oasha.CoreAdminHandler.handleCreateAction core create command qt=/admin/cores&collection.configName=conf1&name=delLiveColl_shard1_replica2&action=CREATE&numShards=2&collection=delLiveColl&shard=shard1&wt=javabin&version=2
   [junit4]   2> 142184 T1541 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica2 state=down collection=delLiveColl
   [junit4]   2> 142184 T1524 oasc.ZkController.publish publishing core=delLiveColl_shard2_replica1 state=down collection=delLiveColl
   [junit4]   2> 142185 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142187 T1524 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142187 T1524 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142188 T1524 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142189 T1524 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 142189 T1524 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142196 T1541 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142196 T1541 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142197 T1541 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142197 T1478 oasc.ZkController.publish publishing core=delLiveColl_shard1_replica2 state=down collection=delLiveColl
   [junit4]   2> 142198 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39630",
   [junit4]   2> 	  "node_name":"127.0.0.1:39630_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142198 T1541 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 0 
   [junit4]   2> 142199 T1541 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142199 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39630",
   [junit4]   2> 	  "node_name":"127.0.0.1:39630_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142200 T1478 oasc.ZkController.preRegister Registering watch for external collection delLiveColl
   [junit4]   2> 142201 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48955",
   [junit4]   2> 	  "node_name":"127.0.0.1:48955_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142202 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard2_replica1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:48955",
   [junit4]   2> 	  "node_name":"127.0.0.1:48955_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142203 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142204 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36085",
   [junit4]   2> 	  "node_name":"127.0.0.1:36085_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 142204 T1489 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica2",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:36085",
   [junit4]   2> 	  "node_name":"127.0.0.1:36085_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 142206 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142306 T1489 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/delLiveColl/state.json version: 0
   [junit4]   2> 142307 T1569 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142307 T1551 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142307 T1533 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142308 T1569 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142308 T1551 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142308 T1533 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142311 T1478 oascc.ZkStateReader.addZkWatch addZkWatch delLiveColl
   [junit4]   2> 142311 T1478 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/delLiveColl/state.json from ZooKeeper... 
   [junit4]   2> 142312 T1478 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 1 
   [junit4]   2> 142312 T1478 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 142313 T1478 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica2
   [junit4]   2> 142313 T1478 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 142313 T1478 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 142314 T1478 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 142315 T1478 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 142315 T1478 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/'
   [junit4]   2> 142332 T1478 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 142334 T1478 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 142340 T1478 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 142346 T1478 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 142357 T1478 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 142358 T1478 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 142368 T1478 oass.IndexSchema.readSchema [delLiveColl_shard1_replica2] Schema name=test
   [junit4]   2> 142486 T1478 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 142489 T1478 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 142491 T1478 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 142498 T1478 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 142500 T1478 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 142501 T1478 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 142502 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 142502 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 142503 T1478 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 142503 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 142504 T1478 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 142504 T1478 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica2' using configuration from collection delLiveColl
   [junit4]   2> 142504 T1478 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 142504 T1478 oasc.SolrCore.<init> [delLiveColl_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/
   [junit4]   2> 142505 T1478 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d29aa9
   [junit4]   2> 142505 T1478 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data
   [junit4]   2> 142506 T1478 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/index/
   [junit4]   2> 142506 T1478 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 142506 T1478 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-002/delLiveColl_shard1_replica2/data/index
   [junit4]   2> 142507 T1478 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 142507 T1478 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ae757e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13fc62e),segFN=segments_1,generation=1}
   [junit4]   2> 142508 T1478 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 142511 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 142511 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 142511 T1478 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 142512 T1478 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 142513 T1478 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 142513 T1478 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 142513 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 142514 T1478 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 142515 T1478 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 142515 T1478 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 142515 T1478 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 142516 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 142516 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 142517 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 142517 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 142517 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 142518 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 142518 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 142518 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 142519 T1478 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 142519 T1478 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 142520 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142521 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142522 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142523 T1478 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 142523 T1478 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 142524 T1478 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 142524 T1478 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 142525 T1478 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=27, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2255999651279229]
   [junit4]   2> 142525 T1478 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@ae757e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@13fc62e),segFN=segments_1,generation=1}
   [junit4]   2> 142525 T1478 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 142526 T1478 oass.SolrIndexSearcher.<init> Opening Searcher@8f43c9[delLiveColl_shard1_replica2] main
   [junit4]   2> 142526 T1478 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 142527 T1478 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 142527 T1478 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 142527 T1478 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 142527 T1478 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 142528 T1478 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 142528 T1478 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 142528 T1478 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 142529 T1478 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 142529 T1478 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 142530 T1586 oasc.SolrCore.registerSearcher [delLiveColl_shard1_replica2] Registered new searcher Searcher@8f43c9[delLiveColl_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 142530 T1478 oasc.CoreContainer.registerCore registering core: delLiveColl_shard1_replica2
   [junit4]   2> 142530 T1478 oasc.ZkController.register Register replica - core:delLiveColl_shard1_replica2 address:http://127.0.0.1:36085 collection:delLiveColl shard:shard1
   [junit4]   2> 142531 T1478 oascc.SolrZkClient.makePath makePath: /collections/delLiveColl/leader_elect/shard1/election
   [junit4]   2> 142534 T1478 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 142535 T1488 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 142535 T1478 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999ms
   [junit4]   2> 142535 T1489 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl"} current state version: 2
   [junit4]   2> 142637 T1489 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/delLiveColl/state.json version: 1
   [junit4]   2> 142637 T1551 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1569 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1488 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1569 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 142637 T1533 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 142638 T1551 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 142639 T1533 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 142639 T1488 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 2 
   [junit4]   2> 143175 T1559 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard1_replica1
   [junit4]   2> 143176 T1559 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 143176 T1559 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 143176 T1559 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 143177 T1559 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 143177 T1559 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/'
   [junit4]   2> 143187 T1559 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 143189 T1559 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 143189 T1524 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica1
   [junit4]   2> 143190 T1524 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 143190 T1524 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 143190 T1524 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 143191 T1524 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 143191 T1524 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-004/delLiveColl_shard2_replica1/'
   [junit4]   2> 143193 T1559 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 143199 T1541 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for delLiveColl_shard2_replica2
   [junit4]   2> 143199 T1541 oasc.ZkController.createCollectionZkNode Check for collection zkNode:delLiveColl
   [junit4]   2> 143200 T1541 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 143200 T1541 oascc.ZkStateReader.readConfigName Load collection config from:/collections/delLiveColl
   [junit4]   2> 143201 T1541 oascc.ZkStateReader.readConfigName path=/collections/delLiveColl configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 143201 T1541 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-005/delLiveColl_shard2_replica2/'
   [junit4]   2> 143202 T1559 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 143207 T1524 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 143211 T1559 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 143212 T1559 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 143212 T1524 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 143213 T1541 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 143215 T1541 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 143216 T1559 oass.IndexSchema.readSchema [delLiveColl_shard1_replica1] Schema name=test
   [junit4]   2> 143218 T1524 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 143219 T1541 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 143224 T1524 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 143225 T1541 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 143238 T1524 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 143239 T1541 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 143239 T1524 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 143240 T1541 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 143246 T1524 oass.IndexSchema.readSchema [delLiveColl_shard2_replica1] Schema name=test
   [junit4]   2> 143246 T1541 oass.IndexSchema.readSchema [delLiveColl_shard2_replica2] Schema name=test
   [junit4]   2> 143343 T1559 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 143349 T1559 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 143351 T1559 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 143365 T1559 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 143368 T1559 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 143370 T1559 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 143371 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 143371 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 143371 T1559 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 143372 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 143372 T1559 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 143373 T1559 oasc.CoreContainer.create Creating SolrCore 'delLiveColl_shard1_replica1' using configuration from collection delLiveColl
   [junit4]   2> 143373 T1559 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 143373 T1559 oasc.SolrCore.<init> [delLiveColl_shard1_replica1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/
   [junit4]   2> 143374 T1559 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1d29aa9
   [junit4]   2> 143374 T1559 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data
   [junit4]   2> 143375 T1559 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index/
   [junit4]   2> 143375 T1559 oasc.SolrCore.initIndex WARN [delLiveColl_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 143375 T1559 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 143376 T1559 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=29, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 143377 T1559 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@123db4b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@215677),segFN=segments_1,generation=1}
   [junit4]   2> 143377 T1559 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 143382 T1559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 143382 T1559 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 143383 T1559 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 143383 T1559 

[...truncated too long message...]

iveColl_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@119fc6e
   [junit4]   2> 153607 T1460 oasc.ZkController.unRegisterConfListener  a listener was removed because of core close
   [junit4]   2> 153607 T1460 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0,transaction_logs_total_size=0,transaction_logs_total_number=0}
   [junit4]   2> 153607 T1460 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 153608 T1460 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 153608 T1460 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 153609 T1460 oasc.SolrCore.closeSearcher [delLiveColl_shard1_replica1] Closing main searcher on request.
   [junit4]   2> 153610 T1625 oascc.ZkStateReader.updateWatchedCollection Updating data for delLiveColl to ver 16 
   [junit4]   2> 153611 T1625 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 153611 T1625 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"delLiveColl_shard1_replica1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:39298",
   [junit4]   2> 	  "node_name":"127.0.0.1:39298_",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"delLiveColl",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 153612 T1625 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/delLiveColl/state.json version: 16
   [junit4]   2> 153613 T1607 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/delLiveColl/state.json, has occurred - updating... 
   [junit4]   2> 153621 T1460 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 153621 T1460 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index;done=false>>]
   [junit4]   2> 153622 T1460 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data/index
   [junit4]   2> 153622 T1460 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data;done=false>>]
   [junit4]   2> 153622 T1460 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001/tempDir-006/delLiveColl_shard1_replica1/data
   [junit4]   2> 153623 T1460 oasc.Overseer.close Overseer (id=93157454132019213-127.0.0.1:39298_-n_0000000004) closing
   [junit4]   2> 153623 T1625 oasc.Overseer$ClusterStateUpdater.run Overseer Loop exiting : 127.0.0.1:39298_
   [junit4]   2> 155124 T1607 oascc.ZkStateReader$7.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 155124 T1569 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 155125 T1460 oejs.AbstractConnector.doStop Stopped ServerConnector@3c9fd6{HTTP/1.1}{127.0.0.1:0}
   [junit4]   2> 155126 T1460 oejsh.ContextHandler.doStop Stopped o.e.j.s.ServletContextHandler@f87fa2{/,null,UNAVAILABLE}
   [junit4]   2> 155127 T1460 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 155128 T1460 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39666 39666
   [junit4]   2> 155671 T1461 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:39666 39666
   [junit4]   2> 155674 T1461 oasc.ZkTestServer$ZKServerMain.runFromConfig WARN Watch limit violations: 
   [junit4]   2> 	Maximum concurrent create/delete watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/aliases.json
   [junit4]   2> 		8	/solr/clusterstate.json
   [junit4]   2> 		4	/solr/collections/delLiveColl/state.json
   [junit4]   2> 		4	/solr/configs/conf1
   [junit4]   2> 	
   [junit4]   2> 	Maximum concurrent children watches above limit:
   [junit4]   2> 	
   [junit4]   2> 		8	/solr/live_nodes
   [junit4]   2> 		5	/solr/overseer/queue
   [junit4]   2> 		5	/solr/overseer/collection-queue-work
   [junit4]   2> 	
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DeleteReplicaTest -Dtests.method=testDistribSearch -Dtests.seed=409FEEBDB49636E3 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SA -Dtests.timezone=Pacific/Rarotonga -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 20.2s J0 | DeleteReplicaTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Should have had a good message here
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([409FEEBDB49636E3:C17960A5C3C956DF]:0)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.deleteLiveReplicaTest(DeleteReplicaTest.java:138)
   [junit4]    > 	at org.apache.solr.cloud.DeleteReplicaTest.doTest(DeleteReplicaTest.java:89)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 155689 T1460 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.DeleteReplicaTest 409FEEBDB49636E3-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=ar_SA, timezone=Pacific/Rarotonga
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=49578496,total=252182528
   [junit4]   2> NOTE: All tests run in this JVM: [TestRebalanceLeaders, TestSort, HdfsRecoveryZkTest, SortByFunctionTest, TestShortCircuitedRequests, TestLRUCache, TestSolrQueryParserDefaultOperatorResource, TestDistributedSearch, AnalysisAfterCoreReloadTest, TestReplicationHandler, SolrTestCaseJ4Test, CacheHeaderTest, TestFastOutputStream, IndexSchemaRuntimeFieldTest, DeleteReplicaTest]
   [junit4] Completed on J0 in 20.24s, 1 test, 1 failure <<< FAILURES!

[...truncated 1322 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 463 suites, 1871 tests, 1 failure, 47 ignored (21 assumptions)

Total time: 61 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build # 11622 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11622/
Java: 32bit/jdk1.8.0_25 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.testDistribSearch

Error Message:
commitWithin did not work on node: http://127.0.0.1:44387/a_gcu/collection1 expected:<68> but was:<67>

Stack Trace:
java.lang.AssertionError: commitWithin did not work on node: http://127.0.0.1:44387/a_gcu/collection1 expected:<68> but was:<67>
	at __randomizedtesting.SeedInfo.seed([3BA9F288B360E2BB:BA4F7C90C43F8287]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:345)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9287 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZkTest
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/init-core-data-001
   [junit4]   2> 731868 T4616 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /a_gcu/
   [junit4]   2> 731873 T4616 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 731874 T4616 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 731874 T4617 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 731974 T4616 oasc.ZkTestServer.run start zk server on port:33255
   [junit4]   2> 731975 T4616 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 731976 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 731979 T4624 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10430eb name:ZooKeeperConnection Watcher:127.0.0.1:33255 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 731979 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 731980 T4616 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 731980 T4616 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 731984 T4616 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 731984 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 731986 T4627 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15b3cd1 name:ZooKeeperConnection Watcher:127.0.0.1:33255/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 731986 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 731986 T4616 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 731987 T4616 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 731988 T4616 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 731989 T4616 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 731990 T4616 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 731992 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 731992 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 731995 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 731995 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 731997 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 731998 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 731999 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 731999 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 732004 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 732005 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 732013 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 732013 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 732015 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 732015 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 732017 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 732017 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 732018 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 732019 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 732020 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 732021 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 732022 T4616 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 732022 T4616 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 732386 T4616 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 732398 T4616 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@165e589{/a_gcu,null,AVAILABLE}
   [junit4]   2> 732400 T4616 oejs.AbstractConnector.doStart Started ServerConnector@209a34{HTTP/1.1}{127.0.0.1:35488}
   [junit4]   2> 732400 T4616 oejs.Server.doStart Started @733620ms
   [junit4]   2> 732401 T4616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 732402 T4616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 732402 T4616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-002
   [junit4]   2> 732402 T4616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-002/'
   [junit4]   2> 732413 T4616 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-002/solr.xml
   [junit4]   2> 732420 T4616 oasc.CoreContainer.<init> New CoreContainer 16178261
   [junit4]   2> 732421 T4616 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-002/]
   [junit4]   2> 732421 T4616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 732421 T4616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 732422 T4616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 732422 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 732422 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 732423 T4616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 732423 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 732423 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 732423 T4616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 732424 T4616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 732424 T4616 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 732424 T4616 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 732425 T4616 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 732425 T4616 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 732425 T4616 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 732426 T4616 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33255/solr
   [junit4]   2> 732426 T4616 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 732426 T4616 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 732427 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 732429 T4641 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18c5420 name:ZooKeeperConnection Watcher:127.0.0.1:33255 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 732429 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 732430 T4616 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 732431 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 732436 T4644 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1da144b name:ZooKeeperConnection Watcher:127.0.0.1:33255/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 732437 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 732438 T4616 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 732441 T4616 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 732443 T4616 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 732445 T4616 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 732449 T4616 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 732451 T4616 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 732452 T4616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35488_a_gcu
   [junit4]   2> 732453 T4616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35488_a_gcu
   [junit4]   2> 732454 T4616 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 732473 T4616 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 732474 T4616 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 732476 T4616 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:35488_a_gcu
   [junit4]   2> 732476 T4616 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 732478 T4616 oasc.Overseer.start Overseer (id=93156972610846723-127.0.0.1:35488_a_gcu-n_0000000000) starting
   [junit4]   2> 732480 T4616 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 732485 T4616 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 732487 T4645 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 732489 T4616 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 732490 T4646 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 732491 T4616 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 732492 T4616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 732495 T4648 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 732496 T4648 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 732496 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 732497 T4648 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 732498 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35488/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:35488_a_gcu",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 732499 T4645 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35488/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:35488_a_gcu",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 732499 T4645 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 732500 T4645 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 732500 T4644 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733497 T4648 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 733497 T4648 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 733498 T4648 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 733498 T4648 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 733498 T4648 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 733499 T4648 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-002/collection1/'
   [junit4]   2> 733499 T4648 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 733499 T4648 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 733500 T4648 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 733509 T4648 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 733511 T4648 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 733514 T4648 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 733518 T4648 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 733526 T4648 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 733527 T4648 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 733531 T4648 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 733630 T4648 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 733635 T4648 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 733637 T4648 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 733650 T4648 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 733652 T4648 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 733655 T4648 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 733656 T4648 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 733656 T4648 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 733656 T4648 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 733657 T4648 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 733658 T4648 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 733658 T4648 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 733658 T4648 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 733658 T4648 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/control/data/
   [junit4]   2> 733659 T4648 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e376ba
   [junit4]   2> 733659 T4648 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/control/data
   [junit4]   2> 733660 T4648 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/control/data/index/
   [junit4]   2> 733660 T4648 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 733660 T4648 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/control/data/index
   [junit4]   2> 733661 T4648 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 733661 T4648 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@19a639d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c5a28a),segFN=segments_1,generation=1}
   [junit4]   2> 733661 T4648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 733664 T4648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 733664 T4648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 733664 T4648 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 733665 T4648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 733665 T4648 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 733665 T4648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 733666 T4648 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 733666 T4648 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 733666 T4648 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 733667 T4648 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 733667 T4648 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 733667 T4648 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 733668 T4648 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 733668 T4648 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 733668 T4648 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 733669 T4648 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 733669 T4648 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 733669 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 733670 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 733670 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 733671 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 733671 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 733671 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 733672 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 733672 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 733672 T4648 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 733673 T4648 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 733675 T4648 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 733676 T4648 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 733677 T4648 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 733677 T4648 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 733678 T4648 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 733679 T4648 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 733679 T4648 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 733679 T4648 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8010367934600408]
   [junit4]   2> 733680 T4648 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@19a639d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@c5a28a),segFN=segments_1,generation=1}
   [junit4]   2> 733680 T4648 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 733680 T4648 oass.SolrIndexSearcher.<init> Opening Searcher@162b286[collection1] main
   [junit4]   2> 733680 T4648 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 733681 T4648 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 733681 T4648 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 733682 T4648 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 733682 T4648 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 733682 T4648 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 733682 T4648 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 733682 T4648 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 733683 T4648 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 733683 T4648 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 733683 T4649 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@162b286[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 733684 T4648 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 733684 T4652 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:35488/a_gcu collection:control_collection shard:shard1
   [junit4]   2> 733685 T4616 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 733685 T4616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 733686 T4616 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 733686 T4652 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 733687 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 733688 T4655 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@de3b9c name:ZooKeeperConnection Watcher:127.0.0.1:33255/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 733688 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 733689 T4616 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 733689 T4616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 733690 T4652 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 733691 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 733691 T4616 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 733691 T4652 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 733692 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 733692 T4652 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C771 name=collection1 org.apache.solr.core.SolrCore@12f5abb url=http://127.0.0.1:35488/a_gcu/collection1 node=127.0.0.1:35488_a_gcu C771_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:35488/a_gcu, node_name=127.0.0.1:35488_a_gcu, state=down}
   [junit4]   2> 733692 T4652 C771 P35488 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35488/a_gcu/collection1/
   [junit4]   2> 733693 T4652 C771 P35488 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 733693 T4652 C771 P35488 oasc.SyncStrategy.syncToMe http://127.0.0.1:35488/a_gcu/collection1/ has no replicas
   [junit4]   2> 733693 T4652 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:35488/a_gcu/collection1/ shard1
   [junit4]   2> 733694 T4652 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 733696 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 733697 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35488/a_gcu",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 1
   [junit4]   2> 733799 T4644 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733799 T4655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733848 T4652 oasc.ZkController.register We are http://127.0.0.1:35488/a_gcu/collection1/ and leader is http://127.0.0.1:35488/a_gcu/collection1/
   [junit4]   2> 733848 T4652 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35488/a_gcu
   [junit4]   2> 733848 T4652 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 733848 T4652 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 733848 T4652 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 733849 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 733851 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35488/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:35488_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 2
   [junit4]   2> 733852 T4645 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:35488/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:35488_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 733954 T4655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 733954 T4644 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 734087 T4616 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 734088 T4616 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 734091 T4616 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@7d8504{/a_gcu,null,AVAILABLE}
   [junit4]   2> 734092 T4616 oejs.AbstractConnector.doStart Started ServerConnector@1b60d0{HTTP/1.1}{127.0.0.1:44387}
   [junit4]   2> 734092 T4616 oejs.Server.doStart Started @735312ms
   [junit4]   2> 734093 T4616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 734093 T4616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 734094 T4616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-003
   [junit4]   2> 734094 T4616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-003/'
   [junit4]   2> 734113 T4616 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-003/solr.xml
   [junit4]   2> 734123 T4616 oasc.CoreContainer.<init> New CoreContainer 29838145
   [junit4]   2> 734124 T4616 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-003/]
   [junit4]   2> 734124 T4616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 734125 T4616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 734125 T4616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 734126 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 734126 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 734127 T4616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 734127 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 734127 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 734128 T4616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 734128 T4616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 734129 T4616 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 734129 T4616 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 734130 T4616 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 734130 T4616 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 734131 T4616 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 734131 T4616 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33255/solr
   [junit4]   2> 734132 T4616 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 734132 T4616 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 734133 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 734134 T4669 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d59516 name:ZooKeeperConnection Watcher:127.0.0.1:33255 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 734134 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 734135 T4616 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 734137 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 734137 T4672 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d01770 name:ZooKeeperConnection Watcher:127.0.0.1:33255/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 734138 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 734141 T4616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 735144 T4616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44387_a_gcu
   [junit4]   2> 735145 T4616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44387_a_gcu
   [junit4]   2> 735149 T4616 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 735152 T4673 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 735152 T4673 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 735153 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 735153 T4673 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 735154 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44387/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:44387_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 3
   [junit4]   2> 735154 T4645 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44387/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:44387_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 735154 T4645 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 735154 T4645 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 735257 T4655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 735257 T4672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 735257 T4644 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736153 T4673 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 736154 T4673 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 736154 T4673 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 736154 T4673 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 736155 T4673 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 736155 T4673 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-003/collection1/'
   [junit4]   2> 736156 T4673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 736156 T4673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 736156 T4673 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 736165 T4673 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 736167 T4673 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 736171 T4673 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 736174 T4673 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 736180 T4673 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 736181 T4673 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 736185 T4673 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 736278 T4673 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 736281 T4673 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 736283 T4673 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 736290 T4673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 736292 T4673 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 736294 T4673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 736295 T4673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 736295 T4673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 736295 T4673 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 736296 T4673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 736296 T4673 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 736296 T4673 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 736296 T4673 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 736297 T4673 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty1/
   [junit4]   2> 736297 T4673 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e376ba
   [junit4]   2> 736297 T4673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty1
   [junit4]   2> 736298 T4673 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty1/index/
   [junit4]   2> 736298 T4673 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 736298 T4673 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty1/index
   [junit4]   2> 736298 T4673 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 736299 T4673 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@f8232d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8ee55b),segFN=segments_1,generation=1}
   [junit4]   2> 736299 T4673 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 736301 T4673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 736301 T4673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 736301 T4673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 736302 T4673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 736302 T4673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 736302 T4673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 736302 T4673 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 736303 T4673 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 736303 T4673 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 736303 T4673 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 736304 T4673 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 736304 T4673 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 736305 T4673 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 736305 T4673 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 736305 T4673 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 736306 T4673 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 736306 T4673 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 736307 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 736307 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 736308 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 736308 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 736308 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 736309 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 736309 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 736309 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 736310 T4673 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 736310 T4673 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 736312 T4673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 736312 T4673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 736313 T4673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 736314 T4673 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 736315 T4673 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 736316 T4673 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 736316 T4673 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 736317 T4673 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8010367934600408]
   [junit4]   2> 736318 T4673 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@f8232d lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8ee55b),segFN=segments_1,generation=1}
   [junit4]   2> 736318 T4673 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 736318 T4673 oass.SolrIndexSearcher.<init> Opening Searcher@d9a242[collection1] main
   [junit4]   2> 736318 T4673 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 736319 T4673 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 736320 T4673 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 736320 T4673 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 736320 T4673 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 736320 T4673 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 736321 T4673 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 736321 T4673 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 736321 T4673 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 736321 T4673 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 736322 T4674 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d9a242[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 736322 T4673 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 736323 T4677 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44387/a_gcu collection:collection1 shard:shard2
   [junit4]   2> 736323 T4616 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 736324 T4677 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 736324 T4616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 736326 T4677 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 736327 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 736328 T4677 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 736328 T4677 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 736328 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 4
   [junit4]   2> ASYNC  NEW_CORE C772 name=collection1 org.apache.solr.core.SolrCore@9bae9c url=http://127.0.0.1:44387/a_gcu/collection1 node=127.0.0.1:44387_a_gcu C772_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:44387/a_gcu, node_name=127.0.0.1:44387_a_gcu, state=down}
   [junit4]   2> 736328 T4677 C772 P44387 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44387/a_gcu/collection1/
   [junit4]   2> 736328 T4677 C772 P44387 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 736329 T4677 C772 P44387 oasc.SyncStrategy.syncToMe http://127.0.0.1:44387/a_gcu/collection1/ has no replicas
   [junit4]   2> 736329 T4677 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44387/a_gcu/collection1/ shard2
   [junit4]   2> 736329 T4677 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 736334 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 736335 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44387/a_gcu",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 4
   [junit4]   2> 736438 T4655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736438 T4644 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736438 T4672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736485 T4677 oasc.ZkController.register We are http://127.0.0.1:44387/a_gcu/collection1/ and leader is http://127.0.0.1:44387/a_gcu/collection1/
   [junit4]   2> 736486 T4677 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44387/a_gcu
   [junit4]   2> 736486 T4677 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 736486 T4677 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 736486 T4677 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 736487 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 736488 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44387/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:44387_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 736488 T4645 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44387/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:44387_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 736591 T4672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736591 T4655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736591 T4644 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 736599 T4616 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 736600 T4616 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 736602 T4616 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@ac561d{/a_gcu,null,AVAILABLE}
   [junit4]   2> 736603 T4616 oejs.AbstractConnector.doStart Started ServerConnector@10bbb11{HTTP/1.1}{127.0.0.1:42753}
   [junit4]   2> 736603 T4616 oejs.Server.doStart Started @737823ms
   [junit4]   2> 736604 T4616 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@387f44
   [junit4]   2> 736605 T4616 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 736605 T4616 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-004
   [junit4]   2> 736605 T4616 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-004/'
   [junit4]   2> 736623 T4616 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-004/solr.xml
   [junit4]   2> 736633 T4616 oasc.CoreContainer.<init> New CoreContainer 17585890
   [junit4]   2> 736634 T4616 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-004/]
   [junit4]   2> 736635 T4616 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 736635 T4616 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 736635 T4616 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 736636 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 736636 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 736637 T4616 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 736637 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 736638 T4616 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 736638 T4616 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 736638 T4616 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 736639 T4616 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 736640 T4616 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 736640 T4616 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 736641 T4616 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 736641 T4616 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 736641 T4616 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:33255/solr
   [junit4]   2> 736642 T4616 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 736642 T4616 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 736644 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 736644 T4691 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@83cfea name:ZooKeeperConnection Watcher:127.0.0.1:33255 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 736644 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 736645 T4616 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 736647 T4616 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 736648 T4694 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f5bff4 name:ZooKeeperConnection Watcher:127.0.0.1:33255/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 736648 T4616 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 736652 T4616 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 737655 T4616 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42753_a_gcu
   [junit4]   2> 737656 T4616 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42753_a_gcu
   [junit4]   2> 737660 T4616 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 737663 T4695 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 737663 T4695 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 737663 T4695 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 737663 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 737668 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42753/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:42753_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 6
   [junit4]   2> 737669 T4645 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:42753/a_gcu",
   [junit4]   2> 	  "node_name":"127.0.0.1:42753_a_gcu",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 737669 T4645 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 737669 T4645 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 737772 T4644 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 737772 T4694 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 737772 T4672 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 737772 T4655 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 738664 T4695 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 738664 T4695 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 738665 T4695 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 738665 T4695 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 738665 T4695 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 738665 T4695 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-004/collection1/'
   [junit4]   2> 738666 T4695 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 738666 T4695 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 738666 T4695 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest%203BA9F288B360E2BB-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 738678 T4695 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 738679 T4695 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 738683 T4695 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 738686 T4695 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 738694 T4695 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 738695 T4695 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 738699 T4695 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 738787 T4695 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 738791 T4695 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 738792 T4695 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 738799 T4695 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 738801 T4695 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 738802 T4695 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 738803 T4695 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 738803 T4695 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 738803 T4695 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 738804 T4695 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 738804 T4695 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 738804 T4695 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 738805 T4695 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 738805 T4695 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty2/
   [junit4]   2> 738805 T4695 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1e376ba
   [junit4]   2> 738806 T4695 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty2
   [junit4]   2> 738806 T4695 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty2/index/
   [junit4]   2> 738806 T4695 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 738806 T4695 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001/tempDir-001/jetty2/index
   [junit4]   2> 738806 T4695 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 738807 T4695 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@34c052 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1bd578b),segFN=segments_1,generation=1}
   [junit4]   2> 738807 T4695 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 738810 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 738810 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 738810 T4695 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 738810 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 738811 T4695 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 738811 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 738811 T4695 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 738811 T4695 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 738811 T4695 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 738812 T4695 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 738812 T4695 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 738812 T4695 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 738813 T4695 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 738813 T4695 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 738813 T4695 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 738814 T4695 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 738814 T4695 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 738814 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 738815 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 738815 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 738815 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 738816 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 738816 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 738817 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 738817 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 738817 T4695 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 738817 T4695 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 738819 T4695 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 738820 T4695 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 738820 T4695 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 738821 T4695 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 738821 T4695 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 738822 T4695 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 738822 T4695 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 738823 T4695 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8010367934600408]
   [junit4]   2> 738824 T4695 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@34c052 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1bd578b),segFN=segments_1,generation=1}
   [junit4]   2> 738824 T4695 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 738824 T4695 oass.SolrIndexSearcher.<init> Opening Searcher@d75ccc[collection1] main
   [junit4]   2> 738825 T4695 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 738825 T4695 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 738825 T4695 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 738826 T4695 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 738826 T4695 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 738826 T4695 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 738826 T4695 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 738827 T4695 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 738827 T4695 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 738827 T4695 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 738828 T4696 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@d75ccc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 738828 T4695 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 738829 T4699 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42753/a_gcu collection:collection1 shard:shard1
   [junit4]   2> 738829 T4616 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 738829 T4616 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 738829 T4699 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 738832 T4699 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 738833 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 738834 T4699 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 738834 T4699 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> ASYNC  NEW_CORE C773 name=collection1 org.apache.solr.core.SolrCore@181360 url=http://127.0.0.1:42753/a_gcu/collection1 node=127.0.0.1:42753_a_gcu C773_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:42753/a_gcu, node_name=127.0.0.1:42753_a_gcu, state=down}
   [junit4]   2> 738834 T4699 C773 P42753 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:42753/a_gcu/collection1/
   [junit4]   2> 738834 T4699 C773 P42753 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 738834 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"collection1"} current state version: 7
   [junit4]   2> 738835 T4699 C773 P42753 oasc.SyncStrategy.syncToMe http://127.0.0.1:42753/a_gcu/collection1/ has no replicas
   [junit4]   2> 738835 T4699 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:42753/a_gcu/collection1/ shard1
   [junit4]   2> 738835 T4699 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 738837 T4644 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 738838 T4645 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
 

[...truncated too long message...]

nit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 759358 T4777 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@30d20d rejected from java.util.concurrent.ThreadPoolExecutor@13fd7f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> 759359 T4777 oaz.ClientCnxn$EventThread.processEvent ERROR Error while calling watcher  java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@11028bb rejected from java.util.concurrent.ThreadPoolExecutor@13fd7f4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 15]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:260)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=3BA9F288B360E2BB -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=mt -Dtests.timezone=America/Atikokan -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] FAILURE 27.5s J1 | BasicDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: commitWithin did not work on node: http://127.0.0.1:44387/a_gcu/collection1 expected:<68> but was:<67>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3BA9F288B360E2BB:BA4F7C90C43F8287]:0)
   [junit4]    > 	at org.apache.solr.cloud.BasicDistributedZkTest.doTest(BasicDistributedZkTest.java:345)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 759369 T4616 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.BasicDistributedZkTest 3BA9F288B360E2BB-001
   [junit4]   2> 27512 T4615 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {other_tl1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), SubjectTerms_mfacet=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_l1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), multiDefault=PostingsFormat(name=MockRandom), intDefault=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_td1=PostingsFormat(name=MockRandom), n_d1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_l=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_f1=PostingsFormat(name=MockRandom), n_tl1=PostingsFormat(name=MockRandom), oddField_s=PostingsFormat(name=Memory doPackFST= false), n_tf1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), a_t=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), id=PostingsFormat(name=LuceneVarGapFixedInterval), text=PostingsFormat(name=Memory doPackFST= false), timestamp=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), foo_i=PostingsFormat(name=MockRandom), n_dt1=PostingsFormat(name=MockRandom), a_i1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), n_ti1=PostingsFormat(name=LuceneVarGapFixedInterval), rnd_b=PostingsFormat(name=LuceneVarGapFixedInterval), _version_=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), foo_b=PostingsFormat(name=Memory doPackFST= false), n_tdt1=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), foo_d=PostingsFormat(name=LuceneVarGapFixedInterval), foo_f=PostingsFormat(name=Memory doPackFST= false)}, docValues:{timestamp=DocValuesFormat(name=Direct)}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=mt, timezone=America/Atikokan
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=52776944,total=328466432
   [junit4]   2> NOTE: All tests run in this JVM: [TestZkChroot, TestFieldTypeCollectionResource, TestSolrDeletionPolicy1, StatelessScriptUpdateProcessorFactoryTest, TestConfigSets, TestFieldCollectionResource, UpdateParamsTest, TestCloudManagedSchema, TestPerFieldSimilarity, OutOfBoxZkACLAndCredentialsProvidersTest, TestHighlightDedupGrouping, ChaosMonkeySafeLeaderTest, SampleTest, URLClassifyProcessorTest, MigrateRouteKeyTest, TestHighFrequencyDictionaryFactory, CollectionsAPIDistributedZkTest, SimplePostToolTest, TestDynamicLoading, JsonLoaderTest, AddSchemaFieldsUpdateProcessorFactoryTest, TermsComponentTest, TestSolrJ, RAMDirectoryFactoryTest, TestManagedSynonymFilterFactory, ConvertedLegacyTest, AliasIntegrationTest, IndexBasedSpellCheckerTest, ChaosMonkeyNothingIsSafeTest, SpellCheckComponentTest, TestSchemaResource, WordBreakSolrSpellCheckerTest, TestValueSourceCache, TestSchemaNameResource, EnumFieldTest, TestUtils, OpenCloseCoreStressTest, SynonymTokenizerTest, BasicZkTest, DebugComponentTest, TestRealTimeGet, TermVectorComponentDistributedTest, ResponseLogComponentTest, CSVRequestHandlerTest, UpdateRequestProcessorFactoryTest, TestSearchPerf, OverseerRolesTest, FullSolrCloudDistribCmdsTest, TestReloadAndDeleteDocs, TestCloudInspectUtil, TermVectorComponentTest, TestCoreDiscovery, BasicFunctionalityTest, VMParamsZkACLAndCredentialsProvidersTest, MultiThreadedOCPTest, TestMaxScoreQueryParser, TestDynamicFieldResource, DistributedQueueTest, TestRawResponseWriter, TestTrie, PreAnalyzedUpdateProcessorTest, TestOrdValues, DistributedQueryComponentCustomSortTest, TestConfigOverlay, TestFastLRUCache, TestRemoteStreaming, CopyFieldTest, ShardRoutingCustomTest, SOLR749Test, SpellPossibilityIteratorTest, OverriddenZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, SharedFSAutoReplicaFailoverTest, TestExactStatsCache, SchemaVersionSpecificBehaviorTest, OverseerCollectionProcessorTest, SpatialFilterTest, TestSuggestSpellingConverter, TestDocSet, TestJmxMonitoredMap, TestRandomFaceting, PolyFieldTest, TestLMJelinekMercerSimilarityFactory, RegexBoostProcessorTest, SharedFSAutoReplicaFailoverUtilsTest, EchoParamsTest, FileBasedSpellCheckerTest, BufferStoreTest, TestTolerantSearch, TestManagedSchema, TestSolrDynamicMBean, TestManagedSchemaFieldResource, PrimitiveFieldTypeTest, TestBulkSchemaAPI, FieldMutatingUpdateProcessorTest, AtomicUpdatesTest, MBeansHandlerTest, TestFieldSortValues, HdfsBasicDistributedZkTest, TestPostingsSolrHighlighter, TestCharFilters, UUIDUpdateProcessorFallbackTest, DirectUpdateHandlerOptimizeTest, CoreMergeIndexesAdminHandlerTest, TestSolrQueryParser, TestCollationFieldDocValues, XsltUpdateRequestHandlerTest, TestRecoveryHdfs, BJQParserTest, TestReload, TestSolrXmlPersistence, BasicDistributedZkTest]
   [junit4] Completed on J1 in 27.78s, 1 test, 1 failure <<< FAILURES!

[...truncated 844 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 463 suites, 1871 tests, 1 failure, 47 ignored (21 assumptions)

Total time: 63 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_25) - Build # 11621 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11621/
Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([186BAB38009D4C12:4038CA1D1DB88328]:0)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)




Build Log:
[...truncated 13334 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001
   [junit4]   2> 4090 T18 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 4133 T18 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 4133 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/'
   [junit4]   2> 4175 T18 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 4185 T18 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 4212 T18 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 4214 T18 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 4249 T18 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 4278 T18 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 4280 T18 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 4281 T18 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4282 T18 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4282 T18 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001
   [junit4]   2> 4283 T18 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/'
   [junit4]   2> 4321 T18 oasc.CoreContainer.<init> New CoreContainer 166601073
   [junit4]   2> 4321 T18 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/]
   [junit4]   2> 4322 T18 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 4322 T18 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 4323 T18 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 4323 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 4324 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 4324 T18 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 4325 T18 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 4325 T18 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 4325 T18 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 4326 T18 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 4326 T18 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 4368 T18 oejul.Log.initialized Logging initialized @5667ms
   [junit4]   2> 4409 T18 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 4420 T18 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4421 T18 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4421 T18 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 4429 T19 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/'
   [junit4]   2> 4465 T19 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 4471 T19 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 4493 T19 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 4493 T19 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 4501 T19 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 4515 T19 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 4516 T19 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 4516 T19 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 4517 T19 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/
   [junit4]   2> 4517 T19 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4517 T19 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001/
   [junit4]   2> 4518 T19 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 4518 T19 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 4518 T19 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 4519 T19 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001
   [junit4]   2> 4519 T19 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001/index/
   [junit4]   2> 4520 T19 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 4520 T19 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001/index
   [junit4]   2> 4521 T19 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4e54334 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53ef2c7),segFN=segments_1,generation=1}
   [junit4]   2> 4521 T19 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4522 T19 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 4546 T19 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 4547 T19 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 4547 T19 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 4548 T19 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 4548 T19 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 4549 T19 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 4549 T19 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 4550 T19 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 4550 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 4551 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 4551 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 4552 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 4552 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 4553 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 4553 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 4554 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 4554 T19 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 4555 T19 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 4555 T19 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 4556 T19 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 4556 T19 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 4559 T19 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4561 T19 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4562 T19 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4564 T19 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 4565 T19 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 4566 T19 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 4566 T19 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 4567 T19 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4e54334 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53ef2c7),segFN=segments_1,generation=1}
   [junit4]   2> 4568 T19 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 4568 T19 oass.SolrIndexSearcher.<init> Opening Searcher@6a82ae7d[collection1] main
   [junit4]   2> 4569 T19 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/conf
   [junit4]   2> 4569 T19 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/conf}
   [junit4]   2> 4570 T19 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/core-home-001/collection1/conf
   [junit4]   2> 4570 T19 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 4571 T19 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 4571 T20 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@6a82ae7d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4572 T20 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 4571 T19 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 4572 T20 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6a82ae7d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4574 T18 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 4584 T18 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C2 name=collection1 org.apache.solr.core.SolrCore@633a92a8
   [junit4]   2> 4594 T18 C2 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 4597 T18 C2 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 4604 T18 C2 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 4605 T18 C2 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 4621 T18 C2 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 4622 T18 C2 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 4666 T18 C2 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4e54334 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53ef2c7),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4e54334 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@53ef2c7),segFN=segments_2,generation=2}
   [junit4]   2> 4667 T18 C2 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 4692 T18 C2 oass.SolrIndexSearcher.<init> Opening Searcher@4b18e442[collection1] main
   [junit4]   2> 4693 T20 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@4b18e442[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C28)))}
   [junit4]   2> 4698 T18 C2 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 4702 T20 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=8 
   [junit4]   2> 4703 T20 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=0 
   [junit4]   2> 4705 T20 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 4706 T20 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 4706 T20 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4b18e442[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):C28)))}
   [junit4]   2> 4707 T18 C2 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 4708 T18 C2 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 4709 T18 C2 oashd.DocBuilder.execute Time taken = 0:0:0.104
   [junit4]   2> 4709 T18 C2 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[1, 2, 3],commit=} 0 122
   [junit4]   2> 4719 T18 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=28 status=0 QTime=8 
   [junit4]   2> 4732 T18 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 4736 T18 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=11 status=0 QTime=0 
   [junit4]   2> 4748 T18 C2 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=14 status=0 QTime=1 
   [junit4]   2> 4785 T18 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 4794 T18 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 45342 T18 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=186BAB38009D4C12 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=tr -Dtests.timezone=Indian/Reunion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   40.6s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([186BAB38009D4C12:4038CA1D1DB88328]:0)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 45386 T18 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 45386 T18 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=166601073
   [junit4]   2> 45387 T18 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@633a92a8
   [junit4]   2> 45387 T18 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 45387 T18 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 45388 T18 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 45388 T18 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 45389 T18 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 45389 T18 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 45390 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 45390 T18 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001/index
   [junit4]   2> 45390 T18 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001;done=false>>]
   [junit4]   2> 45391 T18 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 186BAB38009D4C12-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=PostingsFormat(name=LuceneFixedGap), _root_=PostingsFormat(name=LuceneFixedGap), id=PostingsFormat(name=LuceneVarGapFixedInterval), desc=PostingsFormat(name=Memory doPackFST= false), timestamp=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=tr, timezone=Indian/Reunion
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=34282832,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestWriterImpl, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 41.42s, 2 tests, 1 error <<< FAILURES!

[...truncated 46 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150117_003542_787.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid24861.hprof ...
   [junit4] Heap dump file created [797755131 bytes in 8.194 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 64 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_25 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_40-ea-b20) - Build # 11620 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11620/
Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.core.TestDynamicLoading.testDistribSearch

Error Message:
Could not successfully add blob after 150 attempts. Expecting 2 items. time elapsed 15,554  output  for url is {   "responseHeader":{     "status":0,     "QTime":0},   "response":{     "numFound":1,     "start":0,     "docs":[{         "id":"test/1",         "md5":"f87f983cc90b6e64a367bf5b8ab5cc98",         "blobName":"test",         "version":1,         "timestamp":"2015-01-16T21:55:25.014Z",         "size":5365}]}}

Stack Trace:
java.lang.AssertionError: Could not successfully add blob after 150 attempts. Expecting 2 items. time elapsed 15,554  output  for url is {
  "responseHeader":{
    "status":0,
    "QTime":0},
  "response":{
    "numFound":1,
    "start":0,
    "docs":[{
        "id":"test/1",
        "md5":"f87f983cc90b6e64a367bf5b8ab5cc98",
        "blobName":"test",
        "version":1,
        "timestamp":"2015-01-16T21:55:25.014Z",
        "size":5365}]}}
	at __randomizedtesting.SeedInfo.seed([EB4D2B326838CD9B:6AABA52A1F67ADA7]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.handler.TestBlobHandler.postAndCheck(TestBlobHandler.java:150)
	at org.apache.solr.core.TestDynamicLoading.dynamicLoading(TestDynamicLoading.java:115)
	at org.apache.solr.core.TestDynamicLoading.doTest(TestDynamicLoading.java:71)
	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 9259 lines...]
   [junit4] Suite: org.apache.solr.core.TestDynamicLoading
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/init-core-data-001
   [junit4]   2> 803172 T3667 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 803172 T3667 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /jr_k/w
   [junit4]   2> 803174 T3667 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> 803174 T3667 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 803175 T3668 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 803275 T3667 oasc.ZkTestServer.run start zk server on port:43896
   [junit4]   2> 803275 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 803276 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 803292 T3675 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7e64b7df name:ZooKeeperConnection Watcher:127.0.0.1:43896 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 803293 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 803294 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 803294 T3667 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 803298 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 803299 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 803301 T3678 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4f3de970 name:ZooKeeperConnection Watcher:127.0.0.1:43896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 803301 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 803302 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 803302 T3667 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 803304 T3667 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 803307 T3667 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 803309 T3667 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 803313 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 803313 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 803318 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 803318 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 803322 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 803323 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 803326 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 803327 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 803330 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 803330 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 803333 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 803333 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 803335 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 803336 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 803338 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 803339 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 803341 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 803342 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 803345 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 803346 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 803348 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 803348 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 803600 T3667 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 803601 T3667 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@5449d45f{/jr_k/w,null,AVAILABLE}
   [junit4]   2> 803603 T3667 oejs.AbstractConnector.doStart Started ServerConnector@4a8b01f8{HTTP/1.1}{127.0.0.1:50786}
   [junit4]   2> 803603 T3667 oejs.Server.doStart Started @804978ms
   [junit4]   2> 803604 T3667 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 803604 T3667 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 803605 T3667 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-002
   [junit4]   2> 803605 T3667 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-002/'
   [junit4]   2> 803617 T3667 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-002/solr.xml
   [junit4]   2> 803625 T3667 oasc.CoreContainer.<init> New CoreContainer 954249208
   [junit4]   2> 803626 T3667 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-002/]
   [junit4]   2> 803626 T3667 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 803626 T3667 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 803626 T3667 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 803627 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 803627 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 803627 T3667 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 803627 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 803628 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 803628 T3667 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 803628 T3667 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 803628 T3667 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 803629 T3667 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 803629 T3667 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 803629 T3667 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 803629 T3667 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 803630 T3667 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43896/solr
   [junit4]   2> 803630 T3667 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 803630 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 803631 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 803632 T3692 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1d62a0fe name:ZooKeeperConnection Watcher:127.0.0.1:43896 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 803633 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 803633 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 803635 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 803636 T3695 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c16c64b name:ZooKeeperConnection Watcher:127.0.0.1:43896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 803636 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 803637 T3667 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 803640 T3667 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 803641 T3667 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-running
   [junit4]   2> 803642 T3667 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-completed
   [junit4]   2> 803644 T3667 oascc.SolrZkClient.makePath makePath: /overseer/collection-map-failure
   [junit4]   2> 803645 T3667 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 803646 T3667 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50786_jr_k%2Fw
   [junit4]   2> 803646 T3667 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50786_jr_k%2Fw
   [junit4]   2> 803647 T3667 oascc.SolrZkClient.makePath makePath: /overseer_elect
   [junit4]   2> 803648 T3667 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 803649 T3667 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 803650 T3667 oasc.OverseerElectionContext.runLeaderProcess I am going to be the leader 127.0.0.1:50786_jr_k%2Fw
   [junit4]   2> 803650 T3667 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 803651 T3667 oasc.Overseer.start Overseer (id=93155839864537091-127.0.0.1:50786_jr_k%2Fw-n_0000000000) starting
   [junit4]   2> 803652 T3667 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 803655 T3667 oasc.OverseerAutoReplicaFailoverThread.<init> Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 803656 T3697 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 803656 T3667 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 803657 T3667 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 803658 T3667 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 803660 T3696 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 803661 T3699 oasc.ZkController.publish publishing core=collection1 state=down collection=control_collection
   [junit4]   2> 803661 T3699 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 803661 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 803661 T3699 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 803663 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 0, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50786/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:50786_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 0
   [junit4]   2> 803663 T3696 oasco.ReplicaMutator.updateState Update state numShards=1 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50786/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:50786_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"1",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 803663 T3696 oasco.ClusterStateMutator.createCollection building a new cName: control_collection
   [junit4]   2> 803663 T3696 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 803664 T3695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 804662 T3699 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 804662 T3699 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 804662 T3699 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 804663 T3699 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 804663 T3699 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 804663 T3699 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-002/collection1/'
   [junit4]   2> 804664 T3699 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-002/collection1/lib/.svn/' to classloader
   [junit4]   2> 804664 T3699 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-002/collection1/lib/classes/' to classloader
   [junit4]   2> 804664 T3699 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-002/collection1/lib/README' to classloader
   [junit4]   2> 804675 T3699 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 804677 T3699 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 804680 T3699 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 804685 T3699 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 804694 T3699 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 804695 T3699 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 804699 T3699 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 804818 T3699 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 804822 T3699 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 804823 T3699 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 804830 T3699 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 804832 T3699 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 804833 T3699 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 804834 T3699 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 804834 T3699 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 804835 T3699 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 804835 T3699 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 804835 T3699 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 804835 T3699 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 804836 T3699 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 804836 T3699 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-002/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/control/data/
   [junit4]   2> 804836 T3699 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cf8b47f
   [junit4]   2> 804837 T3699 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/control/data
   [junit4]   2> 804837 T3699 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/control/data/index/
   [junit4]   2> 804837 T3699 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 804837 T3699 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/control/data/index
   [junit4]   2> 804837 T3699 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1197110, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 804838 T3699 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4dc42daa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7522c702),segFN=segments_1,generation=1}
   [junit4]   2> 804838 T3699 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 804840 T3699 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 804841 T3699 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 804841 T3699 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 804841 T3699 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 804841 T3699 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 804842 T3699 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 804842 T3699 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 804842 T3699 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 804842 T3699 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 804843 T3699 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 804843 T3699 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 804843 T3699 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 804843 T3699 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 804844 T3699 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 804844 T3699 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 804844 T3699 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 804844 T3699 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 804845 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 804845 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 804845 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 804846 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 804846 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 804846 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 804846 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 804847 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 804847 T3699 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 804847 T3699 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 804848 T3699 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 804849 T3699 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 804850 T3699 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 804850 T3699 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 804851 T3699 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 804852 T3699 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 804852 T3699 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 804852 T3699 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 804852 T3699 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@4dc42daa lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7522c702),segFN=segments_1,generation=1}
   [junit4]   2> 804853 T3699 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 804853 T3699 oass.SolrIndexSearcher.<init> Opening Searcher@736d92f8[collection1] main
   [junit4]   2> 804853 T3699 oascc.ZkStateReader.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 804854 T3699 oascc.ZkStateReader.readConfigName path=/collections/control_collection configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 804854 T3699 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 804854 T3699 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 804854 T3699 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 804854 T3699 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 804855 T3699 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 804855 T3699 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 804855 T3699 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 804855 T3699 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 804856 T3699 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 804856 T3700 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@736d92f8[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 804856 T3703 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50786/jr_k/w collection:control_collection shard:shard1
   [junit4]   2> 804857 T3667 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 804857 T3667 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 804857 T3703 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 804857 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 804858 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 804859 T3706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@259ba57d name:ZooKeeperConnection Watcher:127.0.0.1:43896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 804859 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 804860 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 804860 T3667 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 804860 T3703 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 804862 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 804862 T3703 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 804862 T3703 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 804862 T3667 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> ASYNC  NEW_CORE C1741 name=collection1 org.apache.solr.core.SolrCore@740279ad url=http://127.0.0.1:50786/jr_k/w/collection1 node=127.0.0.1:50786_jr_k%2Fw C1741_STATE=coll:control_collection core:collection1 props:{core=collection1, base_url=http://127.0.0.1:50786/jr_k/w, node_name=127.0.0.1:50786_jr_k%2Fw, state=down}
   [junit4]   2> 804862 T3703 C1741 P50786 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50786/jr_k/w/collection1/
   [junit4]   2> 804863 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection"} current state version: 1
   [junit4]   2> 804863 T3667 oasc.AbstractFullDistribZkTestBase.createJettys Creating collection1 with stateFormat=2
   [junit4]   2> 804863 T3703 C1741 P50786 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 804863 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 804863 T3703 C1741 P50786 oasc.SyncStrategy.syncToMe http://127.0.0.1:50786/jr_k/w/collection1/ has no replicas
   [junit4]   2> 804864 T3703 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50786/jr_k/w/collection1/ shard1
   [junit4]   2> 804864 T3703 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 804864 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 804865 T3709 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a49b1b8 name:ZooKeeperConnection Watcher:127.0.0.1:43896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 804865 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 804865 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 804866 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 804867 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 2, message = {
   [junit4]   2> 	  "operation":"create",
   [junit4]   2> 	  "name":"collection1",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "stateFormat":2} current state version: 1
   [junit4]   2> 804868 T3696 oasco.ClusterStateMutator.createCollection building a new cName: collection1
   [junit4]   2> 804873 T3695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 804874 T3706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 804877 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50786/jr_k/w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 2
   [junit4]   2> 804878 T3696 oasco.ZkStateWriter.writePendingUpdates going to create_collection /collections/collection1/state.json
   [junit4]   2> 804879 T3706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 804881 T3710 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 804881 T3706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 804917 T3703 oasc.ZkController.register We are http://127.0.0.1:50786/jr_k/w/collection1/ and leader is http://127.0.0.1:50786/jr_k/w/collection1/
   [junit4]   2> 804918 T3703 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50786/jr_k/w
   [junit4]   2> 804918 T3703 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 804918 T3703 oasc.ZkController.publish publishing core=collection1 state=active collection=control_collection
   [junit4]   2> 804918 T3703 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 804919 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 804920 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50786/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:50786_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"} current state version: 4
   [junit4]   2> 804920 T3696 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:50786/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:50786_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard1",
   [junit4]   2> 	  "collection":"control_collection",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 805022 T3695 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 805023 T3706 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 805128 T3667 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 1
   [junit4]   2> 805129 T3667 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 805130 T3667 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@4058fa5{/jr_k/w,null,AVAILABLE}
   [junit4]   2> 805131 T3667 oejs.AbstractConnector.doStart Started ServerConnector@bf4a163{HTTP/1.1}{127.0.0.1:44443}
   [junit4]   2> 805131 T3667 oejs.Server.doStart Started @806506ms
   [junit4]   2> 805132 T3667 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 805132 T3667 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 805132 T3667 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-003
   [junit4]   2> 805133 T3667 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-003/'
   [junit4]   2> 805144 T3667 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-003/solr.xml
   [junit4]   2> 805153 T3667 oasc.CoreContainer.<init> New CoreContainer 1056890940
   [junit4]   2> 805153 T3667 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-003/]
   [junit4]   2> 805154 T3667 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 805154 T3667 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 805154 T3667 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 805154 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 805155 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 805155 T3667 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 805155 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 805155 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 805155 T3667 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 805156 T3667 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 805156 T3667 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 805156 T3667 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 805157 T3667 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 805157 T3667 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 805157 T3667 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 805157 T3667 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43896/solr
   [junit4]   2> 805158 T3667 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 805158 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 805158 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 805160 T3724 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5fb03401 name:ZooKeeperConnection Watcher:127.0.0.1:43896 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 805160 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 805161 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 805162 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 805163 T3727 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4eea615b name:ZooKeeperConnection Watcher:127.0.0.1:43896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 805163 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 805166 T3667 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 806169 T3667 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44443_jr_k%2Fw
   [junit4]   2> 806170 T3667 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44443_jr_k%2Fw
   [junit4]   2> 806173 T3667 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 806176 T3728 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 806176 T3728 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 806177 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 806177 T3728 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 806177 T3728 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 806178 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44443/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:44443_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 806178 T3728 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 806178 T3696 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44443/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:44443_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 806178 T3696 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 806179 T3696 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 806179 T3728 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 0 
   [junit4]   2> 806179 T3728 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 806281 T3696 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 806281 T3727 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 806282 T3727 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 1 
   [junit4]   2> 807179 T3728 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 807179 T3728 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 807180 T3728 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 807180 T3728 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 807181 T3728 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 807181 T3728 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-003/collection1/'
   [junit4]   2> 807182 T3728 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-003/collection1/lib/.svn/' to classloader
   [junit4]   2> 807182 T3728 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-003/collection1/lib/classes/' to classloader
   [junit4]   2> 807182 T3728 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-003/collection1/lib/README' to classloader
   [junit4]   2> 807195 T3728 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 807197 T3728 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 807201 T3728 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 807205 T3728 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 807215 T3728 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 807216 T3728 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 807221 T3728 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 807322 T3728 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 807326 T3728 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 807327 T3728 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 807334 T3728 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 807336 T3728 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 807337 T3728 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 807338 T3728 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 807339 T3728 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 807339 T3728 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 807339 T3728 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 807339 T3728 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 807340 T3728 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 807340 T3728 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 807340 T3728 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-003/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty1/
   [junit4]   2> 807340 T3728 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cf8b47f
   [junit4]   2> 807341 T3728 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty1
   [junit4]   2> 807341 T3728 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty1/index/
   [junit4]   2> 807341 T3728 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 807341 T3728 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty1/index
   [junit4]   2> 807342 T3728 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1197110, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 807342 T3728 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1513c926 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d87be96),segFN=segments_1,generation=1}
   [junit4]   2> 807342 T3728 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 807344 T3728 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 807345 T3728 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 807345 T3728 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 807345 T3728 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 807345 T3728 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 807345 T3728 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 807345 T3728 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 807346 T3728 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 807346 T3728 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 807346 T3728 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 807346 T3728 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 807347 T3728 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 807347 T3728 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 807347 T3728 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 807347 T3728 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 807348 T3728 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 807348 T3728 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 807348 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 807349 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 807350 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 807350 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 807350 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 807351 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 807351 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 807351 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 807352 T3728 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 807352 T3728 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 807353 T3728 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 807354 T3728 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 807355 T3728 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 807355 T3728 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 807356 T3728 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 807356 T3728 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 807357 T3728 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 807357 T3728 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 807357 T3728 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1513c926 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2d87be96),segFN=segments_1,generation=1}
   [junit4]   2> 807358 T3728 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 807358 T3728 oass.SolrIndexSearcher.<init> Opening Searcher@37a0128d[collection1] main
   [junit4]   2> 807358 T3728 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 807359 T3728 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 807359 T3728 oasr.ManagedResourceStorage.newStorageIO Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 807359 T3728 oasr.ManagedResourceStorage$ZooKeeperStorageIO.configure Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 807359 T3728 oasr.RestManager.init Initializing RestManager with initArgs: {}
   [junit4]   2> 807359 T3728 oasr.ManagedResourceStorage.load Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 807360 T3728 oasr.ManagedResourceStorage$ZooKeeperStorageIO.openInputStream No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 807360 T3728 oasr.ManagedResourceStorage.load Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 807360 T3728 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 807360 T3728 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 807361 T3729 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@37a0128d[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 807361 T3728 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 807362 T3732 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44443/jr_k/w collection:collection1 shard:shard2
   [junit4]   2> 807362 T3667 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 807362 T3667 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 807362 T3732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 807366 T3732 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 807367 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 807367 T3732 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 807367 T3732 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 807367 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1"} current state version: 5
   [junit4]   2> ASYNC  NEW_CORE C1742 name=collection1 org.apache.solr.core.SolrCore@541a5e75 url=http://127.0.0.1:44443/jr_k/w/collection1 node=127.0.0.1:44443_jr_k%2Fw C1742_STATE=coll:collection1 core:collection1 props:{core=collection1, base_url=http://127.0.0.1:44443/jr_k/w, node_name=127.0.0.1:44443_jr_k%2Fw, state=down}
   [junit4]   2> 807367 T3732 C1742 P44443 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44443/jr_k/w/collection1/
   [junit4]   2> 807368 T3732 C1742 P44443 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 807368 T3732 C1742 P44443 oasc.SyncStrategy.syncToMe http://127.0.0.1:44443/jr_k/w/collection1/ has no replicas
   [junit4]   2> 807368 T3732 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44443/jr_k/w/collection1/ shard2
   [junit4]   2> 807368 T3696 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 1
   [junit4]   2> 807368 T3732 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 807369 T3727 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 807369 T3727 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 2 
   [junit4]   2> 807372 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 807372 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "operation":"leader",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44443/jr_k/w",
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "state":"active"} current state version: 5
   [junit4]   2> 807373 T3696 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 2
   [junit4]   2> 807374 T3727 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 807374 T3727 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 807422 T3732 oasc.ZkController.register We are http://127.0.0.1:44443/jr_k/w/collection1/ and leader is http://127.0.0.1:44443/jr_k/w/collection1/
   [junit4]   2> 807423 T3732 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44443/jr_k/w
   [junit4]   2> 807423 T3732 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 807423 T3732 oasc.ZkController.publish publishing core=collection1 state=active collection=collection1
   [junit4]   2> 807423 T3732 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 807424 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 807425 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44443/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:44443_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 807425 T3696 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "core_node_name":"core_node1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44443/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:44443_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"active",
   [junit4]   2> 	  "shard":"shard2",
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 807426 T3696 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 3
   [junit4]   2> 807426 T3732 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 3 
   [junit4]   2> 807426 T3727 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 807427 T3727 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 807635 T3667 oasc.AbstractFullDistribZkTestBase.createJettys create jetty 2
   [junit4]   2> 807636 T3667 oejs.Server.doStart jetty-9.2.6.v20141205
   [junit4]   2> 807637 T3667 oejsh.ContextHandler.doStart Started o.e.j.s.ServletContextHandler@54152968{/jr_k/w,null,AVAILABLE}
   [junit4]   2> 807638 T3667 oejs.AbstractConnector.doStart Started ServerConnector@52f3b4ed{HTTP/1.1}{127.0.0.1:44683}
   [junit4]   2> 807638 T3667 oejs.Server.doStart Started @809013ms
   [junit4]   2> 807638 T3667 oass.SolrDispatchFilter.init SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
   [junit4]   2> 807639 T3667 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 807639 T3667 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-004
   [junit4]   2> 807639 T3667 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-004/'
   [junit4]   2> 807650 T3667 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-004/solr.xml
   [junit4]   2> 807658 T3667 oasc.CoreContainer.<init> New CoreContainer 2141062532
   [junit4]   2> 807659 T3667 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-004/]
   [junit4]   2> 807659 T3667 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 90000
   [junit4]   2> 807659 T3667 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: 
   [junit4]   2> 807660 T3667 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 807660 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 807660 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 807660 T3667 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 807661 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 807661 T3667 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 807661 T3667 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 807661 T3667 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 807661 T3667 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 807662 T3667 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 807662 T3667 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 807662 T3667 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 807663 T3667 oasc.CoreContainer.load Host Name: 127.0.0.1
   [junit4]   2> 807663 T3667 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43896/solr
   [junit4]   2> 807663 T3667 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 807663 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 807664 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 807665 T3746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@64db6497 name:ZooKeeperConnection Watcher:127.0.0.1:43896 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 807666 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 807666 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 807668 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 807669 T3749 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@155b9506 name:ZooKeeperConnection Watcher:127.0.0.1:43896/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 807669 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 807673 T3667 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
   [junit4]   2> 808676 T3667 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44683_jr_k%2Fw
   [junit4]   2> 808677 T3667 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44683_jr_k%2Fw
   [junit4]   2> 808681 T3667 oasc.Overseer.close Overseer (id=null) closing
   [junit4]   2> 808685 T3750 oasc.ZkController.publish publishing core=collection1 state=down collection=collection1
   [junit4]   2> 808685 T3750 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 808686 T3695 oasc.DistributedQueue$LatchWatcher.process NodeChildrenChanged fired on path /overseer/queue state SyncConnected
   [junit4]   2> 808686 T3750 oasc.ZkController.preRegister Registering watch for external collection collection1
   [junit4]   2> 808687 T3750 oascc.ZkStateReader.addZkWatch addZkWatch collection1
   [junit4]   2> 808687 T3696 oasc.Overseer$ClusterStateUpdater.run processMessage: queueSize: 1, message = {
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44683/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:44683_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"} current state version: 5
   [junit4]   2> 808687 T3750 oascc.ZkStateReader.addZkWatch Updating collection state at /collections/collection1/state.json from ZooKeeper... 
   [junit4]   2> 808687 T3696 oasco.ReplicaMutator.updateState Update state numShards=2 message={
   [junit4]   2> 	  "core":"collection1",
   [junit4]   2> 	  "roles":null,
   [junit4]   2> 	  "base_url":"http://127.0.0.1:44683/jr_k/w",
   [junit4]   2> 	  "node_name":"127.0.0.1:44683_jr_k%2Fw",
   [junit4]   2> 	  "numShards":"2",
   [junit4]   2> 	  "state":"down",
   [junit4]   2> 	  "shard":null,
   [junit4]   2> 	  "collection":"collection1",
   [junit4]   2> 	  "operation":"state"}
   [junit4]   2> 808688 T3696 oasco.ReplicaMutator.updateState Collection already exists with numShards=2
   [junit4]   2> 808688 T3750 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 4 
   [junit4]   2> 808688 T3696 oasco.ReplicaMutator.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 808688 T3750 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 808689 T3696 oasco.ZkStateWriter.writePendingUpdates going to update_collection /collections/collection1/state.json version: 4
   [junit4]   2> 808689 T3749 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 808689 T3727 oascc.ZkStateReader$7.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json, has occurred - updating... 
   [junit4]   2> 808690 T3749 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 808690 T3727 oascc.ZkStateReader.updateWatchedCollection Updating data for collection1 to ver 5 
   [junit4]   2> 809689 T3750 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 809689 T3750 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 809689 T3750 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 809689 T3750 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 809690 T3750 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 809691 T3750 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-004/collection1/'
   [junit4]   2> 809691 T3750 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-004/collection1/lib/.svn/' to classloader
   [junit4]   2> 809691 T3750 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-004/collection1/lib/classes/' to classloader
   [junit4]   2> 809692 T3750 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading%20EB4D2B326838CD9B-001/tempDir-004/collection1/lib/README' to classloader
   [junit4]   2> 809702 T3750 oasc.ZkController.watchZKConfDir watch zkdir /configs/conf1
   [junit4]   2> 809704 T3750 oasc.Config.<init> loaded config solrconfig.xml with version 0 
   [junit4]   2> 809708 T3750 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 809712 T3750 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 809721 T3750 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 809722 T3750 oass.IndexSchema.readSchema Reading Solr Schema from /configs/conf1/schema.xml
   [junit4]   2> 809726 T3750 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 809841 T3750 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 809844 T3750 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 809845 T3750 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 809852 T3750 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 809854 T3750 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 809855 T3750 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 809855 T3750 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 809856 T3750 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 809856 T3750 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 809857 T3750 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 809857 T3750 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 809857 T3750 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 809858 T3750 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 809858 T3750 oasc.SolrCore.<init> [collection1] Opening new SolrCore at ../../../../../../../../../../mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-004/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty2/
   [junit4]   2> 809858 T3750 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2cf8b47f
   [junit4]   2> 809859 T3750 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty2
   [junit4]   2> 809859 T3750 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty2/index/
   [junit4]   2> 809859 T3750 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 809859 T3750 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001/tempDir-001/jetty2/index
   [junit4]   2> 809860 T3750 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1197110, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 809860 T3750 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1e56e127 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7315290a),segFN=segments_1,generation=1}
   [junit4]   2> 809860 T3750 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 809863 T3750 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 809863 T3750 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 809863 T3750 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 809863 T3750 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 809863 T3750 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 809863 T3750 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 809864 T3750 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 809864 T3750 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 809864 T3750 oasc.SolrCore.loadUpdateProcessorChains no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 809864 T3750 oasc.RequestHandlers.initHandlersFromConfig created /update: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 809865 T3750 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 809865 T3750 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 809865 T3750 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 809865 T3750 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 809866 T3750 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 809866 T3750 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 809866 T3750 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 809867 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 809867 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 809867 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 809868 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 809868 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 809868 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 809869 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 809869 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 809869 T3750 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 809870 T3750 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: solr.admin.ShowFileRequestHandler
   [junit4]   2> 809871 T3750 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 809872 T3750 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 809872 T3750 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 809873 T3750 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 809874 T3750 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 809874 T3750 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 809874 T3750 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 809875 T3750 oasu.RandomMergePolicy.<init> RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 809875 T3750 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1e56e127 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7315290a),segFN=segments_1,generation=1}
   [junit4]   2> 809875 T3750 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 809876 T3750 oass.SolrIndexSearcher.<init> Opening Searcher@30471a80[collection1] main
   [junit4]   2> 809876 T3750 oascc.ZkStateReader.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 809877 T3750 oascc.ZkStateReader.readConfigName path=/collections/collection1 configName=conf1 specified config exists in ZooKeeper
   [junit4]   2> 809877 T3750 oasr.ManagedResourceStorage.newStorag

[...truncated too long message...]

nnected Client is connected to ZooKeeper
   [junit4]   2> 841817 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 841817 T3667 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 841820 T3667 oascc.SolrZkClient.createZkCredentialsToAddAutomatically Using default ZkCredentialsProvider
   [junit4]   2> 841820 T3667 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 841822 T3875 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1b206552 name:ZooKeeperConnection Watcher:127.0.0.1:36533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 841822 T3667 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 841822 T3667 oascc.SolrZkClient.createZkACLProvider Using default ZkACLProvider
   [junit4]   2> 841823 T3667 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 841824 T3667 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 841825 T3667 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 841827 T3667 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 841828 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 841828 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 841830 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 841831 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 841832 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 841833 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 841834 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 841835 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 841836 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 841836 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 841838 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 841838 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 841840 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 841841 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 841843 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 841843 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 841844 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 841844 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 841846 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 841846 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 841847 T3667 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 841848 T3667 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 841850 T3667 oas.SolrTestCaseJ4.tearDown ###Ending testDown
   [junit4]   2> 841850 T3667 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36533 36533
   [junit4]   2> 847267 T3865 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36533 36533
   [junit4]   2> 847269 T3667 oas.SolrTestCaseJ4.tearDown ###Ending testDown
   [junit4]   2> 847269 T3667 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36533 36533
   [junit4]   2> 847269 T3667 oas.SolrTestCaseJ4.tearDown ###Ending testDown
   [junit4]   2> 847269 T3667 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:36533 36533
   [junit4]   2> 847271 T3667 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/temp/solr.core.TestDynamicLoading EB4D2B326838CD9B-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {blobName=PostingsFormat(name=LuceneVarGapFixedInterval), size=FSTOrd50, _version_=PostingsFormat(name=MockRandom), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), version=PostingsFormat(name=MockRandom), md5=PostingsFormat(name=LuceneVarGapFixedInterval), timestamp=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{}, sim=DefaultSimilarity, locale=ar_JO, timezone=Europe/Simferopol
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_40-ea (64-bit)/cpus=8,threads=1,free=120287072,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [SpellCheckComponentTest, CollectionsAPIAsyncDistributedZkTest, TestBadConfig, QueryParsingTest, TestIntervalFaceting, TestSolrXml, HdfsCollectionsAPIDistributedZkTest, EchoParamsTest, DebugComponentTest, TestExactStatsCache, DistribCursorPagingTest, UpdateParamsTest, TestBulkSchemaAPI, TestTrie, PolyFieldTest, CollectionsAPIDistributedZkTest, ZkStateWriterTest, SignatureUpdateProcessorFactoryTest, DistributedExpandComponentTest, TestDynamicFieldCollectionResource, FileUtilsTest, StatsComponentTest, SearchHandlerTest, ZkSolrClientTest, OverriddenZkACLAndCredentialsProvidersTest, TestSolrIndexConfig, TestFieldCollectionResource, FieldMutatingUpdateProcessorTest, TestLMJelinekMercerSimilarityFactory, TestZkChroot, SuggesterTSTTest, TestMergePolicyConfig, TestExpandComponent, DocumentBuilderTest, SimpleMLTQParserTest, HighlighterMaxOffsetTest, TestRealTimeGet, TestFreeTextSuggestions, AliasIntegrationTest, CurrencyFieldOpenExchangeTest, UUIDUpdateProcessorFallbackTest, ResponseLogComponentTest, OverseerCollectionProcessorTest, QueryResultKeyTest, TestChildDocTransformer, TimeZoneUtilsTest, CopyFieldTest, CurrencyFieldXmlFileTest, TestFastLRUCache, TestSearchPerf, TestDefaultSimilarityFactory, TestReversedWildcardFilterFactory, OverseerStatusTest, TestCloudSchemaless, OutOfBoxZkACLAndCredentialsProvidersTest, TestSchemaNameResource, TestRangeQuery, TermsComponentTest, EnumFieldTest, StatelessScriptUpdateProcessorFactoryTest, IndexBasedSpellCheckerTest, TestJmxMonitoredMap, ZkNodePropsTest, TestConfigOverlay, TestConfigSets, TestManagedStopFilterFactory, AnalyticsMergeStrategyTest, ConvertedLegacyTest, StressHdfsTest, FastVectorHighlighterTest, SpatialFilterTest, WordBreakSolrSpellCheckerTest, DistribDocExpirationUpdateProcessorTest, MigrateRouteKeyTest, OpenCloseCoreStressTest, TestOrdValues, TestCollationFieldDocValues, DistributedTermsComponentTest, BasicDistributedZk2Test, TestSolrXmlPersistence, CSVRequestHandlerTest, SpellingQueryConverterTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestDynamicLoading]
   [junit4] Completed on J1 in 44.12s, 2 tests, 1 failure <<< FAILURES!

[...truncated 874 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:187: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 463 suites, 1871 tests, 1 failure, 47 ignored (21 assumptions)

Total time: 63 minutes 42 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_40-ea-b20 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build # 11619 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11619/
Java: 32bit/jdk1.8.0_25 -client -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([884182025F52F548:D012E32742773A72]:0)
	at org.apache.solr.handler.dataimport.AbstractDataImportHandlerTestCase.createMap(AbstractDataImportHandlerTestCase.java:157)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)




Build Log:
[...truncated 13432 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001
   [junit4]   2> 3598 T16 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 3627 T16 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 3627 T16 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/'
   [junit4]   2> 3664 T16 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 3684 T16 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3714 T16 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 3715 T16 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 3728 T16 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 3756 T16 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 3757 T16 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 3758 T16 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3759 T16 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3759 T16 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001
   [junit4]   2> 3760 T16 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/'
   [junit4]   2> 3826 T16 oasc.CoreContainer.<init> New CoreContainer 24974469
   [junit4]   2> 3827 T16 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/]
   [junit4]   2> 3828 T16 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 3828 T16 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3829 T16 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 3829 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3830 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3830 T16 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3831 T16 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3831 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3832 T16 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3833 T16 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3833 T16 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 3834 T16 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 3835 T16 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3835 T16 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3835 T16 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 3839 T17 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/'
   [junit4]   2> 3874 T17 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 3882 T17 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3900 T17 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 3900 T17 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 3908 T17 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 3925 T17 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 3926 T17 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 3926 T17 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3927 T17 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/
   [junit4]   2> 3927 T17 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3928 T17 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001/
   [junit4]   2> 3928 T17 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 3928 T17 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 3928 T17 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 3929 T17 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001
   [junit4]   2> 3929 T17 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001/index/
   [junit4]   2> 3930 T17 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 3930 T17 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001/index
   [junit4]   2> 3931 T17 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dee501 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4415db),segFN=segments_1,generation=1}
   [junit4]   2> 3931 T17 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3933 T17 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 3950 T17 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 3951 T17 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3951 T17 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3951 T17 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3952 T17 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 3952 T17 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 3952 T17 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 3953 T17 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 3953 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 3954 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 3954 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 3955 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 3955 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 3955 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 3956 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 3956 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 3956 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 3957 T17 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3957 T17 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 3957 T17 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 3958 T17 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3960 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3961 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3962 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3963 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3964 T17 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 3964 T17 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3965 T17 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3966 T17 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dee501 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4415db),segFN=segments_1,generation=1}
   [junit4]   2> 3966 T17 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3966 T17 oass.SolrIndexSearcher.<init> Opening Searcher@f0325f[collection1] main
   [junit4]   2> 3967 T17 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/conf
   [junit4]   2> 3967 T17 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/conf}
   [junit4]   2> 3967 T17 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/core-home-001/collection1/conf
   [junit4]   2> 3968 T17 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3968 T17 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 3968 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@f0325f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3969 T17 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3969 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 3969 T18 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@f0325f[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3971 T16 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 3979 T16 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 37359 T16 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=884182025F52F548 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=pt_PT -Dtests.timezone=Iceland -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   33.4s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([884182025F52F548:D012E32742773A72]:0)
   [junit4]    > 	at org.apache.solr.handler.dataimport.AbstractDataImportHandlerTestCase.createMap(AbstractDataImportHandlerTestCase.java:157)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 37407 T16 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C4 name=collection1 org.apache.solr.core.SolrCore@1de5be0
   [junit4]   2> 37414 T16 C4 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 37416 T16 C4 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 37417 T16 C4 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 37419 T16 C4 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 37433 T16 C4 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 37433 T16 C4 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 37443 T16 C4 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dee501 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4415db),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1dee501 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4415db),segFN=segments_2,generation=2}
   [junit4]   2> 37443 T16 C4 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 37450 T16 C4 oass.SolrIndexSearcher.<init> Opening Searcher@166e238[collection1] main
   [junit4]   2> 37451 T16 C4 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 37451 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@166e238[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c25)))}
   [junit4]   2> 37473 T18 C4 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=22 
   [junit4]   2> 37474 T18 C4 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 37476 T18 C4 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=2 
   [junit4]   2> 37476 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 37477 T18 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@166e238[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c25)))}
   [junit4]   2> 37478 T16 C4 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 37479 T16 C4 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 37480 T16 C4 oashd.DocBuilder.execute Time taken = 0:0:0.62
   [junit4]   2> 37480 T16 C4 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[1, 2, 3],commit=} 0 70
   [junit4]   2> 37485 T16 C4 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=25 status=0 QTime=3 
   [junit4]   2> 37499 T16 C4 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 37503 T16 C4 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=9 status=0 QTime=1 
   [junit4]   2> 37508 T16 C4 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=13 status=0 QTime=1 
   [junit4]   2> 37544 T16 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 37550 T16 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 37551 T16 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=24974469
   [junit4]   2> 37551 T16 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1de5be0
   [junit4]   2> 37552 T16 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 37553 T16 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 37553 T16 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 37553 T16 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 37554 T16 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 37555 T16 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 37556 T16 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001;done=false>>]
   [junit4]   2> 37556 T16 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001
   [junit4]   2> 37557 T16 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 37557 T16 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001/init-core-data-001/index
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 884182025F52F548-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50), sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=pt_PT, timezone=Iceland
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=57263744,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestWriterImpl, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 34.03s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150116_205250_267.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid28962.hprof ...
   [junit4] Heap dump file created [608692574 bytes in 9.338 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 77 minutes 14 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -client -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_25) - Build # 11618 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11618/
Java: 64bit/jdk1.8.0_25 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([1C7D352DFE679EA6:442E5408E342519C]:0)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)




Build Log:
[...truncated 13377 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001
   [junit4]   2> 13625 T43 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 13684 T43 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 13685 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/'
   [junit4]   2> 13741 T43 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 13760 T43 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 13811 T43 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 13812 T43 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 13835 T43 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 13888 T43 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 13889 T43 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 13890 T43 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 13892 T43 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 13893 T43 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001
   [junit4]   2> 13893 T43 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/'
   [junit4]   2> 13956 T43 oasc.CoreContainer.<init> New CoreContainer 1118899470
   [junit4]   2> 13957 T43 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/]
   [junit4]   2> 13958 T43 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 13958 T43 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 13959 T43 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 13959 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 13959 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 13960 T43 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 13960 T43 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 13961 T43 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 13961 T43 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 13961 T43 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 13962 T43 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 13992 T43 oejul.Log.initialized Logging initialized @15225ms
   [junit4]   2> 14011 T43 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 14017 T43 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 14018 T43 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 14018 T43 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 14037 T44 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/'
   [junit4]   2> 14097 T44 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 14122 T44 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 14165 T44 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 14165 T44 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 14174 T44 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 14210 T44 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 14211 T44 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 14211 T44 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 14212 T44 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/
   [junit4]   2> 14213 T44 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 14213 T44 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001/
   [junit4]   2> 14213 T44 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 14214 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 14214 T44 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 14215 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001
   [junit4]   2> 14215 T44 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001/index/
   [junit4]   2> 14216 T44 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 14216 T44 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001/index
   [junit4]   2> 14217 T44 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@30315d6e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@475b0528),segFN=segments_1,generation=1}
   [junit4]   2> 14217 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 14218 T44 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 14219 T44 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 14220 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14223 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14223 T44 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 14224 T44 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 14225 T44 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 14225 T44 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 14226 T44 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 14226 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 14227 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 14227 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 14228 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 14229 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 14229 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 14237 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 14238 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 14238 T44 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 14239 T44 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 14239 T44 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 14240 T44 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 14240 T44 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 14243 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 14246 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 14257 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 14260 T44 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 14262 T44 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 14262 T44 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 14263 T44 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 14268 T44 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@30315d6e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@475b0528),segFN=segments_1,generation=1}
   [junit4]   2> 14268 T44 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 14269 T44 oass.SolrIndexSearcher.<init> Opening Searcher@277b28db[collection1] main
   [junit4]   2> 14270 T44 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/conf
   [junit4]   2> 14271 T44 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/conf}
   [junit4]   2> 14271 T44 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/core-home-001/collection1/conf
   [junit4]   2> 14272 T44 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 14272 T44 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 14273 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@277b28db[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 14273 T44 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 14274 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 14274 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@277b28db[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 14276 T43 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 14312 T43 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 32443 T43 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=1C7D352DFE679EA6 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_BA -Dtests.timezone=America/North_Dakota/Center -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   18.2s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1C7D352DFE679EA6:442E5408E342519C]:0)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 32493 T43 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C3 name=collection1 org.apache.solr.core.SolrCore@67013329
   [junit4]   2> 32499 T43 C3 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 32499 T43 C3 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 32511 T43 C3 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 32512 T43 C3 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 32526 T43 C3 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 32526 T43 C3 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 32562 T43 C3 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@30315d6e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@475b0528),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@30315d6e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@475b0528),segFN=segments_2,generation=2}
   [junit4]   2> 32563 T43 C3 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 32572 T43 C3 oass.SolrIndexSearcher.<init> Opening Searcher@4317762[collection1] main
   [junit4]   2> 32573 T43 C3 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 32573 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@4317762[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c29)))}
   [junit4]   2> 32584 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=10 
   [junit4]   2> 32585 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 32586 T45 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 32587 T45 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 32587 T45 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4317762[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c29)))}
   [junit4]   2> 32588 T43 C3 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 32590 T43 C3 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 32590 T43 C3 oashd.DocBuilder.execute Time taken = 0:0:0.78
   [junit4]   2> 32591 T43 C3 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[2, 3, 1],commit=} 0 95
   [junit4]   2> 32593 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=29 status=0 QTime=1 
   [junit4]   2> 32623 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=4 
   [junit4]   2> 32628 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=10 status=0 QTime=1 
   [junit4]   2> 32634 T43 C3 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=16 status=0 QTime=1 
   [junit4]   2> 32693 T43 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 32697 T43 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 32697 T43 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1118899470
   [junit4]   2> 32698 T43 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@67013329
   [junit4]   2> 32700 T43 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 32700 T43 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 32701 T43 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 32701 T43 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 32703 T43 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 32704 T43 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 32706 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 32707 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001/index
   [junit4]   2> 32714 T43 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001;done=false>>]
   [junit4]   2> 32715 T43 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001/init-core-data-001
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 1C7D352DFE679EA6-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=FSTOrd50, _root_=FSTOrd50, id=Lucene50(blocksize=128), desc=PostingsFormat(name=MockRandom), timestamp=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, sim=DefaultSimilarity, locale=sr_BA, timezone=America/North_Dakota/Center
   [junit4]   2> NOTE: Linux 3.13.0-43-generic amd64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=54806488,total=536870912
   [junit4]   2> NOTE: All tests run in this JVM: [TestZKPropertiesWriter, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 19.24s, 2 tests, 1 error <<< FAILURES!

[...truncated 52 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150116_181406_561.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid14118.hprof ...
   [junit4] Heap dump file created [549586688 bytes in 5.960 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 12 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 68 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 64bit/jdk1.8.0_25 -XX:-UseCompressedOops -XX:+UseG1GC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_25) - Build # 11617 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11617/
Java: 32bit/jdk1.8.0_25 -client -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([A9D69BA49B3572:58FAB7BEB9BEFA48]:0)
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:647)
	at java.lang.StringBuilder.append(StringBuilder.java:208)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:483)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)




Build Log:
[...truncated 13437 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001
   [junit4]   2> 24409 T173 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 24438 T173 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 24438 T173 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/'
   [junit4]   2> 24473 T173 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 24507 T173 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 24533 T173 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 24533 T173 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 24542 T173 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 24576 T173 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 24577 T173 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 24578 T173 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 24578 T173 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 24579 T173 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001
   [junit4]   2> 24579 T173 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/'
   [junit4]   2> 24657 T173 oasc.CoreContainer.<init> New CoreContainer 27593565
   [junit4]   2> 24658 T173 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/]
   [junit4]   2> 24658 T173 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 24659 T173 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 24659 T173 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 24659 T173 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 24660 T173 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 24660 T173 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 24660 T173 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 24660 T173 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 24661 T173 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 24661 T173 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 24661 T173 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 24662 T173 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 24662 T173 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 24662 T173 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 24663 T173 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 24675 T174 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/'
   [junit4]   2> 24727 T174 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 24743 T174 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 24796 T174 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 24797 T174 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 24810 T174 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 24896 T174 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 24898 T174 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 24899 T174 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 24900 T174 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/
   [junit4]   2> 24901 T174 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 24902 T174 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001/
   [junit4]   2> 24902 T174 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 24903 T174 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 24904 T174 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 24905 T174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001
   [junit4]   2> 24905 T174 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001/index/
   [junit4]   2> 24906 T174 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 24907 T174 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001/index
   [junit4]   2> 24908 T174 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@183e510 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9a4b2d),segFN=segments_1,generation=1}
   [junit4]   2> 24909 T174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 24921 T174 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 24922 T174 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 24923 T174 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 24923 T174 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 24924 T174 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 24925 T174 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 24926 T174 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 24927 T174 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 24928 T174 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 24928 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 24929 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 24930 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 24944 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 24945 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 24946 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 24946 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 24947 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 24948 T174 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 24949 T174 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 24949 T174 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 24950 T174 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 24950 T174 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 24954 T174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 24955 T174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 24957 T174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 24959 T174 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 24961 T174 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 24966 T174 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 24966 T174 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 24967 T174 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@183e510 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9a4b2d),segFN=segments_1,generation=1}
   [junit4]   2> 24967 T174 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 24967 T174 oass.SolrIndexSearcher.<init> Opening Searcher@e64f83[collection1] main
   [junit4]   2> 24968 T174 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/conf
   [junit4]   2> 24968 T174 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/conf}
   [junit4]   2> 24969 T174 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/core-home-001/collection1/conf
   [junit4]   2> 24969 T174 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 24969 T174 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 24970 T174 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 24971 T175 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@e64f83[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 24971 T175 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 24971 T175 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e64f83[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 24972 T173 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 24979 T173 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 85182 T173 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=A9D69BA49B3572 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=nl -Dtests.timezone=SystemV/EST5EDT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   60.3s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([A9D69BA49B3572:58FAB7BEB9BEFA48]:0)
   [junit4]    > 	at java.util.Arrays.copyOf(Arrays.java:3332)
   [junit4]    > 	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
   [junit4]    > 	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
   [junit4]    > 	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:647)
   [junit4]    > 	at java.lang.StringBuilder.append(StringBuilder.java:208)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:277)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:297)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 85237 T173 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C42 name=collection1 org.apache.solr.core.SolrCore@1953f6c
   [junit4]   2> 85243 T173 C42 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 85244 T173 C42 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 85245 T173 C42 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 85246 T173 C42 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 85261 T173 C42 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 85261 T173 C42 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 85283 T173 C42 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@183e510 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9a4b2d),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@183e510 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@9a4b2d),segFN=segments_2,generation=2}
   [junit4]   2> 85284 T173 C42 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> 85295 T173 C42 oass.SolrIndexSearcher.<init> Opening Searcher@1702ef4[collection1] main
   [junit4]   2> 85296 T173 C42 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 85296 T175 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@1702ef4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c30)))}
   [junit4]   2> 85299 T175 C42 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=2 
   [junit4]   2> 85300 T175 C42 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=0 
   [junit4]   2> 85301 T175 C42 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 85302 T175 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 85302 T175 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1702ef4[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c30)))}
   [junit4]   2> 85303 T173 C42 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 85304 T173 C42 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 85304 T173 C42 oashd.DocBuilder.execute Time taken = 0:0:0.58
   [junit4]   2> 85305 T173 C42 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[3, 1, 2],commit=} 0 67
   [junit4]   2> 85307 T173 C42 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=30 status=0 QTime=1 
   [junit4]   2> 85321 T173 C42 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=2 
   [junit4]   2> 85324 T173 C42 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=12 status=0 QTime=1 
   [junit4]   2> 85329 T173 C42 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=15 status=0 QTime=1 
   [junit4]   2> 85354 T173 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 85358 T173 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 85359 T173 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=27593565
   [junit4]   2> 85359 T173 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1953f6c
   [junit4]   2> 85360 T173 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 85360 T173 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 85360 T173 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 85361 T173 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 85362 T173 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 85362 T173 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 85362 T173 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001;done=false>>]
   [junit4]   2> 85363 T173 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001
   [junit4]   2> 85363 T173 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 85364 T173 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001/init-core-data-001/index
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder A9D69BA49B3572-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=FST50, _root_=FST50, id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), desc=PostingsFormat(name=LuceneVarGapFixedInterval), timestamp=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, sim=DefaultSimilarity, locale=nl, timezone=SystemV/EST5EDT
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_25 (32-bit)/cpus=8,threads=1,free=52700664,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestWriterImpl, TestZKPropertiesWriter, TestSolrEntityProcessorEndToEnd, TestSimplePropertiesWriter, TestBuiltInEvaluators, TestScriptTransformer, TestTemplateTransformer, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 61.00s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150116_155142_304.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid31487.hprof ...
   [junit4] Heap dump file created [593856225 bytes in 8.814 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 82 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_25 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.8.0_40-ea-b20) - Build # 11616 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/11616/
Java: 32bit/jdk1.8.0_40-ea-b20 -client -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy

Error Message:
Java heap space

Stack Trace:
java.lang.OutOfMemoryError: Java heap space
	at __randomizedtesting.SeedInfo.seed([4DC5877C99353865:1596E6598410F75F]:0)
	at sun.text.resources.FormatData.getContents(FormatData.java:120)
	at sun.util.resources.ParallelListResourceBundle.loadLookupTablesIfNecessary(ParallelListResourceBundle.java:168)
	at sun.util.resources.ParallelListResourceBundle.handleKeySet(ParallelListResourceBundle.java:134)
	at sun.util.resources.ParallelListResourceBundle.keySet(ParallelListResourceBundle.java:143)
	at sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
	at sun.util.locale.provider.LocaleResources.getDecimalFormatSymbolsData(LocaleResources.java:181)
	at java.text.DecimalFormatSymbols.initialize(DecimalFormatSymbols.java:616)
	at java.text.DecimalFormatSymbols.<init>(DecimalFormatSymbols.java:113)
	at sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(DecimalFormatSymbolsProviderImpl.java:85)
	at java.text.DecimalFormatSymbols.getInstance(DecimalFormatSymbols.java:180)
	at java.util.Formatter.getZero(Formatter.java:2283)
	at java.util.Formatter.<init>(Formatter.java:1892)
	at java.util.Formatter.<init>(Formatter.java:1947)
	at java.lang.String.format(String.java:2969)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:296)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:877)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)




Build Log:
[...truncated 13391 lines...]
   [junit4] Suite: org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001
   [junit4]   2> 3098 T16 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 3124 T16 oas.SolrTestCaseJ4.initCore ####initCore
   [junit4]   2> 3125 T16 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/'
   [junit4]   2> 3163 T16 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 3201 T16 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3244 T16 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 3245 T16 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 3259 T16 oass.IndexSchema.readSchema [null] Schema name=dih_test
   [junit4]   2> 3295 T16 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 3296 T16 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 3296 T16 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3297 T16 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3297 T16 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001
   [junit4]   2> 3298 T16 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/'
   [junit4]   2> 3348 T16 oasc.CoreContainer.<init> New CoreContainer 23941996
   [junit4]   2> 3348 T16 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/]
   [junit4]   2> 3349 T16 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 600000
   [junit4]   2> 3349 T16 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: null
   [junit4]   2> 3349 T16 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 60000
   [junit4]   2> 3350 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 3350 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxConnections to: 10000
   [junit4]   2> 3350 T16 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 3351 T16 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 3351 T16 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 3351 T16 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 3352 T16 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 3352 T16 oashc.HttpShardHandlerFactory.getParameter Setting useRetries to: false
   [junit4]   2> 3353 T16 oasu.UpdateShardHandler.<init> Creating UpdateShardHandler HTTP client with params: socketTimeout=30000&connTimeout=30000&retry=true
   [junit4]   2> 3353 T16 oasl.LogWatcher.createWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3353 T16 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3354 T16 oasc.CoreContainer.load Host Name: 
   [junit4]   2> 3357 T17 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/'
   [junit4]   2> 3399 T17 oasc.SolrConfig.refreshRequestParams current version of requestparams : -1
   [junit4]   2> 3408 T17 oasc.SolrConfig.<init> Using Lucene MatchVersion: 6.0.0
   [junit4]   2> 3427 T17 oasc.SolrConfig.<init> Loaded SolrConfig: dataimport-solrconfig.xml
   [junit4]   2> 3428 T17 oass.IndexSchema.readSchema Reading Solr Schema from /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/conf/dataimport-schema.xml
   [junit4]   2> 3436 T17 oass.IndexSchema.readSchema [collection1] Schema name=dih_test
   [junit4]   2> 3467 T17 oass.IndexSchema.readSchema default search field in schema is desc
   [junit4]   2> 3467 T17 oass.IndexSchema.readSchema query parser default operator is OR
   [junit4]   2> 3468 T17 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 3468 T17 oasc.CoreContainer.create Creating SolrCore 'collection1' using configuration from instancedir /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/
   [junit4]   2> 3469 T17 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 3469 T17 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001/
   [junit4]   2> 3469 T17 oasc.SolrCore.<init> JMX monitoring not detected for core: collection1
   [junit4]   2> 3470 T17 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for newSearcher: org.apache.solr.core.QuerySenderListener{queries=[{q=solr,start=0,rows=10}, {q=rocks,start=0,rows=10}, {q=static newSearcher warming query from solrconfig.xml}]}
   [junit4]   2> 3470 T17 oasc.SolrCore.initListeners [collection1] Added SolrEventListener for firstSearcher: org.apache.solr.core.QuerySenderListener{queries=[]}
   [junit4]   2> 3471 T17 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001
   [junit4]   2> 3471 T17 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001/index/
   [junit4]   2> 3471 T17 oasc.SolrCore.initIndex WARN [collection1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001/index' doesn't exist. Creating new index...
   [junit4]   2> 3472 T17 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001/index
   [junit4]   2> 3472 T17 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1d4b0d4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@122d73c),segFN=segments_1,generation=1}
   [junit4]   2> 3473 T17 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3474 T17 oasup.UpdateRequestProcessorChain.init creating updateRequestProcessorChain "" (default)
   [junit4]   2> 3492 T17 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "" (default)
   [junit4]   2> 3493 T17 oasc.RequestHandlers.initHandlersFromConfig created /update/json: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3493 T17 oasc.RequestHandlers.initHandlersFromConfig created /update/csv: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3493 T17 oasc.RequestHandlers.initHandlersFromConfig created /update/json/docs: org.apache.solr.handler.UpdateRequestHandler
   [junit4]   2> 3494 T17 oasc.RequestHandlers.initHandlersFromConfig created /config: org.apache.solr.handler.SolrConfigHandler
   [junit4]   2> 3494 T17 oasc.RequestHandlers.initHandlersFromConfig created /schema: org.apache.solr.handler.SchemaHandler
   [junit4]   2> 3494 T17 oasc.RequestHandlers.initHandlersFromConfig created /replication: org.apache.solr.handler.ReplicationHandler
   [junit4]   2> 3495 T17 oasc.RequestHandlers.initHandlersFromConfig created /get: org.apache.solr.handler.RealTimeGetHandler
   [junit4]   2> 3495 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/luke: org.apache.solr.handler.admin.LukeRequestHandler
   [junit4]   2> 3495 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/system: org.apache.solr.handler.admin.SystemInfoHandler
   [junit4]   2> 3496 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/mbeans: org.apache.solr.handler.admin.SolrInfoMBeanHandler
   [junit4]   2> 3496 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/plugins: org.apache.solr.handler.admin.PluginInfoHandler
   [junit4]   2> 3496 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/threads: org.apache.solr.handler.admin.ThreadDumpHandler
   [junit4]   2> 3497 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/properties: org.apache.solr.handler.admin.PropertiesRequestHandler
   [junit4]   2> 3497 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/logging: org.apache.solr.handler.admin.LoggingHandler
   [junit4]   2> 3497 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/file: org.apache.solr.handler.admin.ShowFileRequestHandler
   [junit4]   2> 3498 T17 oasc.RequestHandlers.initHandlersFromConfig created /admin/ping: org.apache.solr.handler.PingRequestHandler
   [junit4]   2> 3498 T17 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 3498 T17 oasc.RequestHandlers.initHandlersFromConfig created /dataimport: org.apache.solr.handler.dataimport.DataImportHandler
   [junit4]   2> 3499 T17 oasc.RequestHandlers.initHandlersFromConfig created /search: org.apache.solr.handler.component.SearchHandler
   [junit4]   2> 3499 T17 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 3501 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3502 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3503 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3504 T17 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 3505 T17 oasc.SolrCore.initStatsCache Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 3505 T17 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 3506 T17 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 3506 T17 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1d4b0d4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@122d73c),segFN=segments_1,generation=1}
   [junit4]   2> 3506 T17 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 3507 T17 oass.SolrIndexSearcher.<init> Opening Searcher@ea39a7[collection1] main
   [junit4]   2> 3507 T17 oasr.ManagedResourceStorage$FileStorageIO.configure File-based storage initialized to use dir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/conf
   [junit4]   2> 3508 T17 oasr.RestManager.init Initializing RestManager with initArgs: {storageDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/conf}
   [junit4]   2> 3508 T17 oasr.ManagedResourceStorage.load Reading _rest_managed.json using file:dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/core-home-001/collection1/conf
   [junit4]   2> 3508 T17 oasr.RestManager.init Initializing 0 registered ManagedResources
   [junit4]   2> 3509 T17 oash.ReplicationHandler.inform Commits will be reserved for  10000
   [junit4]   2> 3509 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@ea39a7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3509 T17 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 3511 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 3512 T18 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ea39a7[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3513 T16 oas.SolrTestCaseJ4.initCore ####initCore end
   [junit4]   2> 3516 T16 oas.SolrTestCaseJ4.setUp ###Starting testThreeLevelHierarchy
   [junit4]   2> ASYNC  NEW_CORE C6 name=collection1 org.apache.solr.core.SolrCore@fe5ed7
   [junit4]   2> 3520 T16 C6 oashd.DataImporter.loadDataConfig Data Configuration loaded successfully
   [junit4]   2> 3521 T16 C6 oashd.DataImporter.doFullImport Starting Full Import
   [junit4]   2> 3522 T16 C6 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 3523 T16 C6 oasu.DirectUpdateHandler2.deleteAll [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 3537 T16 C6 oashd.DocBuilder.finish Import completed successfully
   [junit4]   2> 3538 T16 C6 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 3561 T16 C6 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1d4b0d4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@122d73c),segFN=segments_1,generation=1}
   [junit4]   2> 		commit{dir=MockDirectoryWrapper(RAMDirectory@1d4b0d4 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@122d73c),segFN=segments_2,generation=2}
   [junit4]   2> 3562 T16 C6 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
   [junit4]   2> ASYNC  NEW_CORE C7 name=collection1 org.apache.solr.core.SolrCore@fe5ed7
   [junit4]   2> 3578 T16 C7 oass.SolrIndexSearcher.<init> Opening Searcher@797f12[collection1] main
   [junit4]   2> 3579 T16 C7 oasu.DirectUpdateHandler2.commit end_commit_flush
   [junit4]   2> 3579 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener sending requests to Searcher@797f12[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c26)))}
   [junit4]   2> 3580 T18 C7 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=solr&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 3581 T18 C7 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=rocks&distrib=false&start=0&rows=10&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 3582 T18 C7 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=static+newSearcher+warming+query+from+solrconfig.xml&distrib=false&event=newSearcher} hits=0 status=0 QTime=1 
   [junit4]   2> 3583 T18 oasc.QuerySenderListener.newSearcher QuerySenderListener done.
   [junit4]   2> 3583 T18 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@797f12[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.0.0):c26)))}
   [junit4]   2> 3583 T16 C7 oashd.SimplePropertiesWriter.readIndexerProperties WARN Unable to read: dataimport.properties
   [junit4]   2> 3584 T16 C7 oashd.SimplePropertiesWriter.persist Wrote last indexed time to dataimport.properties
   [junit4]   2> 3584 T16 C7 oashd.DocBuilder.execute Time taken = 0:0:0.61
   [junit4]   2> 3585 T16 C7 oasup.LogUpdateProcessor.finish [collection1] webapp=null path=null params={debug=on&commit=true&dataConfig=<dataConfig>%0a++<dataSource+type%3D'MockDataSource'+/>%0a++<document>%0a++++<entity+name%3D'PARENT'+query%3D'select+*+from+PARENT'>%0a++++++<field+column%3D'id'+/>%0a++++++<field+column%3D'desc'+/>%0a++++++<field+column%3D'type_s'+/>%0a++++++<entity+child%3D'true'+name%3D'CHILD'+query%3D"select+*+from+CHILD+where+parent_id%3D'${PARENT.id}'">%0a++++++++<field+column%3D'id'+/>%0a++++++++<field+column%3D'desc'+/>%0a++++++++<field+column%3D'type_s'+/>%0a++++++++++<entity+child%3D'true'+name%3D'GRANDCHILD'+query%3D"select+*+from+GRANDCHILD+where+parent_id%3D'${CHILD.id}'">%0a++++++++++++<field+column%3D'id'+/>%0a++++++++++++<field+column%3D'desc'+/>%0a++++++++++++<field+column%3D'type_s'+/>%0a++++++++++</entity>%0a++++++</entity>%0a++++</entity>%0a++</document>%0a</dataConfig>&clean=true&wt=xml&command=full-import} {deleteByQuery=*:*,add=[2, 1, 3],commit=} 0 68
   [junit4]   2> 3588 T16 C7 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=*:*&qt=standard&start=0&rows=20&version=2.2} hits=26 status=0 QTime=2 
   [junit4]   2> 3604 T16 C7 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:parent&qt=standard&start=0&rows=20&version=2.2} hits=3 status=0 QTime=6 
   [junit4]   2> 3607 T16 C7 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:child&qt=standard&start=0&rows=20&version=2.2} hits=9 status=0 QTime=1 
   [junit4]   2> 3611 T16 C7 oasc.SolrCore.execute [collection1] webapp=null path=null params={q=type_s:grand_child&qt=standard&start=0&rows=20&version=2.2} hits=14 status=0 QTime=1 
   [junit4]   2> 3638 T16 oas.SolrTestCaseJ4.tearDown ###Ending testThreeLevelHierarchy
   [junit4]   2> 3652 T16 oas.SolrTestCaseJ4.setUp ###Starting testRandomDepthHierarchy
   [junit4]   2> 58915 T16 oas.SolrTestCaseJ4.tearDown ###Ending testRandomDepthHierarchy
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestHierarchicalDocBuilder -Dtests.method=testRandomDepthHierarchy -Dtests.seed=4DC5877C99353865 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_UY -Dtests.timezone=Africa/Addis_Ababa -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   55.3s J0 | TestHierarchicalDocBuilder.testRandomDepthHierarchy <<<
   [junit4]    > Throwable #1: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4DC5877C99353865:1596E6598410F75F]:0)
   [junit4]    > 	at sun.text.resources.FormatData.getContents(FormatData.java:120)
   [junit4]    > 	at sun.util.resources.ParallelListResourceBundle.loadLookupTablesIfNecessary(ParallelListResourceBundle.java:168)
   [junit4]    > 	at sun.util.resources.ParallelListResourceBundle.handleKeySet(ParallelListResourceBundle.java:134)
   [junit4]    > 	at sun.util.resources.ParallelListResourceBundle.keySet(ParallelListResourceBundle.java:143)
   [junit4]    > 	at sun.util.resources.ParallelListResourceBundle.containsKey(ParallelListResourceBundle.java:129)
   [junit4]    > 	at sun.util.locale.provider.LocaleResources.getDecimalFormatSymbolsData(LocaleResources.java:181)
   [junit4]    > 	at java.text.DecimalFormatSymbols.initialize(DecimalFormatSymbols.java:616)
   [junit4]    > 	at java.text.DecimalFormatSymbols.<init>(DecimalFormatSymbols.java:113)
   [junit4]    > 	at sun.util.locale.provider.DecimalFormatSymbolsProviderImpl.getInstance(DecimalFormatSymbolsProviderImpl.java:85)
   [junit4]    > 	at java.text.DecimalFormatSymbols.getInstance(DecimalFormatSymbols.java:180)
   [junit4]    > 	at java.util.Formatter.getZero(Formatter.java:2283)
   [junit4]    > 	at java.util.Formatter.<init>(Formatter.java:1892)
   [junit4]    > 	at java.util.Formatter.<init>(Formatter.java:1947)
   [junit4]    > 	at java.lang.String.format(String.java:2969)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createMockedIterator(TestHierarchicalDocBuilder.java:296)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:319)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createChildren(TestHierarchicalDocBuilder.java:321)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.createRandomizedConfig(TestHierarchicalDocBuilder.java:261)
   [junit4]    > 	at org.apache.solr.handler.dataimport.TestHierarchicalDocBuilder.testRandomDepthHierarchy(TestHierarchicalDocBuilder.java:176)
   [junit4]   2> 58955 T16 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 58955 T16 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=23941996
   [junit4]   2> 58955 T16 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@fe5ed7
   [junit4]   2> 58956 T16 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=3,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 58956 T16 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 58956 T16 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 58957 T16 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 58957 T16 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 58958 T16 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 58958 T16 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001 [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001;done=false>>]
   [junit4]   2> 58959 T16 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001
   [junit4]   2> 58959 T16 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001/index;done=false>>]
   [junit4]   2> 58959 T16 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001/init-core-data-001/index
   [junit4]   2> NOTE: leaving temporary files on disk at: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/J0/temp/solr.handler.dataimport.TestHierarchicalDocBuilder 4DC5877C99353865-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene50): {type_s=FSTOrd50, _root_=FSTOrd50, id=PostingsFormat(name=Memory doPackFST= true), desc=Lucene50(blocksize=128), timestamp=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{}, sim=RandomSimilarityProvider(queryNorm=false,coord=crazy): {}, locale=es_UY, timezone=Africa/Addis_Ababa
   [junit4]   2> NOTE: Linux 3.13.0-43-generic i386/Oracle Corporation 1.8.0_40-ea (32-bit)/cpus=8,threads=1,free=58260488,total=518979584
   [junit4]   2> NOTE: All tests run in this JVM: [TestWriterImpl, TestHierarchicalDocBuilder]
   [junit4] Completed on J0 in 55.91s, 2 tests, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J0: stdout was not empty, see: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/contrib/solr-dataimporthandler/test/temp/junit4-J0-20150116_131313_930.sysout
   [junit4] >>> JVM J0: stdout (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/heapdumps/java_pid17219.hprof ...
   [junit4] Heap dump file created [591579212 bytes in 8.258 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 9 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:519: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:467: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:61: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/extra-targets.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:207: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:454: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:510: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1348: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:956: There were test failures: 38 suites, 156 tests, 1 error, 3 ignored

Total time: 82 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
[description-setter] Description set: Java: 32bit/jdk1.8.0_40-ea-b20 -client -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any