java.util.EmptyStackException during SPLITSHARD

[Oliver Schrenk <os...@elmar.nl>]

Hi,

trying to experiment with overcharging on our Solr 4.7.2 cluster and called SPLITSHARD command which after ~30 minutes of work failed with

	curl "http://solrhost:1234/solr/admin/collections?collection=acme&shard=shard1&action=SPLITSHARD"
	<?xml version="1.0" encoding="UTF-8"?>
	<response>
	<lst name="responseHeader"><int name="status">500</int><int name="QTime">836129</int></lst><lst name="success"><lst><lst name="responseHeader"><int name="status">0</int><int name="QTime">1499</int></lst><str name="core">elmar_v3_shard1_1_replica1</str></lst><lst><lst name="responseHeader"><int name="status">0</int><int name="QTime">1507</int></lst><str name="core">elmar_v3_shard1_0_replica1</str></lst><lst><lst name="responseHeader"><int name="status">0</int><int name="QTime">1002</int></lst></lst><lst><lst name="responseHeader"><int name="status">0</int><int name="QTime">1001</int></lst></lst><lst><lst name="responseHeader"><int name="status">0</int><int name="QTime">713458</int></lst></lst><lst><lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><str name="core">elmar_v3_shard1_0_replica1</str><str name="status">EMPTY_BUFFER</str></lst><lst><lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst><str name="core">elmar_v3_shard1_1_replica1</str><str name="status">EMPTY_BUFFER</str></lst></lst><str name="Operation splitshard caused exception:">java.util.EmptyStackException:java.util.EmptyStackException</str><lst name="exception"><null name="msg"/><int name="rspCode">-1</int></lst><lst name="error"><str name="trace">org.apache.solr.common.SolrException
		at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:248)
		at org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:484)
		at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:165)
		at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
		at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:720)
		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:265)
		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:205)
		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
		at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
		at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
		at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
		at org.eclipse.jetty.server.Server.handle(Server.java:370)
		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
		at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
		at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
		at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
		at java.lang.Thread.run(Thread.java:745)
	</str><int name="code">500</int></lst>
	</response>

It also seems that for 30 minutes it was very slow and not even half way done:

	acme@solrhost:/var/lib/solr/cores$ du -h acme_shard1_replica1/
	271M	acme_shard1_replica1/data/tlog
	5.4G	acme_shard1_replica1/data/index
	5.7G	acme_shard1_replica1/data
	5.7G	acme_shard1_replica1/
	acme@solrhost:/var/lib/solr/cores$ du -h acme_shard1_1_replica1/
	4.0K	acme_shard1_1_replica1/data/tlog
	1004M	acme_shard1_1_replica1/data/index
	1004M	acme_shard1_1_replica1/data
	1004M	acme_shard1_1_replica1/
	acme@solrhost:/var/lib/solr/cores$ du -h acme_shard1_0_replica1/
	4.0K	acme_shard1_0_replica1/data/tlog
	898M	acme_shard1_0_replica1/data/index
	898M	acme_shard1_0_replica1/data
	898M	acme_shard1_0_replica1/


What is the recommended cleanup strategy? 

Also is there a way for me run the command successfully on my end without upgrading? 


Cheers,
Oliver