[jira] [Created] (SOLR-8431) Parent shard cannot be deleted after shard splitting

["Tan Lay How (JIRA)" <ji...@apache.org>]

Tan Lay How created SOLR-8431:
---------------------------------

             Summary: Parent shard cannot be deleted after shard splitting
                 Key: SOLR-8431
                 URL: https://issues.apache.org/jira/browse/SOLR-8431
             Project: Solr
          Issue Type: Bug
    Affects Versions: 4.10.1
            Reporter: Tan Lay How
            Priority: Blocker


I performed shard splitting on 2 of the shards out of 3 with the async request id. The shard splitting task is failed when it trying to attach the replica for the splitted shard, but i found out on both of the leader splitted shards, document is splitted correctly (total numFound of both shards = total numFound of parent shard)

So, i proceed to manually change the clusterstate.json, splitted shards changed status from construction to active, and parent shards changed from active to inactive. I also manually attach the replica for the splitted shards, then only remove the parent shards by unload the parent shards core.

Here the problems come, when i issue a commit in SolrCloud, both of the parent return back in the solr cloud cloud graph and clusterstate.json with node status = down & shard status = active.

1) I bring up the parent shards node again and try with another unload core again, but every time i issue a commit, it will back into the graph and clusterstate with node status = down & shard status = active.

2) So my second attempt is to delete the shards with collection API using /admin/collections?action=DELETESHARD

I got the error as below :

org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:Cannot >unload non-existent core

Operation deleteshard caused exception: org.apache.solr.common.SolrException: Could not fully remove collection: >candidates shard: candidates_shard2 at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:364) at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:320) at org.apache.solr.handler.admin.CollectionsHandler.handleDeleteShardAction(CollectionsHandler.java:563) at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:176) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:267) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:612) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AutoLoginValve.invoke(AutoLoginValve.java:67) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:304) at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:82) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)
3) My third attempt is the delete the replica using the collection API admin/collections?action=DELETEREPLICA

I got the error as below :

org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Could not remove replica

org.apache.solr.common.SolrException: Could not remove replica : candidates/candidates_shard2/core_node48 at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:364) at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:320) at org.apache.solr.handler.admin.CollectionsHandler.handleRemoveReplica(CollectionsHandler.java:495) at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:184) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:729) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:267) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:612) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) at org.apache.catalina.valves.AutoLoginValve.invoke(AutoLoginValve.java:67) at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:304) at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:82) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)


The parent shards cannot be removed and when the delete query came in, Solr will return "No registered leader was found after waiting xx" due to the parent shards node status is down but the shards status still appear active in the cluster.

The same goes for select query, will return error "No server hosting shards".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org