Running update-runtimelib seems to create load on SOLR

[Liviu Matei <li...@gmail.com>]

Hi,

We are having a SOLR Cloud instance running on 3 nodes. There we have
defined multiple collections each having 1 shard and 2 replicas. Each of
these collections is having a runtime lib associated which defines a
custom handler. From time to time I need to update the runtime lib and when
that happens I can see that the load on SOLR increases and on some nodes it
seems that the request is timing out and I get a response like the one
bellow:

REST response={
  "responseHeader":{
    "status":500,
    "QTime":30013},
  "errorMessages":["2 out of 3 the property overlay to be of version 2,375
within 30 seconds! Failed cores: [
http://solr01.solr01.aiapps.oraclevcn.com:8983/solr/collectionC_shard1_replica_n2/,
http://solr03.solr03.aiapps.oraclevcn.com:8983/solr/collectionC_shard1_replica_n9/
]\n"],
  "WARNING":"This response format is experimental.  It is likely to change
in the future.",
  "error":{
    "metadata":[
      "error-class","org.apache.solr.common.SolrException",
      "root-error-class","org.apache.solr.common.SolrException"],
    "msg":"2 out of 3 the property overlay to be of version 2,375 within 30
seconds! Failed cores: [
http://solr01.solr01.aiapps.oraclevcn.com:8983/solr/collectionC_shard1_replica_n2/,
http://solr03.solr03.aiapps.oraclevcn.com:8983/solr/collectionC_shard1_replica_n9/
]",
    "trace":"org.apache.solr.common.SolrException: 2 out of 3 the property
overlay to be of version 2,375 within 30 seconds! Failed cores: [
http://solr01.solr01.aiapps.oraclevcn.com:8983/solr/collectionC_shard1_replica_n2/,
http://solr03.solr03.aiapps.oraclevcn.com:8983/solr/collectionC_shard1_replica_n9/]\n\tat
org.apache.solr.handler.SolrConfigHandler.waitForAllReplicasState(SolrConfigHandler.java:768)\n\tat
org.apache.solr.handler.SolrConfigHandler.access$400(SolrConfigHandler.java:101)\n\tat
org.apache.solr.handler.SolrConfigHandler$Command.handleCommands(SolrConfigHandler.java:499)\n\tat
org.apache.solr.handler.SolrConfigHandler$Command.handlePOST(SolrConfigHandler.java:345)\n\tat
org.apache.solr.handler.SolrConfigHandler$Command.access$100(SolrConfigHandler.java:158)\n\tat
org.apache.solr.handler.SolrConfigHandler.handleRequestBody(SolrConfigHandler.java:136)\n\tat
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
org.apache.solr.core.SolrCore.execute(SolrCore.java:2551)\n\tat
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:733)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:531)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:502)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)\n\tat
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)\n\tat
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)\n\tat
java.lang.Thread.run(Thread.java:748)\n",
    "code":500}}


To update the runtime lib is done via a POST with the JSON:
{
"update-runtimelib": {
"name":"customLibrary",
"version":"1"
   }
 }
Any help with this would be great.

Thanks a lot!
Liviu