You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "raj.yadav" <ra...@cse.ism.ac.in> on 2020/12/11 11:44:42 UTC
[SOLR-8.5.2] Commit through curl command is causing delay in
issuing commit
Solr Setup: (Running in solrCloud mode)
It has 6 shards, and each shard has only one replica (which is also a
leader) and the replica type is NRT.
Each shards are hosted on the separate physical host.
Zookeeper => We are using external zookeeper ensemble (3 separate node
cluster)
Shard and Host name
shard1_0=>solr_199
shard1_1=>solr_200
shard2_0=> solr_254
shard2_1=> solr_132
shard3_0=>solr_133
shard3_1=>solr_198
*Request rate on the system is currently zero and only hourly indexing
running on it.*
We are using curl command to issue commit.
/curl
"http://solr_254:8389/solr/my_collection/update?openSearcher=true&commit=true&wt=json"/
(Here we are using solr_254 host to issue commit)
*On using above command all the shards have started processing commit (i.e
getting `start commit` request) except the one used in curl command (i.e
shard2_0 which is hosted on solr_254). Individually each shards takes around
10 to 12 min to process hard commit (most of this time is spent on reloading
external files).
As per logs, shard2_0 is getting `start commit` request after 10 minutes
(approx). This leads to following timeout error.*
2020-12-06 18:47:47.013 ERROR
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at:
http://solr_132:9744/solr/my_collection_shard2_1_replica_n21/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2Fsolr_254%3A9744%2Fsolr%2Fmy_collection_shard2_0_replica_n11%2F
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:407)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:753)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient.request(ConcurrentUpdateHttp2SolrClient.java:369)
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1290)
at
org.apache.solr.update.SolrCmdDistributor.doRequest(SolrCmdDistributor.java:344)
at
org.apache.solr.update.SolrCmdDistributor.lambda$submit$0(SolrCmdDistributor.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.TimeoutException
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:216)
at
org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:398)
... 13 more
Above timeout error is between solr_254 and solr_132. Similar errors are
there between solr_254 and other 4 shards
Since query load is zero, mostly CPU utilization is around 3%.
After issuing curl commit command, CPU goes up to 14% on all shards except
shard2_0 (host: solr_254, the one used in curl command).
And after 10 minutes (i.e after getting the `start commit` request) CPU on
shard2_0 also goes up to 14%.
As I mentioned earlier each shards take around 10-12 mins to process commit
and due to delay in starting commit process on one shard (shard2_0) our
overall commit time is doubled now. (22-24 minutes approx)
In our solr-5.4.0(having similar setup), we use similar command curl command
to issue commit and there all the shards are getting `start commit` request
at same time. Including the one used in curl command.
I'm not able to figure out why this is happening? Is there something changed
in internal functioning when we issue commit through curl? Is it do
something with HTTP2? We do not use autoCommit feature available in
solrconfig.xml. Its not suitable for our system. Apart from curl command are
there any other alternate way to issue commit?
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: [SOLR-8.5.2] Commit through curl command is causing delay in
issuing commit
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi All,
For further investigation, I have raised a JIRA ticket.
https://issues.apache.org/jira/browse/SOLR-15045
In case, anyone has any information to share, feel free to mention it here.
Regards,
Raj
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: [SOLR-8.5.2] Commit through curl command is causing delay in
issuing commit
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi All,
As I mentioned in my previous post that reloading/refreshing of the external
file is consuming most of the time during a commit operation.
In order to nullify the impact of external files, I had deleted external
files from all the shards and issued commit through the curl command. Commit
operation got completed in 3 seconds. Individual shards took 1.5 seconds to
complete the commit operation. But there was a delay of around 1.5 seconds
on the shard whose hostname was used to issue the commit. Hence overall
commit time is 3 seconds.
During this operation, there was no timeout or any other kind of error
(except `external file not found` error which is expected). I'm not able to
figure what might be causing the delay on hostname_shard. Is there any
setting that impacts curls operation and we might have accidentally changed
it.
I have been trying to solve this issue for the last 15 days, can someone
please help in resolving it.
Let me know in case any information/logs are missing.
Regards,
Raj
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: [SOLR-8.5.2] Commit through curl command is causing delay in
issuing commit
Posted by "raj.yadav" <ra...@cse.ism.ac.in>.
Hi All,
Till we investigate further about this issue.
Can anyone please share what other ways we can issue a commit or point me to
existing documents that have a relevant example.
Regards,
Raj
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html