You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by "Patil, Prafull via users" <us...@solr.apache.org> on 2026/06/02 04:20:30 UTC
Solr replica recovery failures
Hello,
We have the Solr cloud setup on our Teamcenter PROD environment as below:
Solr + ZooKeeper : server1 OSLO (Leader for all replicas)
Solr + ZooKeeper : server2 HOU (Follower)
Solr + ZooKeeper : server3 SNG (Follower)
Solr version : 9.7.0
No of shards: 10 shards for collection1
No of replicas: 3 per shard
Java: JDK21
Teamcenter : 2412.0006
We are experiencing that, sometimes, all of a sudden one of the follower server HOU/SNG lags behind and all of the HOU/SNG replicas gets stuck in recovery mode for infinite time. As per the solr logs, during the replica recovery, it tries to copy segment files from the leader solr and it fails with 2 minutes timeouts while copying files above ~150 MB size.
The solr.log has errors as :
"WARN (recoveryExecutor-12-thread-9-processing-hou4140.verit.dnv.com:8984_solr collection1_shard4_replica_n38 collection1 shard4 core_node41) [c:collection1 s:shard4 r:core_node41 x:collection1_shard4_replica_n38 t:] o.a.s.h.IndexFetcher Error in fetching file: _1jnu_Lucene99_0.pos (downloaded 320864256 of 511797594 bytes) => java.io.IOException: java.util.concurrent.TimeoutException: Total timeout 120000 ms elapsed
at org.eclipse.jetty.client.util.InputStreamResponseListener$Input.toIOException(InputStreamResponseListener.java:343)
java.io.IOException: java.util.concurrent.TimeoutException: Total timeout 120000 ms elapsed"
Basically, it's failing to copy big size files(150+ MB) from leader to follower and failing after 120000 ms.
Any thoughts on below questions would be appreciated:
1.
Why all of a sudden the follower servers data lags behind
2.
Is there any way to increase the 120 sec timeout?
We have already tried adding params like "-Dsolr.http.timeout" and "-Dsolr.indexfetcher.timeout" but nothing helped us.
Thank you,
Prafull
Best Regards,
Prafull Prakash Patil
PLM Operations, Pune
Mobile : +917032529590
**************************************************************************************
This e-mail and any attachments thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. If you have received this transmission in error, please immediately notify the sender by return e-mail and delete this message and its attachments. Unauthorized use, copying or further full or partial distribution of this e-mail or its contents is prohibited.
**************************************************************************************
Re: Re:Solr replica recovery failures
Posted by "Patil, Prafull via users" <us...@solr.apache.org>.
Wow Mr. Luke Kot-Zaniewski , enabling http1 connection did the magic for us. Since our Solr is functioning with all active nodes/replicas for last 24 hours now, I am not really worried about whether it's a workaround or a permanent fix. Thanks a ton !
Best Regards,
Prafull Prakash Patil
PLM Operations, Pune
Mobile : +917032529590
________________________________
From: Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) via users <us...@solr.apache.org>
Sent: Tuesday, June 2, 2026 7:15 PM
To: users@solr.apache.org <us...@solr.apache.org>
Cc: Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) <lk...@bloomberg.net>
Subject: Re:Solr replica recovery failures
This sounds a lot like this issue: https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSOLR-18087&data=05%7C02%7CPrafull.Patil%40dnv.com%7Cf1cb33100d3f4c00298308dec0ad4d26%7Cadf10e2bb6e941d6be2fc12bb566019c%7C0%7C0%7C639160047729959247%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=mcaW3hWSXqlWpb%2FG6oBheJacyEttloqDDkudh59jTss%3D&reserved=0<https://issues.apache.org/jira/browse/SOLR-18087>
Especially the:
> (downloaded 320864256 of 511797594 bytes)
IME if the download just stops midway its usually because
of a deadlocked HTTP2 connection from follower to leader.
The workaround is to run with -Dsolr.http1=true. The
underlying issue is still unresolved AFAIK (I remember
still seeing this when running from main fairly recently).
From: users@solr.apache.org At: 06/02/26 00:20:59 UTC-4:00To: users@solr.apache.org
Cc: Prafull.Patil@dnv.com
Subject: Solr replica recovery failures
Hello,
We have the Solr cloud setup on our Teamcenter PROD environment as below:
Solr + ZooKeeper : server1 OSLO (Leader for all replicas)
Solr + ZooKeeper : server2 HOU (Follower)
Solr + ZooKeeper : server3 SNG (Follower)
Solr version : 9.7.0
No of shards: 10 shards for collection1
No of replicas: 3 per shard
Java: JDK21
Teamcenter : 2412.0006
We are experiencing that, sometimes, all of a sudden one of the follower server
HOU/SNG lags behind and all of the HOU/SNG replicas gets stuck in recovery mode
for infinite time. As per the solr logs, during the replica recovery, it tries
to copy segment files from the leader solr and it fails with 2 minutes timeouts
while copying files above ~150 MB size.
The solr.log has errors as :
"WARN (recoveryExecutor-12-thread-9-processing-hou4140.verit.dnv.com:8984_solr
collection1_shard4_replica_n38 collection1 shard4 core_node41) [c:collection1
s:shard4 r:core_node41 x:collection1_shard4_replica_n38 t:]
o.a.s.h.IndexFetcher Error in fetching file: _1jnu_Lucene99_0.pos (downloaded
320864256 of 511797594 bytes) => java.io.IOException:
java.util.concurrent.TimeoutException: Total timeout 120000 ms elapsed
at
org.eclipse.jetty.client.util.InputStreamResponseListener$Input.toIOException(In
putStreamResponseListener.java:343)
java.io.IOException: java.util.concurrent.TimeoutException: Total timeout
120000 ms elapsed"
Basically, it's failing to copy big size files(150+ MB) from leader to follower
and failing after 120000 ms.
Any thoughts on below questions would be appreciated:
1.
Why all of a sudden the follower servers data lags behind
2.
Is there any way to increase the 120 sec timeout?
We have already tried adding params like "-Dsolr.http.timeout" and
"-Dsolr.indexfetcher.timeout" but nothing helped us.
Thank you,
Prafull
Best Regards,
Prafull Prakash Patil
PLM Operations, Pune
Mobile : +917032529590
********************************************************************************
******
This e-mail and any attachments thereto may contain confidential information
and/or information protected by intellectual property rights for the exclusive
attention of the intended addressees named above. If you have received this
transmission in error, please immediately notify the sender by return e-mail
and delete this message and its attachments. Unauthorized use, copying or
further full or partial distribution of this e-mail or its contents is
prohibited.
********************************************************************************
******
**************************************************************************************
This e-mail and any attachments thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. If you have received this transmission in error, please immediately notify the sender by return e-mail and delete this message and its attachments. Unauthorized use, copying or further full or partial distribution of this e-mail or its contents is prohibited.
**************************************************************************************
Re:Solr replica recovery failures
Posted by "Luke Kot-Zaniewski (BLOOMBERG/ 919 3RD A) via users" <us...@solr.apache.org>.
This sounds a lot like this issue: https://issues.apache.org/jira/browse/SOLR-18087
Especially the:
> (downloaded 320864256 of 511797594 bytes)
IME if the download just stops midway its usually because
of a deadlocked HTTP2 connection from follower to leader.
The workaround is to run with -Dsolr.http1=true. The
underlying issue is still unresolved AFAIK (I remember
still seeing this when running from main fairly recently).
From: users@solr.apache.org At: 06/02/26 00:20:59 UTC-4:00To: users@solr.apache.org
Cc: Prafull.Patil@dnv.com
Subject: Solr replica recovery failures
Hello,
We have the Solr cloud setup on our Teamcenter PROD environment as below:
Solr + ZooKeeper : server1 OSLO (Leader for all replicas)
Solr + ZooKeeper : server2 HOU (Follower)
Solr + ZooKeeper : server3 SNG (Follower)
Solr version : 9.7.0
No of shards: 10 shards for collection1
No of replicas: 3 per shard
Java: JDK21
Teamcenter : 2412.0006
We are experiencing that, sometimes, all of a sudden one of the follower server
HOU/SNG lags behind and all of the HOU/SNG replicas gets stuck in recovery mode
for infinite time. As per the solr logs, during the replica recovery, it tries
to copy segment files from the leader solr and it fails with 2 minutes timeouts
while copying files above ~150 MB size.
The solr.log has errors as :
"WARN (recoveryExecutor-12-thread-9-processing-hou4140.verit.dnv.com:8984_solr
collection1_shard4_replica_n38 collection1 shard4 core_node41) [c:collection1
s:shard4 r:core_node41 x:collection1_shard4_replica_n38 t:]
o.a.s.h.IndexFetcher Error in fetching file: _1jnu_Lucene99_0.pos (downloaded
320864256 of 511797594 bytes) => java.io.IOException:
java.util.concurrent.TimeoutException: Total timeout 120000 ms elapsed
at
org.eclipse.jetty.client.util.InputStreamResponseListener$Input.toIOException(In
putStreamResponseListener.java:343)
java.io.IOException: java.util.concurrent.TimeoutException: Total timeout
120000 ms elapsed"
Basically, it's failing to copy big size files(150+ MB) from leader to follower
and failing after 120000 ms.
Any thoughts on below questions would be appreciated:
1.
Why all of a sudden the follower servers data lags behind
2.
Is there any way to increase the 120 sec timeout?
We have already tried adding params like "-Dsolr.http.timeout" and
"-Dsolr.indexfetcher.timeout" but nothing helped us.
Thank you,
Prafull
Best Regards,
Prafull Prakash Patil
PLM Operations, Pune
Mobile : +917032529590
********************************************************************************
******
This e-mail and any attachments thereto may contain confidential information
and/or information protected by intellectual property rights for the exclusive
attention of the intended addressees named above. If you have received this
transmission in error, please immediately notify the sender by return e-mail
and delete this message and its attachments. Unauthorized use, copying or
further full or partial distribution of this e-mail or its contents is
prohibited.
********************************************************************************
******