You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Reej Nayagam <re...@gmail.com> on 2021/08/16 11:17:35 UTC
Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.8.2
Hi Shawn ,
Please find the gc logs from our test server. Your help is much appreciated.
https://drive.google.com/drive/folders/1VQqlRxiDnHWPUlmTUv83a8CqCtvBP4le?usp=sharing
Also our Solr version is 8.8.2, so if there is a fix after 8.4+ , the 8.8.2
version should have this anyway right.
I've increased the autocommit maxtime to 60000, opensearcher is false and
autosoftcommit maxtime to 120000 to check just for 2 collections, and for
those 2 I'm not getting the Overlapping on Decksearchers so frequently ,
but the below errors are happening frequently.
*Request Handler Base : java.io.IOException:Task Queue processing has
stalled for 20191 ms with 0 remaining elements to process*
*Httpsolrcall : java.io.IOException:Task Queue processing has stalled for
20191 ms with 0 remaining elements to process*
*Solrwriter : Exception while deleting by query (throws err in Post Import
Delete query)*
Initially in solr.in.cmd the heap and GC settings were not enabled , now
it's enabled. Added the details (solr.in.jpeg) in the above link as well.
Uploaded the error log screenshot (solr_logging.jpeg)
Kindly help to check and advise .
Thanks
Reej
On Fri, Aug 13, 2021 at 9:57 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/13/2021 7:36 AM, Reej M wrote:
> > Have any of you found a solution for the Task Queue processing has
> stalled for 20077 ms with 0 remaining elements to process.
> > We are using solr 8.8.2, randomly we get this error while indexing. Is
> there any way we need to tune the solr.autocommit.maxtime?
> > For few cores we have it as 15000, for few we have it as 500, 1 or -1.
> Also the maxwarmingsearchers is set to 2
> > Kindly advise on how to tune this.
>
> A value of 500 or 1 is far too aggressive. It means that a commit will
> fire off either half a second or one millisecond after any updates are
> made. A value of -1 turns the feature off and you do not want that.
>
> The example configs have it set to 15000 (15 seconds), I personally
> would go with 60000. And I would make sure that openSearcher is set to
> false on autoCommit. That option on autoSoftCommit makes no sense, so
> you won't find it in example configs. For autoSoftCommit the maxTime
> value should be much larger -- two minutes (120000) is about as low as I
> would dare go on that.
>
> I don't know that any autoCommit setting is going to help with that
> particular message, though. The message was added to 8.4 by this issue:
>
> https://issues.apache.org/jira/browse/SOLR-13975
>
> With version before 8.4, the stall would still occur but you would not
> be notified, and it would probably stall forever -- 8.4 can apparently
> break the stall.
>
> As for what might be causing it, I do not know. It might be that your
> heap is too small, causing Java to spend a lot of time doing garbage
> collection in order to keep Solr running. Or it might be a general
> performance issue. If you can provide your GC logs that Solr writes, I
> can look into the possibility of garbage collection pauses. You'll need
> to put them on a file sharing site and provide links -- the mailing list
> eats message attachments.
>
> Thanks,
> Shawn
>
>
Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.8.2
Posted by Reej Nayagam <re...@gmail.com>.
Hi All,
Anyone is facing this issue as I am.
When the issue happens sometimes the index is erased completely and is 0.
Full Import
failed:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Error from server at
https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/: Server Error
request: https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/
Remote error message: Task queue processing has stalled for 20217 ms with 0
remaining elements to process.
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1193)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1102)
at org.apache.solr.handler.dataimport.SolrWriter.close(SolrWriter.java:61)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:282)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:427)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:486)
at
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:469)
at java.lang.Thread.run(Thread.java:748)
If I index directly through solr admin UI normally don't face any issues.
Indexing is success.
But when the indexing job is triggered by the scheduler job, I'm facing
this issue and error happens frequently.
*Thanks,*
*Reej*
On Fri, Aug 20, 2021 at 7:42 AM Reej Nayagam <re...@gmail.com> wrote:
> Hi Shawn,
> Thanks for your time to check the logs.
> we are getting the " ErrorReporingConcurrentUpdateSolrClinet Error when
> calling solrcmdDistributor"
>
> Please find the log details :
> https://drive.google.com/file/d/1lj6oR4pS9hfgySP-QBY5ipnAZXfAUWFT/view?usp=sharing
>
> And it comes frequently for the indexing that happens every 15 mins. This
> happens with solr 8.8.2.
> In production we have the same data-config.xml running Solr 4.10.4, we
> don't have any issues in production.
> In solr 4 we were using a solrserver connection passing the leader url for
> indexing.
> Now with solr8, as suggested we are using cloudsolrclient passing the ip's
> of the zookeepers.
> What we assume is when indexing is called it does it in one node, and when
> it tries to do a distributed update another node fails, not sure why this
> is happening.
> Any advice is highly appreciated.
>
> *Thanks,*
> *Reej*
>
>
> On Tue, Aug 17, 2021 at 9:52 PM Shawn Heisey <ap...@elyograg.org> wrote:
>
>> On 8/16/2021 5:17 AM, Reej Nayagam wrote:
>> > Hi Shawn ,
>> > Please find the gc logs from our test server. Your help is much
>> appreciated.
>> >
>> >
>> https://drive.google.com/drive/folders/1VQqlRxiDnHWPUlmTUv83a8CqCtvBP4le?usp=sharing
>>
>> I analyzed the logs with the gceasy.io website. Garbage collection is
>> healthy. I did not expect to see Solr's default heap size of 512MB ...
>> most users must increase that for success. Are we dealing with a very
>> small index here?
>>
>> > Also our Solr version is 8.8.2, so if there is a fix after 8.4+ , the
>> 8.8.2
>> > version should have this anyway right.
>>
>>
>> As for the "stall" message ... that's part of the code update for that
>> issue. The fact that you are seeing the message means that Solr is
>> breaking out of the stalled process -- there's nothing left to fix in
>> the code.
>>
>> > I've increased the autocommit maxtime to 60000, opensearcher is false
>> and
>> > autosoftcommit maxtime to 120000 to check just for 2 collections, and
>> for
>> > those 2 I'm not getting the Overlapping on Decksearchers so frequently ,
>> > but the below errors are happening frequently.
>> >
>> > *Request Handler Base : java.io.IOException:Task Queue processing has
>> > stalled for 20191 ms with 0 remaining elements to process*
>> > *Httpsolrcall : java.io.IOException:Task Queue processing has stalled
>> for
>> > 20191 ms with 0 remaining elements to process*
>> > *Solrwriter : Exception while deleting by query (throws err in Post
>> Import
>> > Delete query)*
>>
>>
>> As already mentioned, the fact that you are seeing this message means
>> that you have the fix for that issue, which was not reported before. As
>> for what's causing it to stall, I have no idea. It's not garbage
>> collection pauses, which was the candidate I was expecting to be a
>> problem.
>>
>> > Initially in solr.in.cmd the heap and GC settings were not enabled ,
>> now
>> > it's enabled. Added the details (solr.in.jpeg) in the above link as
>> well.
>> > Uploaded the error log screenshot (solr_logging.jpeg)
>>
>> You have explicitly enabled Solr's default heap size of 512MB. Solr has
>> GC tuning even if you don't specify it yourself. I believe with 8.8.2
>> it will use G1GC. Your settings use CMS. I was actually the one that
>> came up with what's being used currently ... which I did long before the
>> bin\solr.cmd and bin/solr scripts were created. See my persoinal wiki
>> page on the Solr wiki:
>>
>> https://cwiki.apache.org/confluence/display/solr/shawnheisey
>>
>> The picture of the logging screen is not very helpful. Almost all of
>> the details are hidden. You can open each logging entry on the web page
>> by clicking the little "i" icon ... but the solr.log file would show all
>> the detail at once.
>>
>> Thanks,
>> Shawn
>>
>>
Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.8.2
Posted by Reej Nayagam <re...@gmail.com>.
Hi Shawn,
Thanks for your time to check the logs.
we are getting the " ErrorReporingConcurrentUpdateSolrClinet Error when
calling solrcmdDistributor"
Please find the log details :
https://drive.google.com/file/d/1lj6oR4pS9hfgySP-QBY5ipnAZXfAUWFT/view?usp=sharing
And it comes frequently for the indexing that happens every 15 mins. This
happens with solr 8.8.2.
In production we have the same data-config.xml running Solr 4.10.4, we
don't have any issues in production.
In solr 4 we were using a solrserver connection passing the leader url for
indexing.
Now with solr8, as suggested we are using cloudsolrclient passing the ip's
of the zookeepers.
What we assume is when indexing is called it does it in one node, and when
it tries to do a distributed update another node fails, not sure why this
is happening.
Any advice is highly appreciated.
*Thanks,*
*Reej*
On Tue, Aug 17, 2021 at 9:52 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/16/2021 5:17 AM, Reej Nayagam wrote:
> > Hi Shawn ,
> > Please find the gc logs from our test server. Your help is much
> appreciated.
> >
> >
> https://drive.google.com/drive/folders/1VQqlRxiDnHWPUlmTUv83a8CqCtvBP4le?usp=sharing
>
> I analyzed the logs with the gceasy.io website. Garbage collection is
> healthy. I did not expect to see Solr's default heap size of 512MB ...
> most users must increase that for success. Are we dealing with a very
> small index here?
>
> > Also our Solr version is 8.8.2, so if there is a fix after 8.4+ , the
> 8.8.2
> > version should have this anyway right.
>
>
> As for the "stall" message ... that's part of the code update for that
> issue. The fact that you are seeing the message means that Solr is
> breaking out of the stalled process -- there's nothing left to fix in
> the code.
>
> > I've increased the autocommit maxtime to 60000, opensearcher is false
> and
> > autosoftcommit maxtime to 120000 to check just for 2 collections, and for
> > those 2 I'm not getting the Overlapping on Decksearchers so frequently ,
> > but the below errors are happening frequently.
> >
> > *Request Handler Base : java.io.IOException:Task Queue processing has
> > stalled for 20191 ms with 0 remaining elements to process*
> > *Httpsolrcall : java.io.IOException:Task Queue processing has stalled for
> > 20191 ms with 0 remaining elements to process*
> > *Solrwriter : Exception while deleting by query (throws err in Post
> Import
> > Delete query)*
>
>
> As already mentioned, the fact that you are seeing this message means
> that you have the fix for that issue, which was not reported before. As
> for what's causing it to stall, I have no idea. It's not garbage
> collection pauses, which was the candidate I was expecting to be a problem.
>
> > Initially in solr.in.cmd the heap and GC settings were not enabled , now
> > it's enabled. Added the details (solr.in.jpeg) in the above link as
> well.
> > Uploaded the error log screenshot (solr_logging.jpeg)
>
> You have explicitly enabled Solr's default heap size of 512MB. Solr has
> GC tuning even if you don't specify it yourself. I believe with 8.8.2
> it will use G1GC. Your settings use CMS. I was actually the one that
> came up with what's being used currently ... which I did long before the
> bin\solr.cmd and bin/solr scripts were created. See my persoinal wiki
> page on the Solr wiki:
>
> https://cwiki.apache.org/confluence/display/solr/shawnheisey
>
> The picture of the logging screen is not very helpful. Almost all of
> the details are hidden. You can open each logging entry on the web page
> by clicking the little "i" icon ... but the solr.log file would show all
> the detail at once.
>
> Thanks,
> Shawn
>
>
Re: ConcurrentUpdateSolrClient stall prevention bug in Solr 8.8.2
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/16/2021 5:17 AM, Reej Nayagam wrote:
> Hi Shawn ,
> Please find the gc logs from our test server. Your help is much appreciated.
>
> https://drive.google.com/drive/folders/1VQqlRxiDnHWPUlmTUv83a8CqCtvBP4le?usp=sharing
I analyzed the logs with the gceasy.io website. Garbage collection is
healthy. I did not expect to see Solr's default heap size of 512MB ...
most users must increase that for success. Are we dealing with a very
small index here?
> Also our Solr version is 8.8.2, so if there is a fix after 8.4+ , the 8.8.2
> version should have this anyway right.
As for the "stall" message ... that's part of the code update for that
issue. The fact that you are seeing the message means that Solr is
breaking out of the stalled process -- there's nothing left to fix in
the code.
> I've increased the autocommit maxtime to 60000, opensearcher is false and
> autosoftcommit maxtime to 120000 to check just for 2 collections, and for
> those 2 I'm not getting the Overlapping on Decksearchers so frequently ,
> but the below errors are happening frequently.
>
> *Request Handler Base : java.io.IOException:Task Queue processing has
> stalled for 20191 ms with 0 remaining elements to process*
> *Httpsolrcall : java.io.IOException:Task Queue processing has stalled for
> 20191 ms with 0 remaining elements to process*
> *Solrwriter : Exception while deleting by query (throws err in Post Import
> Delete query)*
As already mentioned, the fact that you are seeing this message means
that you have the fix for that issue, which was not reported before. As
for what's causing it to stall, I have no idea. It's not garbage
collection pauses, which was the candidate I was expecting to be a problem.
> Initially in solr.in.cmd the heap and GC settings were not enabled , now
> it's enabled. Added the details (solr.in.jpeg) in the above link as well.
> Uploaded the error log screenshot (solr_logging.jpeg)
You have explicitly enabled Solr's default heap size of 512MB. Solr has
GC tuning even if you don't specify it yourself. I believe with 8.8.2
it will use G1GC. Your settings use CMS. I was actually the one that
came up with what's being used currently ... which I did long before the
bin\solr.cmd and bin/solr scripts were created. See my persoinal wiki
page on the Solr wiki:
https://cwiki.apache.org/confluence/display/solr/shawnheisey
The picture of the logging screen is not very helpful. Almost all of
the details are hidden. You can open each logging entry on the web page
by clicking the little "i" icon ... but the solr.log file would show all
the detail at once.
Thanks,
Shawn