You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Reej M <re...@gmail.com> on 2021/08/24 07:51:20 UTC
Re: Help Needed: Distributed update Async Exception solr 8.8.2 -
Update
Hi All,
I’ve went through the section "Ignoring Commits from Client Applications in SolrCloud” in the solr 8.8 documentation.
Our Java application is issuing a commit after indexing, so as of I understood from the document, when running in cloud mode, the applications should not explicitly send a commit request.
So as mentioned in the doc, I’ve updated the solrconfig.xml with the updateRequestProcessorChain with the class IgnoreCommitOptimizeUpdateProcessorFactory.
After doing this, I’ve not seen the Async exception during distributed update error yet for the past 30 minutes.
So my doubt now is, Do we need to do any changes or wire anything for the optimisation and commit to be carried out effectively, as we are ignoring commit here or does solr internally will handle it.
Our auto commit maxtime is 60000 and Opensearcher - false
Auto soft commit maxtime : 180000 .
Please let us know if any changes are required in the config.
Thanks
Reej
> On 22 Aug 2021, at 7:10 PM, Reej Nayagam <re...@gmail.com> wrote:
>
> Hi All,
> We are getting a distributed update async exception during indexing. Solr version v8.8.2 and we are using cloudsolrclient and passing 3 zk ip’s for connecting. This indexing happens every 15 mins and at times the entire indexing is getting wiped out with data being pulled from the DB.
>
> Earlier we are using v4.10 and for indexing we used find the leader URL and connect for indexing and we didn’t have any issues with it.
>
> We need to push this new solr version 8.8.2 to prod and this issue is holding us back
>
> Please advise or help if there is any workaround.
>
>
> Full Import failed:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException: Async exception during distributed update: Error from server at https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/ <https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/>: Server Error
> request: https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/ <https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/>
> Remote error message: Task queue processing has stalled for 20217 ms with 0 remaining elements to process.
> at org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1193)
> at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1102)
> at org.apache.solr.handler.dataimport.SolrWriter.close(SolrWriter.java:61)
> at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:282)
> at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:427)
> at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:486)
> at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:469)
> at java.lang.Thread.run(Thread.java:748)
>
> --
> Thanks,
> Reej
Re: Help Needed: Distributed update Async Exception solr 8.8.2 - Update
Posted by Reej Nayagam <re...@gmail.com>.
Hi Shawn,
Thanks a lot. Managed to stop the commits from client side and no more
errors now.
Appreciate your timely response. Thank you !
*Regards,*
*Reej*
On Wed, Aug 25, 2021 at 7:59 PM Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/24/2021 10:08 PM, Reej Nayagam wrote:
> > Okay, Got your point. But we cannot modify the java code to stop
> commits
> > for now. So my manager suggests we comment out the auto commit in
> > solrconfig instead, We are not sure if that is correct. His point is, let
> > us commit every time we index (that is through java passing the commit &
> > optimise params) and remove the autocommit config in solrconfig.xml to
> > commit every 60000 milliseconds. WIll it be the right approach?
>
> Don't remove the autoCommit. Frequent hard commits are vital for good
> operation -- it flushes data to disk and starts a new transaction log.
> Doing it with openSearcher set to false makes it VERY fast.
>
> Solr ships with autoCommit at 15000 -- up to four times more frequently
> than you have it configured ... and it doesn't cause problems for
> users. I like to increase that to 60000 just so things are a little bit
> less busy, but 15000 would work too.
>
> The commits that were causing problems for you are the ones sent by your
> indexing software, and those commits DO open a new searcher. Opening a
> new searcher is the expensive part of a commit ... so your autoCommit is
> not a problem.
>
>
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> (the article says SolrCloud ... but it applies just as much when Solr is
> NOT in Cloud mode)
>
> Thanks,
> Shawn
>
>
Re: Help Needed: Distributed update Async Exception solr 8.8.2 -
Update
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/24/2021 10:08 PM, Reej Nayagam wrote:
> Okay, Got your point. But we cannot modify the java code to stop commits
> for now. So my manager suggests we comment out the auto commit in
> solrconfig instead, We are not sure if that is correct. His point is, let
> us commit every time we index (that is through java passing the commit &
> optimise params) and remove the autocommit config in solrconfig.xml to
> commit every 60000 milliseconds. WIll it be the right approach?
Don't remove the autoCommit. Frequent hard commits are vital for good
operation -- it flushes data to disk and starts a new transaction log.
Doing it with openSearcher set to false makes it VERY fast.
Solr ships with autoCommit at 15000 -- up to four times more frequently
than you have it configured ... and it doesn't cause problems for
users. I like to increase that to 60000 just so things are a little bit
less busy, but 15000 would work too.
The commits that were causing problems for you are the ones sent by your
indexing software, and those commits DO open a new searcher. Opening a
new searcher is the expensive part of a commit ... so your autoCommit is
not a problem.
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
(the article says SolrCloud ... but it applies just as much when Solr is
NOT in Cloud mode)
Thanks,
Shawn
Re: Help Needed: Distributed update Async Exception solr 8.8.2 - Update
Posted by Reej Nayagam <re...@gmail.com>.
On Wed, Aug 25, 2021 at 12:03 AM Shawn Heisey <el...@elyograg.org> wrote:
> On 8/24/2021 1:51 AM, Reej M wrote:
> > So my doubt now is, Do we need to do any changes or wire anything for
> the optimisation and commit to be carried out effectively, as we are
> ignoring commit here or does solr internally will handle it.
> > Our auto commit maxtime is 60000 and Opensearcher - false
> > Auto soft commit maxtime : 180000 .
>
>
> This will do a soft commit three minutes after new data is indexed.
> That should be sufficient. The automatic commits will still happen even
> if you are ignoring commit requests. Note that you should modify your
> indexing software to not send commit or optimize, rather than relying on
> the update processor to ignore them.
>
Okay, Got your point. But we cannot modify the java code to stop commits
for now. So my manager suggests we comment out the auto commit in
solrconfig instead, We are not sure if that is correct. His point is, let
us commit every time we index (that is through java passing the commit &
optimise params) and remove the autocommit config in solrconfig.xml to
commit every 60000 milliseconds. WIll it be the right approach?
>
> Thanks,
> Shawn
>
>
Re: Help Needed: Distributed update Async Exception solr 8.8.2 -
Update
Posted by Shawn Heisey <el...@elyograg.org>.
On 8/24/2021 1:51 AM, Reej M wrote:
> So my doubt now is, Do we need to do any changes or wire anything for the optimisation and commit to be carried out effectively, as we are ignoring commit here or does solr internally will handle it.
> Our auto commit maxtime is 60000 and Opensearcher - false
> Auto soft commit maxtime : 180000 .
This will do a soft commit three minutes after new data is indexed.
That should be sufficient. The automatic commits will still happen even
if you are ignoring commit requests. Note that you should modify your
indexing software to not send commit or optimize, rather than relying on
the update processor to ignore them.
Thanks,
Shawn
Fwd: Help Needed: Distributed update Async Exception solr 8.8.2 - Update
Posted by Reej Nayagam <re...@gmail.com>.
Hi All,
I’ve went through the section "Ignoring Commits from Client Applications
in SolrCloud” in the solr 8.8 documentation.
Our Java application is issuing a commit after indexing, so as of I
understood from the document, when running in cloud mode, the applications
should not explicitly send a commit request.
So as mentioned in the doc, I’ve updated the solrconfig.xml with the
*updateRequestProcessorChain* with the class
*IgnoreCommitOptimizeUpdateProcessorFactory.*
After doing this, I’ve not seen the *Async exception during distributed
update error *yet for the past 30 minutes.
So my doubt now is, Do we need to do any changes or wire anything for
the optimisation and commit to be carried out effectively, as we are
ignoring commit here or does solr internally will handle it.
Our auto commit maxtime is 60000 and Opensearcher - false
Auto soft commit maxtime : 180000 .
Please let us know if any changes are required in the config.
Also, I could see the Error in solrwriter in the postimportdeletequery
"Exception while deleting query: "
So not sure if this has completely fixed or it'll reoccur again, confused.
*Thanks*
*Reej*
On 22 Aug 2021, at 7:10 PM, Reej Nayagam <re...@gmail.com> wrote:
Hi All,
We are getting a distributed update async exception during indexing. Solr
version v8.8.2 and we are using cloudsolrclient and passing 3 zk ip’s for
connecting. This indexing happens every 15 mins and at times the entire
indexing is getting wiped out with data being pulled from the DB.
Earlier we are using v4.10 and for indexing we used find the leader URL and
connect for indexing and we didn’t have any issues with it.
We need to push this new solr version 8.8.2 to prod and this issue is
holding us back
Please advise or help if there is any workaround.
Full Import
failed:org.apache.solr.update.processor.DistributedUpdateProcessor$DistributedUpdatesAsyncException:
Async exception during distributed update: Error from server at
https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/: Server Error
request: https://192.168.3.153:3883/solr/PUBLIC_shard1_replica_n1/
Remote error message: Task queue processing has stalled for 20217 ms with 0
remaining elements to process.
at
org.apache.solr.update.processor.DistributedZkUpdateProcessor.doDistribFinish(DistributedZkUpdateProcessor.java:1193)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1102)
at org.apache.solr.handler.dataimport.SolrWriter.close(SolrWriter.java:61)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:282)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:427)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:486)
at
org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:469)
at java.lang.Thread.run(Thread.java:748)
--
*Thanks,*
*Reej*