Handling failure when adding docs to Solr using SolrJ

[Steven White <sw...@gmail.com>]

Hi everyone,

I'm trying to figure out when and how I should handle failures that may
occur during indexing.  In the sample code below, look at my comment and
let me know what state my index is in when things fail:

   SolrClient solrClient = new HttpSolrClient.Builder(url).build();

   solrClient.add(solrDocs);

   // #1: What to do if add() fails?  And how do I know if all or some of
my docs in 'solrDocs' made it to the index or not ('solrDocs' is a list of
1 or more doc), should I retry add() again?  Retry with a smaller chunk?
Etc.

   if (doCommit == true)
   {
      solrClient.commit();

       // #2: What to do if commit() fails?  Re-issue commit() again?
   }

Thanks

Steven

Re: Handling failure when adding docs to Solr using SolrJ

[Erick Erickson <er...@gmail.com>]

I recommend _against_ issuing explicit commits from the client, let
your solrconfig.xml autocommit settings take care of it. Make sure
either your soft or hard commits open a new searcher for the docs
to be searchable.

I’ll bend a little bit if you can _guarantee_ that you only ever have one
indexing client running and basically only ever issue the commit at the
end.

There’s another strategy, do the solrClient.add() command with the
commitWithin parameter.

As far as failures, look at 
https://lucene.apache.org/solr/7_3_0/solr-core/org/apache/solr/update/processor/TolerantUpdateProcessor.html
that’ll give you a better clue about _which_ docs failed. From there, though,
it’s a bit if debugging to figure out why that particular doc failed, usually people
record the docs that failed for later analysis. and/or look at the Solr logs which
usually give a more detailed reason of _why_ a document failed...

Best,
Erick

> On Sep 17, 2020, at 1:09 PM, Steven White <sw...@gmail.com> wrote:
> 
> Hi everyone,
> 
> I'm trying to figure out when and how I should handle failures that may
> occur during indexing.  In the sample code below, look at my comment and
> let me know what state my index is in when things fail:
> 
>   SolrClient solrClient = new HttpSolrClient.Builder(url).build();
> 
>   solrClient.add(solrDocs);
> 
>   // #1: What to do if add() fails?  And how do I know if all or some of
> my docs in 'solrDocs' made it to the index or not ('solrDocs' is a list of
> 1 or more doc), should I retry add() again?  Retry with a smaller chunk?
> Etc.
> 
>   if (doCommit == true)
>   {
>      solrClient.commit();
> 
>       // #2: What to do if commit() fails?  Re-issue commit() again?
>   }
> 
> Thanks
> 
> Steven