Recent configuration change to our site causes frequent index corruption

[cyndefromva <cy...@gmail.com>]

I have Rails 5 application that uses solr to index and search our site. The
sunspot gem is used to integrate ruby and sunspot.  It's a relatively small
site (no more 100,000 records) and has moderate usage (except for the
googlebot).

Until recently we regularly received 503 errors; reloading the page
generally cleared it up, but that was not exactly the user experience we
wantedso we added the following initializer to force the retry on failures:

Sunspot.session =
Sunspot::SessionProxy::Retry5xxSessionProxy.new(Sunspot.session)

As a result, about every third day the site locks up until we rebuild the
data directory (stop solr, move data directory to another location, start
solr, reindex). 

At the point it starts failing I see a java exception: "java.io-IOException:
Too many open files" in the solr log file and a SolrException (Error open
new searcher) is returned to the user.

In the solrconfig.xml file we have autoCommit and autoSoftCommit set as
follows:

  <autoCommit>
     <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
     <openSearcher>false</openSearcher>
  </autoCommit>

  <autoSoftCommit>
     <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
  </autoSoftCommit>

Which I believe means there should be a hard commit every 15 seconds.

But it appears to be calling commit more frequently. In the solr log I see
the following commit written miliseconds from each other:

  UpdateHandler start
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

I also see the following written right below it:

PERFORMANCE WARNING: Overlapping onDeckSearchers=2

Note: maxWarmingSearchers is set to 2.


I would really appreciate any help I can get to resolve this issue.

Thank you!





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Recent configuration change to our site causes frequent index corruption

[Erick Erickson <er...@gmail.com>]

bq: Error opening new searcher. exceeded limit of maxWarmingSearchers=2

did you make sure that your indexing client isn't issuing commits all
the time? The other possible culprit (although I'd be very surprised)
is if you have your filterCache and queryResultCache autowarm settings
set extremely high. I usually start with 16 or so.

Best,
Erick

On Fri, Jul 27, 2018 at 10:02 AM, cyndefromva <cy...@gmail.com> wrote:
> That makes sense, the ulimit was too small and I've updated it.
>
> I'm just curious why are there still so many 503 errors being generated
> (Error - Rsolr::Error::Http - 503 Service Unavailable - retrying ...)
>
> Is it related to all the "Error opening new searcher. exceeded limit of
> maxWarmingSearchers=2, try again later" java exceptions and if so is there a
> way to reduce these?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Recent configuration change to our site causes frequent index corruption

[Shawn Heisey <ap...@elyograg.org>]

On 7/27/2018 11:02 AM, cyndefromva wrote:
> I'm just curious why are there still so many 503 errors being generated
> (Error - Rsolr::Error::Http - 503 Service Unavailable - retrying ...)
>
> Is it related to all the "Error opening new searcher. exceeded limit of
> maxWarmingSearchers=2, try again later" java exceptions and if so is there a
> way to reduce these?

No, that's not likely to be the cause of the http errors.  Even if there
are problems opening a *new* searcher, there should be an existing
searcher that works just fine.

To get rid of that message in the log, you need to do commits a lot less
frequently.

https://wiki.apache.org/solr/FAQ#What_does_.22exceeded_limit_of_maxWarmingSearchers.3DX.22_mean.3F

If it Solr that is returning the 503 error, there should be ERROR
entries in the solr.log file.

Usually 503 is an error returned by a load balancer or a proxy, not a
web service.  If there are *extremely* serious problems with the Solr
install, Solr might return a 503, but those kinds of problems will
generally keep Solr from working at all.  If Solr sometimes works, then
I don't think it's very likely that Solr is returning the 503.

If there is something sitting between your client and Solr, you would
need to check the logs on that system as well as Solr's logs.  You may
also need to discuss this with the people who created sunspot. 

Thanks,
Shawn

Re: Recent configuration change to our site causes frequent index corruption

[cyndefromva <cy...@gmail.com>]

That makes sense, the ulimit was too small and I've updated it. 

I'm just curious why are there still so many 503 errors being generated
(Error - Rsolr::Error::Http - 503 Service Unavailable - retrying ...)

Is it related to all the "Error opening new searcher. exceeded limit of
maxWarmingSearchers=2, try again later" java exceptions and if so is there a
way to reduce these?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Recent configuration change to our site causes frequent index corruption

[Shawn Heisey <ap...@elyograg.org>]

On 7/26/2018 1:32 PM, cyndefromva wrote:
> At the point it starts failing I see a java exception: "java.io-IOException:
> Too many open files" in the solr log file and a SolrException (Error open
> new searcher) is returned to the user.

The operating system where Solr is running needs its open file limit 
increased.  Exactly how to do this will depend on what OS it is.  Most 
Linux systems need to have /etc/security/limits.conf edited.

> But it appears to be calling commit more frequently. In the solr log I see
> the following commit written miliseconds from each other:
>
>    UpdateHandler start
> commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
>
> I also see the following written right below it:
>
> PERFORMANCE WARNING: Overlapping onDeckSearchers=2

You have autoCommit with openSearcher set to false.  The commit log 
entry you've pasted is openSearcher=true ... it is not coming from 
autoCommit.

The overlapping onDeckSearchers is also not being caused by autoCommit 
-- that will not open a new searcher.  The commit log also has 
softCommit=false, which means that it is an explicit hard commit, most 
likely coming from your indexing application.  If autoSoftCommit or 
commitWithin were happening, it would be a soft commit.

Thanks,
Shawn