Re: Too many segments flushed (was: Too many commits)

[Patrick Recchia <pa...@gmail.com>]

Thanks for the great info.
Both of you.

Yes, I know we have configuration issue with commitWithin and autoCommit.
I cannot change our config freely, so this will probably stay like this for
some time.
I will try to indeed introduce autoSoftCommit (60 seconds is our maximus,
as we try to achieve 'near real time') at some point.
Same goes for ramBufferSizeMB.

By looking at the logs, it is clear we commit every minute

On the other hand The Infostream logging say also that we flush a new
segment every second. Because of the flush policy being hit
(FlushByRamOrCountsPolicy)
Which triggers a flush when total ram exceeds limit of 100MB.

My impression is that, when the flush occurs, there are several (how many?)
segments in memory, and only the largest one is being flushed. so only a
limited amount of memory.

Is that correct? Does solr (lucene, actually, at this point) create a
seperate segment per indexing thread?
In that case, maybe is it worth mentioning in the documentation,
under ramBufferSizeMB; that this buffer will be split among all indexing
threads?

I tried to go through the code to understand how and when new segments are
created, but couldn't understand it fully.

Thanks,
Patrick








On Thu, May 3, 2018 at 1:27 AM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 5/2/2018 11:45 AM, Patrick Recchia wrote:
> > Is there any logging I can turn on to know when a commit happens and/or
> > when a segment is flushed?
>
> The normal INFO-level logging that Solr ships with will log all
> commits.  It probably doesn't log segment flushes unless they happen as
> a result of a commit, though.  The infoStream logging would have that
> information.
>
> Your autoCommit settings are ensuring that commitWithin is never going
> to actually cause a commit.  Your interval for autoCommit is 60000 (one
> minute), commitWithin is 500000 (a little over eight minutes).  The
> autoCommit has openSearcher set to true, so there will always be a
> commit with a new searcher occurring within one minute after an update
> is sent, and commitWithin will never be needed.
>
> Here's what I think I would try:  On autoCommit, set openSearcher to
> false.  If you want to have less than an eight minute window for
> document visibility, then reduce commitWithin to 120000.  Increase
> ramBufferSizeMB to 256 or 512, which might require an increase in heap
> size as well.  Instead of using commitWithin, you could configure
> autoSoftCommit with a maxTime of 120000.
>
> Here's some additional info about commits:
>
> https://lucidworks.com/2013/08/23/understanding-
> transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> The title says "SolrCloud" but the concepts are equally applicable when
> not running in cloud mode.
>
> Thanks,
> Shawn
>
>


-- 
One way of describing a computer is as an electric box which hums.
Never ascribe to malice what can be explained by stupidity
--
Patrick Recchia
GSM (BE): +32 486 828311
GSM(IT): +39 347 2300830