default values for numRecordsToKeep and maxNumLogsToKeep

[suresh pendap <su...@gmail.com>]

Hi,
After looking at the source code I see that the default values for
numRecordsToKeep is 100 and maxNumLogsToKeep is 10.

So it seems by default the replica can only have 1000 document updates lag
before the replica goes for a Full recovery from the leader.

I would like to know the rationale for keeping so low values for these
configurations.

If the idea for these configuration params is to avoid full recovery for
down replicas then shouldn't the default values for these config params be
high?

I understand that higher values would mean more disk space consumed by the
update logs, but the current default values seem to be very low.

Is my understanding of these configuration params correct?

Thanks
Suresh

Re: default values for numRecordsToKeep and maxNumLogsToKeep

[Erick Erickson <er...@gmail.com>]

I'm going to punt on the rationale since I wasn't involved in that discussion.

numRecordsToKeep can be configured in the <updateLog> section of
solrconfig.xml if you want to change it though.

Best,
Erick

On Tue, Jul 18, 2017 at 10:53 AM, suresh pendap <su...@gmail.com> wrote:
> Hi,
> After looking at the source code I see that the default values for
> numRecordsToKeep is 100 and maxNumLogsToKeep is 10.
>
> So it seems by default the replica can only have 1000 document updates lag
> before the replica goes for a Full recovery from the leader.
>
> I would like to know the rationale for keeping so low values for these
> configurations.
>
> If the idea for these configuration params is to avoid full recovery for
> down replicas then shouldn't the default values for these config params be
> high?
>
> I understand that higher values would mean more disk space consumed by the
> update logs, but the current default values seem to be very low.
>
> Is my understanding of these configuration params correct?
>
> Thanks
> Suresh

Re: default values for numRecordsToKeep and maxNumLogsToKeep

[Erick Erickson <er...@gmail.com>]

bq:  I am pretty sure that anytime a core starts for *any* reason, all
the transaction logs that are present will get replayed.

This isn't quite true. If Solr is shut down gracefully, or a hard
commit happened before shutdown (with no new docs added) then the tlog
will _not_ be replayed on startup. It's only when Solr is killed
without a commit and thus the tlog is not truncated (and segments not
closed) by hard commit that the tlog will be replayed on startup.
Which is why I strongly recommend that people stop Solr with the
script rather than "kill -9".

Best,
Erick

On Thu, Jul 20, 2017 at 5:39 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 7/18/2017 11:53 AM, suresh pendap wrote:
>> After looking at the source code I see that the default values for
>> numRecordsToKeep is 100 and maxNumLogsToKeep is 10.
>>
>> So it seems by default the replica can only have 1000 document updates lag
>> before the replica goes for a Full recovery from the leader.
>
> I don't think that's quite right.  In many situations, the number of
> documents in the transaction log will likely be less than 1000.
>
> Enough logs will be kept that *at least* 100 documents are there, if
> that can be accomplished with ten logfiles or less.  Based on a quick
> reading of the code, if the newest ten logs have less than 100
> documents, then there will be less than 100 docs available.  This would
> not end up being a problem for data integrity, because small infrequent
> updates would be the only way to end up with less than 100 docs, and in
> that situation, the small number of documents in the transaction log,
> when replayed at core startup, will be enough to ensure integrity.
>
> I think the reasons the default numbers are so small is an attempt to
> keep startup time low.  I am pretty sure that anytime a core starts for
> *any* reason, all the transaction logs that are present will get
> replayed.  I know for sure that this happens on Solr restart; I think it
> also happens on core reload.  By keeping the required minimum documents
> at a low value like 100, there's a better chance that the transaction
> logs will be small, and therefore core startup will be fast.
>
> On a system where there are no hard commits, all updates end up going
> into a single super-large transaction log.  This meets the default
> configuration numbers, because there are less than ten logs present, and
> what is present contains at least 100 documents.  Unfortunately, this
> means that when the core starts, it will replay that HUGE transaction
> log, a process that could take hours.  This situation is prevented by
> enabling autoCommit with a relatively short maxTime value.  Setting
> openSearcher to false in the autoCommit ensures that document visibility
> behavior is not altered by autoCommit.
>
> Thanks,
> Shawn
>

Re: default values for numRecordsToKeep and maxNumLogsToKeep

[Shawn Heisey <ap...@elyograg.org>]

On 7/18/2017 11:53 AM, suresh pendap wrote:
> After looking at the source code I see that the default values for
> numRecordsToKeep is 100 and maxNumLogsToKeep is 10.
>
> So it seems by default the replica can only have 1000 document updates lag
> before the replica goes for a Full recovery from the leader.

I don't think that's quite right.  In many situations, the number of
documents in the transaction log will likely be less than 1000.

Enough logs will be kept that *at least* 100 documents are there, if
that can be accomplished with ten logfiles or less.  Based on a quick
reading of the code, if the newest ten logs have less than 100
documents, then there will be less than 100 docs available.  This would
not end up being a problem for data integrity, because small infrequent
updates would be the only way to end up with less than 100 docs, and in
that situation, the small number of documents in the transaction log,
when replayed at core startup, will be enough to ensure integrity.

I think the reasons the default numbers are so small is an attempt to
keep startup time low.  I am pretty sure that anytime a core starts for
*any* reason, all the transaction logs that are present will get
replayed.  I know for sure that this happens on Solr restart; I think it
also happens on core reload.  By keeping the required minimum documents
at a low value like 100, there's a better chance that the transaction
logs will be small, and therefore core startup will be fast.

On a system where there are no hard commits, all updates end up going
into a single super-large transaction log.  This meets the default
configuration numbers, because there are less than ten logs present, and
what is present contains at least 100 documents.  Unfortunately, this
means that when the core starts, it will replay that HUGE transaction
log, a process that could take hours.  This situation is prevented by
enabling autoCommit with a relatively short maxTime value.  Setting
openSearcher to false in the autoCommit ensures that document visibility
behavior is not altered by autoCommit.

Thanks,
Shawn