You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Wei <we...@gmail.com> on 2017/09/29 04:55:40 UTC
solr cloud without hard commit?
Hello All,
What are the impacts if solr cloud is configured to have only soft commits
but no hard commits? In this way if a non-leader node crashes, will it
still be able to recover from the leader? Basically we are wondering in a
read heavy & write heavy scenario, whether taking hard commit out could
help to improve query performance and what are the consequences.
Thanks,
Wei
Re: solr cloud without hard commit?
Posted by Erick Erickson <er...@gmail.com>.
ramBufferSizeMB limits the amount of memory used for indexing and when
it's exceeded the buffer is flushed to disk into a new segment. This
is independent of hard/soft commits.
Soft commits do not _force_ the in-memory structures to be written to
a segment and do not update the segments file, hard commits do both of
these things. For the purposes of that blog I decided it just leads to
confusion to mention segments being created wrt soft commits; they're
not "real" until a hard commit.
bq: If the indexing rate is high, there's not really much difference
between soft commits and hard commits
Not necessarily true. Soft commits open a new searcher and to all the
autowarming, whereas hard commits with openSearcher=false do not. This
can be a _very_ expensive operation.
Second difference: the segments file is not updated unless you hard
commit. So even though I have a bunch of segments written to disk, if
I crash my server they won't be found on restart. Graceful shutdown is
a different story.
At least that's my understanding.
Best,
Erick
On Mon, Oct 2, 2017 at 2:49 AM, alessandro.benedetti
<a....@sease.io> wrote:
> Hi Erick,
> you said :
> ""mentions that for soft commit, "new segments are created that will
> be merged""
>
> Wait, how did that get in there? Ignore it, I'm taking it out. "
>
> but I think you were not wrong, based on another mailing list thread message
> by Shawn, I read :
> [1]
>
> "If you are using the correct DirectoryFactory type, a soft commit has
> the *possibility* of not writing to disk, but the amount of memory
> reserved is fairly small.
>
> Looking into the source code for NRTCachingDirectoryFactory, I see that
> maxMergeSizeMB defaults to 4, and maxCachedMB defaults to 48. This is a
> little bit different than what the javadoc states for
> NRTCachingDirectory (5 and 60):
>
> http://lucene.apache.org/core/6_6_0/core/org/apache/lucene/store/NRTCachingDirectory.html
>
> The way I read this, assuming the amount of segment data created is
> small, only the first few soft commits will be entirely handled in
> memory. After that, older segments must be flushed to disk to make room
> for new ones.
>
> If the indexing rate is high, there's not really much difference between
> soft commits and hard commits. This also assumes that you have left the
> directory at the default of NRTCachingDirectoryFactory. If this has
> been changed, then there is no caching in RAM, and soft commit probably
> behaves *exactly* the same as hard commit.
> "
>
> [1]
> http://lucene.472066.n3.nabble.com/High-disk-write-usage-td4344356.html#a4344551
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: solr cloud without hard commit?
Posted by "alessandro.benedetti" <a....@sease.io>.
Hi Erick,
you said :
""mentions that for soft commit, "new segments are created that will
be merged""
Wait, how did that get in there? Ignore it, I'm taking it out. "
but I think you were not wrong, based on another mailing list thread message
by Shawn, I read :
[1]
"If you are using the correct DirectoryFactory type, a soft commit has
the *possibility* of not writing to disk, but the amount of memory
reserved is fairly small.
Looking into the source code for NRTCachingDirectoryFactory, I see that
maxMergeSizeMB defaults to 4, and maxCachedMB defaults to 48. This is a
little bit different than what the javadoc states for
NRTCachingDirectory (5 and 60):
http://lucene.apache.org/core/6_6_0/core/org/apache/lucene/store/NRTCachingDirectory.html
The way I read this, assuming the amount of segment data created is
small, only the first few soft commits will be entirely handled in
memory. After that, older segments must be flushed to disk to make room
for new ones.
If the indexing rate is high, there's not really much difference between
soft commits and hard commits. This also assumes that you have left the
directory at the default of NRTCachingDirectoryFactory. If this has
been changed, then there is no caching in RAM, and soft commit probably
behaves *exactly* the same as hard commit.
"
[1]
http://lucene.472066.n3.nabble.com/High-disk-write-usage-td4344356.html#a4344551
-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: solr cloud without hard commit?
Posted by Erick Erickson <er...@gmail.com>.
1> "mentions that for soft commit, "new segments are created that will
be merged""
Wait, how did that get in there? Ignore it, I'm taking it out.
2> it doesn't matter whether segments are fsync'd or not. There will
be a series of segments written to disk when ramBufferSizeMB is
exceeded without a hard commit. But they are _not_ recorded in the
segments file. So even though they may be perfectly intact, they
aren't "found" if Solr has to restart. The sequence for hard commit
is:
a> flush the segment (actually, fsync)
b> write the names of current valid segments to the segments file.
When a searcher is opened, it gets the list of current valid segments
from the segments file. Since <a> doesn't get done on soft commit, any
segments written to disk since the last commit are invisible.
We haven't found much if any improvement by setting ramBufferSizeMB
greater than the default. There'll be less merging if you raise it,
but it's generally best to just leave it alone.
3> Yes and Yes. If you disable transaction logs (the <updateLog> entry
in solrconfig.xml), then PeerSync won't work. Replication will work
since that only replicates segment files, but that's old-style
master/slave replication.
Best,
Erick
On Fri, Sep 29, 2017 at 1:06 PM, Wei <we...@gmail.com> wrote:
> Thanks Emir and Erick! Helps me a lot to understand the commit process. A
> few more questions:
>
> 1. https://lucidworks.com/2013/08/23/understanding-
> transaction-logs-softcommit-and-commit-in-sorlcloud/ mentions that for
> soft commit, "new segments are created that will be merged". Does that
> mean without hard commit, soft commits will create many small segments in
> memory and that could also slow down query? As I understand merge policy
> only kicks in with hard commit.
>
> 2. Without hard commit configure, will the segments still be fsync to disk
> when accumulated updates exceeds rambuffersizeMB? Is there any concern to
> increase rambuffersizeMB to a large value?
>
> 3. Can transaction logs be disabled in solr cloud? Will
> functionalities(replication, peer sync) break without transaction logs?
>
> Thanks,
> Wei
>
>
> On Fri, Sep 29, 2017 at 8:33 AM, Erick Erickson <er...@gmail.com>
> wrote:
>
>> More than you want to know about hard and soft commits here:
>> https://lucidworks.com/2013/08/23/understanding-
>> transaction-logs-softcommit-and-commit-in-sorlcloud/
>>
>> You don't need to read it though, Emir did an admirable job of telling
>> you why turning off hard commits is a terrible idea.
>>
>> Best,
>> Erick
>>
>> On Fri, Sep 29, 2017 at 1:07 AM, Emir Arnautović
>> <em...@sematext.com> wrote:
>> > Hi Wei,
>> > Hard commits are about data durability. It will roll over transaction
>> logs and create index new index segment. If configured with
>> openSearcher=false, they do not affect query performance much (other then
>> taking some resources) since they do not invalidate caches. If you have
>> transaction logs enabled, without hard commits it would grow infinitely and
>> can result in full disk. In case of heavy indexing, even rare hard commits
>> can result in large transaction logs causing Solr restart after crash to
>> take a while because transaction logs are replayed.
>> > Soft commits are the one that are affecting query performance and should
>> be as rare as your requirements allow. They invalidate caches causing cold
>> searches or if you have warming set up, take resources to do the warming.
>> >
>> > I would recommend to keep hard commits, set to every 20-60 seconds
>> (depending on indexing volume) and make sure openSearcher is set to false.
>> >
>> > HTH,
>> > Emir
>> >
>> >> On 29 Sep 2017, at 06:55, Wei <we...@gmail.com> wrote:
>> >>
>> >> Hello All,
>> >>
>> >> What are the impacts if solr cloud is configured to have only soft
>> commits
>> >> but no hard commits? In this way if a non-leader node crashes, will it
>> >> still be able to recover from the leader? Basically we are wondering
>> in a
>> >> read heavy & write heavy scenario, whether taking hard commit out could
>> >> help to improve query performance and what are the consequences.
>> >>
>> >> Thanks,
>> >> Wei
>> >
>>
Re: solr cloud without hard commit?
Posted by Wei <we...@gmail.com>.
Thanks Emir and Erick! Helps me a lot to understand the commit process. A
few more questions:
1. https://lucidworks.com/2013/08/23/understanding-
transaction-logs-softcommit-and-commit-in-sorlcloud/ mentions that for
soft commit, "new segments are created that will be merged". Does that
mean without hard commit, soft commits will create many small segments in
memory and that could also slow down query? As I understand merge policy
only kicks in with hard commit.
2. Without hard commit configure, will the segments still be fsync to disk
when accumulated updates exceeds rambuffersizeMB? Is there any concern to
increase rambuffersizeMB to a large value?
3. Can transaction logs be disabled in solr cloud? Will
functionalities(replication, peer sync) break without transaction logs?
Thanks,
Wei
On Fri, Sep 29, 2017 at 8:33 AM, Erick Erickson <er...@gmail.com>
wrote:
> More than you want to know about hard and soft commits here:
> https://lucidworks.com/2013/08/23/understanding-
> transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> You don't need to read it though, Emir did an admirable job of telling
> you why turning off hard commits is a terrible idea.
>
> Best,
> Erick
>
> On Fri, Sep 29, 2017 at 1:07 AM, Emir Arnautović
> <em...@sematext.com> wrote:
> > Hi Wei,
> > Hard commits are about data durability. It will roll over transaction
> logs and create index new index segment. If configured with
> openSearcher=false, they do not affect query performance much (other then
> taking some resources) since they do not invalidate caches. If you have
> transaction logs enabled, without hard commits it would grow infinitely and
> can result in full disk. In case of heavy indexing, even rare hard commits
> can result in large transaction logs causing Solr restart after crash to
> take a while because transaction logs are replayed.
> > Soft commits are the one that are affecting query performance and should
> be as rare as your requirements allow. They invalidate caches causing cold
> searches or if you have warming set up, take resources to do the warming.
> >
> > I would recommend to keep hard commits, set to every 20-60 seconds
> (depending on indexing volume) and make sure openSearcher is set to false.
> >
> > HTH,
> > Emir
> >
> >> On 29 Sep 2017, at 06:55, Wei <we...@gmail.com> wrote:
> >>
> >> Hello All,
> >>
> >> What are the impacts if solr cloud is configured to have only soft
> commits
> >> but no hard commits? In this way if a non-leader node crashes, will it
> >> still be able to recover from the leader? Basically we are wondering
> in a
> >> read heavy & write heavy scenario, whether taking hard commit out could
> >> help to improve query performance and what are the consequences.
> >>
> >> Thanks,
> >> Wei
> >
>
Re: solr cloud without hard commit?
Posted by Erick Erickson <er...@gmail.com>.
More than you want to know about hard and soft commits here:
https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
You don't need to read it though, Emir did an admirable job of telling
you why turning off hard commits is a terrible idea.
Best,
Erick
On Fri, Sep 29, 2017 at 1:07 AM, Emir Arnautović
<em...@sematext.com> wrote:
> Hi Wei,
> Hard commits are about data durability. It will roll over transaction logs and create index new index segment. If configured with openSearcher=false, they do not affect query performance much (other then taking some resources) since they do not invalidate caches. If you have transaction logs enabled, without hard commits it would grow infinitely and can result in full disk. In case of heavy indexing, even rare hard commits can result in large transaction logs causing Solr restart after crash to take a while because transaction logs are replayed.
> Soft commits are the one that are affecting query performance and should be as rare as your requirements allow. They invalidate caches causing cold searches or if you have warming set up, take resources to do the warming.
>
> I would recommend to keep hard commits, set to every 20-60 seconds (depending on indexing volume) and make sure openSearcher is set to false.
>
> HTH,
> Emir
>
>> On 29 Sep 2017, at 06:55, Wei <we...@gmail.com> wrote:
>>
>> Hello All,
>>
>> What are the impacts if solr cloud is configured to have only soft commits
>> but no hard commits? In this way if a non-leader node crashes, will it
>> still be able to recover from the leader? Basically we are wondering in a
>> read heavy & write heavy scenario, whether taking hard commit out could
>> help to improve query performance and what are the consequences.
>>
>> Thanks,
>> Wei
>
Re: solr cloud without hard commit?
Posted by Emir Arnautović <em...@sematext.com>.
Hi Wei,
Hard commits are about data durability. It will roll over transaction logs and create index new index segment. If configured with openSearcher=false, they do not affect query performance much (other then taking some resources) since they do not invalidate caches. If you have transaction logs enabled, without hard commits it would grow infinitely and can result in full disk. In case of heavy indexing, even rare hard commits can result in large transaction logs causing Solr restart after crash to take a while because transaction logs are replayed.
Soft commits are the one that are affecting query performance and should be as rare as your requirements allow. They invalidate caches causing cold searches or if you have warming set up, take resources to do the warming.
I would recommend to keep hard commits, set to every 20-60 seconds (depending on indexing volume) and make sure openSearcher is set to false.
HTH,
Emir
> On 29 Sep 2017, at 06:55, Wei <we...@gmail.com> wrote:
>
> Hello All,
>
> What are the impacts if solr cloud is configured to have only soft commits
> but no hard commits? In this way if a non-leader node crashes, will it
> still be able to recover from the leader? Basically we are wondering in a
> read heavy & write heavy scenario, whether taking hard commit out could
> help to improve query performance and what are the consequences.
>
> Thanks,
> Wei