Re: solr cloud without hard commit?

["alessandro.benedetti" <a....@sease.io>]

Hi Erick,
you said :
""mentions that for soft commit, "new segments are created that will 
be merged"" 

Wait, how did that get in there? Ignore it, I'm taking it out. "

but I think you were not wrong, based on another mailing list thread message
by Shawn, I read :
[1]

"If you are using the correct DirectoryFactory type, a soft commit has 
the *possibility* of not writing to disk, but the amount of memory 
reserved is fairly small. 

Looking into the source code for NRTCachingDirectoryFactory, I see that 
maxMergeSizeMB defaults to 4, and maxCachedMB defaults to 48.  This is a 
little bit different than what the javadoc states for 
NRTCachingDirectory (5 and 60): 

http://lucene.apache.org/core/6_6_0/core/org/apache/lucene/store/NRTCachingDirectory.html

The way I read this, assuming the amount of segment data created is 
small, only the first few soft commits will be entirely handled in 
memory.  After that, older segments must be flushed to disk to make room 
for new ones. 

If the indexing rate is high, there's not really much difference between 
soft commits and hard commits.  This also assumes that you have left the 
directory at the default of NRTCachingDirectoryFactory.  If this has 
been changed, then there is no caching in RAM, and soft commit probably 
behaves *exactly* the same as hard commit. 
"

[1]
http://lucene.472066.n3.nabble.com/High-disk-write-usage-td4344356.html#a4344551



-----
---------------
Alessandro Benedetti
Search Consultant, R&D Software Engineer, Director
Sease Ltd. - www.sease.io
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: solr cloud without hard commit?

[Erick Erickson <er...@gmail.com>]

ramBufferSizeMB limits the amount of memory used for indexing and when
it's exceeded the buffer is flushed to disk into a new segment. This
is independent of hard/soft commits.

Soft commits do not _force_ the in-memory structures to be written to
a segment and do not update the segments file, hard commits do both of
these things. For the purposes of that blog I decided it just leads to
confusion to mention segments being created wrt soft commits; they're
not "real" until a hard commit.

bq: If the indexing rate is high, there's not really much difference
between soft commits and hard commits

Not necessarily true. Soft commits open a new searcher and to all the
autowarming, whereas hard commits with openSearcher=false do not. This
can be a _very_ expensive operation.

Second difference: the segments file is not updated unless you hard
commit. So even though I have a bunch of segments written to disk, if
I crash my server they won't be found on restart. Graceful shutdown is
a different story.

At least that's my understanding.

Best,
Erick



On Mon, Oct 2, 2017 at 2:49 AM, alessandro.benedetti
<a....@sease.io> wrote:
> Hi Erick,
> you said :
> ""mentions that for soft commit, "new segments are created that will
> be merged""
>
> Wait, how did that get in there? Ignore it, I'm taking it out. "
>
> but I think you were not wrong, based on another mailing list thread message
> by Shawn, I read :
> [1]
>
> "If you are using the correct DirectoryFactory type, a soft commit has
> the *possibility* of not writing to disk, but the amount of memory
> reserved is fairly small.
>
> Looking into the source code for NRTCachingDirectoryFactory, I see that
> maxMergeSizeMB defaults to 4, and maxCachedMB defaults to 48.  This is a
> little bit different than what the javadoc states for
> NRTCachingDirectory (5 and 60):
>
> http://lucene.apache.org/core/6_6_0/core/org/apache/lucene/store/NRTCachingDirectory.html
>
> The way I read this, assuming the amount of segment data created is
> small, only the first few soft commits will be entirely handled in
> memory.  After that, older segments must be flushed to disk to make room
> for new ones.
>
> If the indexing rate is high, there's not really much difference between
> soft commits and hard commits.  This also assumes that you have left the
> directory at the default of NRTCachingDirectoryFactory.  If this has
> been changed, then there is no caching in RAM, and soft commit probably
> behaves *exactly* the same as hard commit.
> "
>
> [1]
> http://lucene.472066.n3.nabble.com/High-disk-write-usage-td4344356.html#a4344551
>
>
>
> -----
> ---------------
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html