You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Erick Erickson <er...@gmail.com> on 2015/03/31 01:55:24 UTC
Soft commits and segments
Is it the expected case that when a soft commit happens, a new segment
is opened? 'Cause that's what I'm seeing. Thinking about it I don't
see how Lucene could successfully MMap the underlying disk files if
new segments weren't opened, and if they were all held in Java's
memory "BOOM" (that as a Big OOM).
So I'm guessing at this point that I need to revise my model from
"soft commits do not write new segments" to "soft commits do not write
new segments _durably_".
Thanks,
Erick
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Soft commits and segments
Posted by Erick Erickson <er...@gmail.com>.
Robert, thanks a bunch, your comment got me looking in the right place
I'm seeing what I expect now. I can at least try to explain it without
stumbling too much..
Thanks again!
Erick
On Mon, Mar 30, 2015 at 5:24 PM, Robert Muir <rc...@gmail.com> wrote:
> Yes, pending docs buffered in indexwriter are flushed when you reopen.
>
> one way to avoid revising your model is to use NRTCachingDirectory.
> When properly configured this can defer the writes to the filesystem
> until they truly really need to be there, e.g. indexwriter.commit.
>
> On Mon, Mar 30, 2015 at 7:55 PM, Erick Erickson <er...@gmail.com> wrote:
>> Is it the expected case that when a soft commit happens, a new segment
>> is opened? 'Cause that's what I'm seeing. Thinking about it I don't
>> see how Lucene could successfully MMap the underlying disk files if
>> new segments weren't opened, and if they were all held in Java's
>> memory "BOOM" (that as a Big OOM).
>>
>> So I'm guessing at this point that I need to revise my model from
>> "soft commits do not write new segments" to "soft commits do not write
>> new segments _durably_".
>>
>> Thanks,
>> Erick
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Soft commits and segments
Posted by Robert Muir <rc...@gmail.com>.
Yes, pending docs buffered in indexwriter are flushed when you reopen.
one way to avoid revising your model is to use NRTCachingDirectory.
When properly configured this can defer the writes to the filesystem
until they truly really need to be there, e.g. indexwriter.commit.
On Mon, Mar 30, 2015 at 7:55 PM, Erick Erickson <er...@gmail.com> wrote:
> Is it the expected case that when a soft commit happens, a new segment
> is opened? 'Cause that's what I'm seeing. Thinking about it I don't
> see how Lucene could successfully MMap the underlying disk files if
> new segments weren't opened, and if they were all held in Java's
> memory "BOOM" (that as a Big OOM).
>
> So I'm guessing at this point that I need to revise my model from
> "soft commits do not write new segments" to "soft commits do not write
> new segments _durably_".
>
> Thanks,
> Erick
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org