You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Gili Nachum <gi...@gmail.com> on 2016/01/04 11:14:11 UTC
Does soft commit re-opens searchers in disk?
Hello,
When a new document is added, it becomes visible after a soft commit,
during which it is written to a Lucene RAMDirectory (in heap). Then after a
hard commit, the RAMDirectory is removed from memory and the docs are
written to the index on disk.
What happens if I hard commit (write to disk) with openSearcher=false.
Would I lose document visibility? since it's no longer in memory AND the
hard commit didn't open a new searcher on disk?
Does soft commit also re-opens Searchers over the index on disk?
Here's my commit configuration:
<autoCommit>
<maxTime>600000</maxTime>
* <openSearcher>false</openSearcher>*
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
</autoSoftCommit>
Thanks.
Re: Does soft commit re-opens searchers in disk?
Posted by Daniel Collins <da...@gmail.com>.
Yes, my earlier statement was a little vague, apologies on that, a soft
commit inherently *has* to open a new searcher, a soft commit with
openSearcher=false (Erick correctly points out such a thing doesn't exist)
would be a pointless no-op, nothing visible and nothing written to disk!
On 7 January 2016 at 17:21, Erick Erickson <er...@gmail.com> wrote:
> <1> right
> <2> not complete
>
> bq: I assume that the hard commit clears the Java heap...created by
> soft commits
>
> ONLY if openSeacher=true. Otherwise all that happens is the current
> segments
> are flushed to disk, closed and new ones opened. And just to be clear,
> openSearcher
> is only relevant for hard commits, it is meaningless with soft commits.
>
>
>
> On Wed, Jan 6, 2016 at 11:22 PM, Gili Nachum <gi...@gmail.com> wrote:
> > Thank you guys. My takeaways:
> > 1. Everything visible prior to a soft commit remains visible after soft
> > commit (regardless of <openSearcher>false</openSearcher>)
> > 2. I assume that the hard commit clears the Java heap from the
> RAMDirectory
> > created by soft commits, therefore to keep things visible, it must open
> > Searchers in disk despite <openSearcher>false</openSearcher>, at least
> for
> > the newly added segments.
> >
> > On Mon, Jan 4, 2016 at 4:29 PM, Daniel Collins <da...@gmail.com>
> > wrote:
> >
> >> If you have already done a soft commit and that opened a new searcher,
> then
> >> the document will be visible from that point on. The results returned
> by
> >> that searcher cannot be changed by the hard commit (whatever that is
> doing
> >> under the hood, the segment that has that document in must still be
> visible
> >> to the searcher). I don't know exactly how the soft commit stores its
> >> segment, but there must be some kind of reference counting like there is
> >> for disk segments since the searcher has that "segment" open
> (regardless of
> >> whether that segment is in RAM or on disk).
> >>
> >> On 4 January 2016 at 14:05, Emir Arnautovic <
> emir.arnautovic@sematext.com>
> >> wrote:
> >>
> >> > Hi Gili,
> >> > Visibility is related to searcher - if you reopen searcher it will be
> >> > visible. If hard commit happens without reopening searcher, documents
> >> will
> >> > not be visible till next soft commit happens.
> >> > You can find more details about commits on
> >> >
> >>
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> >> >
> >> > HTH,
> >> > Emir
> >> >
> >> >
> >> > On 04.01.2016 11:14, Gili Nachum wrote:
> >> >
> >> >> Hello,
> >> >>
> >> >> When a new document is added, it becomes visible after a soft commit,
> >> >> during which it is written to a Lucene RAMDirectory (in heap). Then
> >> after
> >> >> a
> >> >> hard commit, the RAMDirectory is removed from memory and the docs are
> >> >> written to the index on disk.
> >> >> What happens if I hard commit (write to disk) with
> openSearcher=false.
> >> >> Would I lose document visibility? since it's no longer in memory AND
> the
> >> >> hard commit didn't open a new searcher on disk?
> >> >>
> >> >> Does soft commit also re-opens Searchers over the index on disk?
> >> >>
> >> >> Here's my commit configuration:
> >> >>
> >> >> <autoCommit>
> >> >> <maxTime>600000</maxTime>
> >> >> * <openSearcher>false</openSearcher>*
> >> >> </autoCommit>
> >> >> <autoSoftCommit>
> >> >> <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
> >> >> </autoSoftCommit>
> >> >>
> >> >> Thanks.
> >> >>
> >> >>
> >> > --
> >> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> > Solr & Elasticsearch Support * http://sematext.com/
> >> >
> >> >
> >>
>
Re: Does soft commit re-opens searchers in disk?
Posted by Erick Erickson <er...@gmail.com>.
<1> right
<2> not complete
bq: I assume that the hard commit clears the Java heap...created by
soft commits
ONLY if openSeacher=true. Otherwise all that happens is the current segments
are flushed to disk, closed and new ones opened. And just to be clear,
openSearcher
is only relevant for hard commits, it is meaningless with soft commits.
On Wed, Jan 6, 2016 at 11:22 PM, Gili Nachum <gi...@gmail.com> wrote:
> Thank you guys. My takeaways:
> 1. Everything visible prior to a soft commit remains visible after soft
> commit (regardless of <openSearcher>false</openSearcher>)
> 2. I assume that the hard commit clears the Java heap from the RAMDirectory
> created by soft commits, therefore to keep things visible, it must open
> Searchers in disk despite <openSearcher>false</openSearcher>, at least for
> the newly added segments.
>
> On Mon, Jan 4, 2016 at 4:29 PM, Daniel Collins <da...@gmail.com>
> wrote:
>
>> If you have already done a soft commit and that opened a new searcher, then
>> the document will be visible from that point on. The results returned by
>> that searcher cannot be changed by the hard commit (whatever that is doing
>> under the hood, the segment that has that document in must still be visible
>> to the searcher). I don't know exactly how the soft commit stores its
>> segment, but there must be some kind of reference counting like there is
>> for disk segments since the searcher has that "segment" open (regardless of
>> whether that segment is in RAM or on disk).
>>
>> On 4 January 2016 at 14:05, Emir Arnautovic <em...@sematext.com>
>> wrote:
>>
>> > Hi Gili,
>> > Visibility is related to searcher - if you reopen searcher it will be
>> > visible. If hard commit happens without reopening searcher, documents
>> will
>> > not be visible till next soft commit happens.
>> > You can find more details about commits on
>> >
>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>> >
>> > HTH,
>> > Emir
>> >
>> >
>> > On 04.01.2016 11:14, Gili Nachum wrote:
>> >
>> >> Hello,
>> >>
>> >> When a new document is added, it becomes visible after a soft commit,
>> >> during which it is written to a Lucene RAMDirectory (in heap). Then
>> after
>> >> a
>> >> hard commit, the RAMDirectory is removed from memory and the docs are
>> >> written to the index on disk.
>> >> What happens if I hard commit (write to disk) with openSearcher=false.
>> >> Would I lose document visibility? since it's no longer in memory AND the
>> >> hard commit didn't open a new searcher on disk?
>> >>
>> >> Does soft commit also re-opens Searchers over the index on disk?
>> >>
>> >> Here's my commit configuration:
>> >>
>> >> <autoCommit>
>> >> <maxTime>600000</maxTime>
>> >> * <openSearcher>false</openSearcher>*
>> >> </autoCommit>
>> >> <autoSoftCommit>
>> >> <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
>> >> </autoSoftCommit>
>> >>
>> >> Thanks.
>> >>
>> >>
>> > --
>> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > Solr & Elasticsearch Support * http://sematext.com/
>> >
>> >
>>
Re: Does soft commit re-opens searchers in disk?
Posted by Gili Nachum <gi...@gmail.com>.
Thank you guys. My takeaways:
1. Everything visible prior to a soft commit remains visible after soft
commit (regardless of <openSearcher>false</openSearcher>)
2. I assume that the hard commit clears the Java heap from the RAMDirectory
created by soft commits, therefore to keep things visible, it must open
Searchers in disk despite <openSearcher>false</openSearcher>, at least for
the newly added segments.
On Mon, Jan 4, 2016 at 4:29 PM, Daniel Collins <da...@gmail.com>
wrote:
> If you have already done a soft commit and that opened a new searcher, then
> the document will be visible from that point on. The results returned by
> that searcher cannot be changed by the hard commit (whatever that is doing
> under the hood, the segment that has that document in must still be visible
> to the searcher). I don't know exactly how the soft commit stores its
> segment, but there must be some kind of reference counting like there is
> for disk segments since the searcher has that "segment" open (regardless of
> whether that segment is in RAM or on disk).
>
> On 4 January 2016 at 14:05, Emir Arnautovic <em...@sematext.com>
> wrote:
>
> > Hi Gili,
> > Visibility is related to searcher - if you reopen searcher it will be
> > visible. If hard commit happens without reopening searcher, documents
> will
> > not be visible till next soft commit happens.
> > You can find more details about commits on
> >
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> >
> > HTH,
> > Emir
> >
> >
> > On 04.01.2016 11:14, Gili Nachum wrote:
> >
> >> Hello,
> >>
> >> When a new document is added, it becomes visible after a soft commit,
> >> during which it is written to a Lucene RAMDirectory (in heap). Then
> after
> >> a
> >> hard commit, the RAMDirectory is removed from memory and the docs are
> >> written to the index on disk.
> >> What happens if I hard commit (write to disk) with openSearcher=false.
> >> Would I lose document visibility? since it's no longer in memory AND the
> >> hard commit didn't open a new searcher on disk?
> >>
> >> Does soft commit also re-opens Searchers over the index on disk?
> >>
> >> Here's my commit configuration:
> >>
> >> <autoCommit>
> >> <maxTime>600000</maxTime>
> >> * <openSearcher>false</openSearcher>*
> >> </autoCommit>
> >> <autoSoftCommit>
> >> <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
> >> </autoSoftCommit>
> >>
> >> Thanks.
> >>
> >>
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
>
Re: Does soft commit re-opens searchers in disk?
Posted by Daniel Collins <da...@gmail.com>.
If you have already done a soft commit and that opened a new searcher, then
the document will be visible from that point on. The results returned by
that searcher cannot be changed by the hard commit (whatever that is doing
under the hood, the segment that has that document in must still be visible
to the searcher). I don't know exactly how the soft commit stores its
segment, but there must be some kind of reference counting like there is
for disk segments since the searcher has that "segment" open (regardless of
whether that segment is in RAM or on disk).
On 4 January 2016 at 14:05, Emir Arnautovic <em...@sematext.com>
wrote:
> Hi Gili,
> Visibility is related to searcher - if you reopen searcher it will be
> visible. If hard commit happens without reopening searcher, documents will
> not be visible till next soft commit happens.
> You can find more details about commits on
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> HTH,
> Emir
>
>
> On 04.01.2016 11:14, Gili Nachum wrote:
>
>> Hello,
>>
>> When a new document is added, it becomes visible after a soft commit,
>> during which it is written to a Lucene RAMDirectory (in heap). Then after
>> a
>> hard commit, the RAMDirectory is removed from memory and the docs are
>> written to the index on disk.
>> What happens if I hard commit (write to disk) with openSearcher=false.
>> Would I lose document visibility? since it's no longer in memory AND the
>> hard commit didn't open a new searcher on disk?
>>
>> Does soft commit also re-opens Searchers over the index on disk?
>>
>> Here's my commit configuration:
>>
>> <autoCommit>
>> <maxTime>600000</maxTime>
>> * <openSearcher>false</openSearcher>*
>> </autoCommit>
>> <autoSoftCommit>
>> <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
>> </autoSoftCommit>
>>
>> Thanks.
>>
>>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
Re: Does soft commit re-opens searchers in disk?
Posted by Emir Arnautovic <em...@sematext.com>.
Hi Gili,
Visibility is related to searcher - if you reopen searcher it will be
visible. If hard commit happens without reopening searcher, documents
will not be visible till next soft commit happens.
You can find more details about commits on
https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
HTH,
Emir
On 04.01.2016 11:14, Gili Nachum wrote:
> Hello,
>
> When a new document is added, it becomes visible after a soft commit,
> during which it is written to a Lucene RAMDirectory (in heap). Then after a
> hard commit, the RAMDirectory is removed from memory and the docs are
> written to the index on disk.
> What happens if I hard commit (write to disk) with openSearcher=false.
> Would I lose document visibility? since it's no longer in memory AND the
> hard commit didn't open a new searcher on disk?
>
> Does soft commit also re-opens Searchers over the index on disk?
>
> Here's my commit configuration:
>
> <autoCommit>
> <maxTime>600000</maxTime>
> * <openSearcher>false</openSearcher>*
> </autoCommit>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:30000}</maxTime>
> </autoSoftCommit>
>
> Thanks.
>
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/