You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Cam Bazz <ca...@gmail.com> on 2008/09/15 19:49:48 UTC
IndexWriter commit
Hello,
What is the difference between flush in <2.4 and commit?
Also I have been looking over docs, and they mention commit(long) but
there is no commit(long) method but only commit()
Best.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter commit
Posted by Cam Bazz <ca...@gmail.com>.
Hello Dipen,
I think what he meant is that if power is off the last transaction is
trashed, but your index is not.
Best.
On Mon, Sep 15, 2008 at 10:55 PM, Dipen <di...@gmail.com> wrote:
> hi michael,
> this is rather hard for me to understand, if a system loses power
> (electricity), how can it be ensured that fsync() call will happen at all,
> this commit function relies on fsync() but what if OS doesnt have time or
> power in this case to actually call fsync() and synchronize. I read about
> fsync() long time bacck and had this same question.
>
> Is there anything like ultimate transaction preservation procedure as a
> state of no power is almost unpedictable and unplannable. This is rather
> silly question I know, but just coz you mentioned and I happened to read, I
> wouldn loose this opportunity to ask this :)
>
> thanks
>
> On Tue, Sep 16, 2008 at 12:18 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> There is no difference, unless your computer/OS crashes or loses power
>> shortly after you had call the method.
>>
>> In that case, there's a big difference: commit() guarantees your index will
>> be intact (assuming the storage system holding your index was not damaged)
>> but with flush(), which doesn't sync() the index files it writes, it's
>> entirely possible that the index will become corrupt because some files were
>> still in the OS's write cache when it crashed.
>>
>> But the guarantee only holds if the underlying storage system is "honest"
>> about fsync(), ie, it truly flushes all written bytes for that file to disk
>> before returning.
>>
>> Mike
>>
>>
>> Cam Bazz wrote:
>>
>> Hello,
>>>
>>> What is the difference between flush in <2.4 and commit?
>>>
>>> Also I have been looking over docs, and they mention commit(long) but
>>> there is no commit(long) method but only commit()
>>>
>>> Best.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter commit
Posted by Michael McCandless <lu...@mikemccandless.com>.
It's only if power is lost *after* the call to IndexWriter.commit()
has successfully returned, that the guarantee holds.
commit() does not return until all newly written and referenced files
in the index have been successfully fsync'd (and the OS does not
return from fsync until all bytes are on stable storage).
If you lose power while commit() is still running, then the only
guarantee is that the index will be consistent as of the last time you
successfully ran commit (or, IndexWriter.close, which also calls
commit).
In any event, as of 2.4, a power loss (or OS crash) should not corrupt
your index (unlike pre-2.4), but, you will lose all changes since the
last successful commit.
Mike
Dipen wrote:
> hi michael,
> this is rather hard for me to understand, if a system loses power
> (electricity), how can it be ensured that fsync() call will happen
> at all,
> this commit function relies on fsync() but what if OS doesnt have
> time or
> power in this case to actually call fsync() and synchronize. I read
> about
> fsync() long time bacck and had this same question.
>
> Is there anything like ultimate transaction preservation procedure
> as a
> state of no power is almost unpedictable and unplannable. This is
> rather
> silly question I know, but just coz you mentioned and I happened to
> read, I
> wouldn loose this opportunity to ask this :)
>
> thanks
>
> On Tue, Sep 16, 2008 at 12:18 AM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>>
>> There is no difference, unless your computer/OS crashes or loses
>> power
>> shortly after you had call the method.
>>
>> In that case, there's a big difference: commit() guarantees your
>> index will
>> be intact (assuming the storage system holding your index was not
>> damaged)
>> but with flush(), which doesn't sync() the index files it writes,
>> it's
>> entirely possible that the index will become corrupt because some
>> files were
>> still in the OS's write cache when it crashed.
>>
>> But the guarantee only holds if the underlying storage system is
>> "honest"
>> about fsync(), ie, it truly flushes all written bytes for that file
>> to disk
>> before returning.
>>
>> Mike
>>
>>
>> Cam Bazz wrote:
>>
>> Hello,
>>>
>>> What is the difference between flush in <2.4 and commit?
>>>
>>> Also I have been looking over docs, and they mention commit(long)
>>> but
>>> there is no commit(long) method but only commit()
>>>
>>> Best.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter commit
Posted by Dipen <di...@gmail.com>.
hi michael,
this is rather hard for me to understand, if a system loses power
(electricity), how can it be ensured that fsync() call will happen at all,
this commit function relies on fsync() but what if OS doesnt have time or
power in this case to actually call fsync() and synchronize. I read about
fsync() long time bacck and had this same question.
Is there anything like ultimate transaction preservation procedure as a
state of no power is almost unpedictable and unplannable. This is rather
silly question I know, but just coz you mentioned and I happened to read, I
wouldn loose this opportunity to ask this :)
thanks
On Tue, Sep 16, 2008 at 12:18 AM, Michael McCandless <
lucene@mikemccandless.com> wrote:
>
> There is no difference, unless your computer/OS crashes or loses power
> shortly after you had call the method.
>
> In that case, there's a big difference: commit() guarantees your index will
> be intact (assuming the storage system holding your index was not damaged)
> but with flush(), which doesn't sync() the index files it writes, it's
> entirely possible that the index will become corrupt because some files were
> still in the OS's write cache when it crashed.
>
> But the guarantee only holds if the underlying storage system is "honest"
> about fsync(), ie, it truly flushes all written bytes for that file to disk
> before returning.
>
> Mike
>
>
> Cam Bazz wrote:
>
> Hello,
>>
>> What is the difference between flush in <2.4 and commit?
>>
>> Also I have been looking over docs, and they mention commit(long) but
>> there is no commit(long) method but only commit()
>>
>> Best.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: IndexWriter commit
Posted by Michael McCandless <lu...@mikemccandless.com>.
There is no difference, unless your computer/OS crashes or loses power
shortly after you had call the method.
In that case, there's a big difference: commit() guarantees your index
will be intact (assuming the storage system holding your index was not
damaged) but with flush(), which doesn't sync() the index files it
writes, it's entirely possible that the index will become corrupt
because some files were still in the OS's write cache when it crashed.
But the guarantee only holds if the underlying storage system is
"honest" about fsync(), ie, it truly flushes all written bytes for
that file to disk before returning.
Mike
Cam Bazz wrote:
> Hello,
>
> What is the difference between flush in <2.4 and commit?
>
> Also I have been looking over docs, and they mention commit(long) but
> there is no commit(long) method but only commit()
>
> Best.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter commit
Posted by Cam Bazz <ca...@gmail.com>.
Hello,
Thanks a bunch Michael. Its been a long time I wanted to upgrade to
2.4. It seems major change has been done.
Best.
On Mon, Sep 15, 2008 at 9:49 PM, Michael McCandless
<lu...@mikemccandless.com> wrote:
>
> Oh and I just committed a fix to IndexWriter's javadocs -- commit(long) is a
> private method that should never have been in the javadocs. Thanks for
> raising this!
>
> Mike
>
> Cam Bazz wrote:
>
>> Hello,
>>
>> What is the difference between flush in <2.4 and commit?
>>
>> Also I have been looking over docs, and they mention commit(long) but
>> there is no commit(long) method but only commit()
>>
>> Best.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: IndexWriter commit
Posted by Michael McCandless <lu...@mikemccandless.com>.
Oh and I just committed a fix to IndexWriter's javadocs --
commit(long) is a private method that should never have been in the
javadocs. Thanks for raising this!
Mike
Cam Bazz wrote:
> Hello,
>
> What is the difference between flush in <2.4 and commit?
>
> Also I have been looking over docs, and they mention commit(long) but
> there is no commit(long) method but only commit()
>
> Best.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org