You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Mikhail Khludnev <mk...@griddynamics.com> on 2014/09/23 00:00:57 UTC
How to openIfChanged the most recent merge?
Hello!
I'm in trouble with Lucene Index Writer. I'm benchmarking some algorithm
which might seem like NRT-case, but I'm not sure that I need it
particularly. The overall problem is to writing "join index" (column holds
docnums) via updating binary docvalues after commit. i.e.:
- update docs
- commit
- read docs (openIfChanged() before )
- updateDocVals
- commit
It's clunky but it works, until guess what happens... merge.Oh my.
Once a time I have segments
segments_ec:2090 _7c(5.0):C117/8:delGen=8:....
_7j(5.0):C1:fieldInfosGen=1:dvGen=1 _7k(5.0):C1)
I apply one update and trigger commit, as a result I have:
segments_ee:2102 _7c(5.0):C117/9:delGen=9:..
_7k(5.0):C1:fieldInfosGen=1:dvGen=1 _7l(5.0):C1)
however, somewhere inside of the this commit call, pretty
SerialMergeScheduler bakes the single solid segment
_7m(5.0):C117
however, it wasn't exposed in via any segments file so far.
And now I get into trouble:
if I call DR.openIfChanged(segments_ec) (even after IW.waitMerges()), I've
got segments_ee that's fairly reasonable, to keep it incremental and fast.
but if I use that IndexWriter, it applies new updates on top of that
merged one (_7m(5.0):C117), not on segments_ee. And it broke my assertions.
I rather need to open reader of that merged _7m(5.0):C117, which IW keeps
somewhere internally, and it's better to do if fancy&incremental. If you
can point me on how NRT can solve I'd happy to switch on it.
Incredibly thank you for your time!!!
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: How to openIfChanged the most recent merge?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
It's awesome! Incredibly thank you!
On Sun, Sep 28, 2014 at 12:46 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> OK I ran the test and saw the failure, thank you!
>
> I think I understand why you are seeing what you are seeing.
>
> First off, you are not actually using an NRT reader when
> hardReopenBeforeDVUpdate is false, because in readerReopenIfChanged,
> when oldReader == null, you must do:
>
> return DirectoryReader.open(writer2, true);
>
> so that your initial reader is in fact NRT. All subsequent reopens
> from then on will then be NRT.
>
> When I make that change to your test, it seems to pass (or at least
> run for much longer than it did before...).
>
> However, if I remove the writer.commit() before the reopen, the test
> fails. The reason is that IW commit and NRT reader reopen do not
> reflect merges "just kicked off" by that flush, even when using SMS.
> So, there will always be this "off by 1", in that you'll get a reader
> with 10 segments (pre-merge) not 1 segment (post-merge).
>
> One possible workaround here w/o having to call crazy-expensive commit
> would be to call reopenIfChanged twice in a row (and fix your reopen
> method to properly handle null return from openIfChanged); when I
> tried that in your test, it also seemed to run forever...
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Sep 26, 2014 at 2:44 PM, Mikhail Khludnev
> <mk...@griddynamics.com> wrote:
> >
> >
> > On Fri, Sep 26, 2014 at 7:07 PM, Michael McCandless
> > <lu...@mikemccandless.com> wrote:
> >>
> >> Sorry I can't make heads or tails of what you are saying here ... can
> >> you maybe make a small test case that fails with "ant test"? Boil it
> >> down as much as possible...
> >
> >
> > Sure. I'm really sorry for being so confusing.
> > I changed constant
> >
> https://github.com/m-khl/lucene-merge-visibility/commit/a4a01c2c91d9c30850602b8dddf23de5363c4851#diff-86ebfbf440fe69ee36a52705cb92b824R44
> > to make it fail.
> > the branch reader-vs-merge at
> > https://github.com/m-khl/lucene-merge-visibility/tree/reader-vs-merge
> > in lucene/core there is a failed test
> > $> ant test -Dtestcase=TestNumDValUpdVsReaderVisibility
> >
> > it's verbose, because it uses sysout as infostream.
> > [junit4] FAILURE 2.40s | TestNumDValUpdVsReaderVisibility.testSimple
> <<<
> > [junit4] > Throwable #1: java.lang.AssertionError: failed on
> id:doc-18
> > expected:<17> but was:<18>
> > [junit4] > at
> > __randomizedtesting.SeedInfo.seed([73A18231908F4ADC:4B12A6CFB77C9E0D]:0)
> > [junit4] > at
> >
> org.apache.lucene.index.TestNumDValUpdVsReaderVisibility.testSimple(TestNumDValUpdVsReaderVisibility.java:134)
> >
> >
> >>
> >>
> >> The gist seems to be if you use an NRT reader something fails, but if
> >> you instead open a new reader, that something passes?
> >
> > I don't use NTR, and perhaps it's a solution. I just don't know how to do
> > that.
> > Note: closing writer, open reader - works (but I suppose it's slow); just
> > committing and reopening reader - it fails;
> >>
> >> But what
> >> exactly is failing?
> >
> > - let I have merge factor 10 and SerialMergeSceduler.
> > - I did 9 commits already and have 9 segments in the index
> > - I add a few docs and commit
> > - 10th commit triggers merge synchronously, it's done.
> > - now if I reopen reader it see 10 unmerged segments (merged single
> segment
> > index, isn't visible for reopen) /*test FAILS*/
> > - but if I fully close writer&reader and open reader, I've got single
> > segment merged index. /*test PASS */
> >
> > - usually such behavior gets no probs, it's reasonable, and fine.
> > - but I do a mad thing
> > - I use that reader (with 10 segments) to get docnum and write it as a
> > docvalue;
> > - after I commit only docvalues update (no docs update) and reopen
> reader,
> > I've got single segment index, which was already written by merge at the
> > previous commit.
> > - and here is a problem because a docnum obtained at 10 segments index,
> > doesn't match to docnum at single segment index (there was a deletion)
> >
> >>
> >> And what is a "solid" segment here?
> >
> > I meant an index contains of single segment, at contrast from index
> contains
> > of many ones.
> >
> > Thank you!
> >>
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >>
> >> On Thu, Sep 25, 2014 at 6:00 PM, Mikhail Khludnev
> >> <mk...@griddynamics.com> wrote:
> >> > Hello Mike!
> >> >
> >> > Thanks for your attention.
> >> > I pushed the mad case at
> >> >
> >> >
> https://github.com/m-khl/lucene-merge-visibility/commit/fa2d60be5b13eb57e0527c843119cf62cfa83a7d#diff-86ebfbf440fe69ee36a52705cb92b824R120
> >> >
> >> > it does the following
> >> >
> >> > - writes a pair of doc
> >> > - commit
> >> > - reopen reader, searches for one of them
> >> > - update this doc with its' docnum (I know it's weird, but I should
> work
> >> > if
> >> > reopened reader sees that update)
> >> > - commit this DV update
> >> > - search that doc and check the written doc val.
> >> > it passes if hardReopenBeforeDVUpdate=true and fails otherwise
> >> >
> >> > I know that changing docnum is natural, but I expect it doesnt change
> >> > while
> >> > I update docvals.
> >> > here how it flips:
> >> > at the commit after doc update we have many segments
> >> >
> >> > now checkpoint "_0(6.0.0):C2/1:delGen=1:fieldInfosGen=1:dvGen=1
> >> > _1(6.0.0):C2:fieldInfosGen=1:dvGen=1 _2(6.0.0):C2:
> >> > commit: wrote segments file "segments_j"
> >> >
> >> > but also there is a solid segment, which is merged but haven't
> >> > committed/published
> >> > after commitMerge: _a(6.0.0):c19
> >> >
> >> > and after DV update commit we have that solid segment visible
> >> >
> >> > now checkpoint "_a(6.0.0):c19:fieldInfosGen=1:dvGen=1" [1 segments ;
> >> > isCommit = true]
> >> > IFD 0 [Thu Sep 25 23:56:22 SAST 2014;
> >> >
> >> >
> TEST-TestNumDValUpdVsReaderVisibility.testSimple-seed#[6131CF35B3A45FC3]]:
> >> > deleteCommits: now decRef commit "segments_j"
> >> > ...
> >> > wrote segments file "segments_k"
> >> >
> >> > I'm using SerialMergeScheduler, and expect to see single solid segment
> >> > after
> >> > I commit document updates and it triggers the merge.
> >> > How I can reopen reader which sees it?
> >> > Thanks
> >> >
> >> >
> >> > On Wed, Sep 24, 2014 at 10:07 PM, Michael McCandless
> >> > <lu...@mikemccandless.com> wrote:
> >> >>
> >> >> I don't understand what's actually happening / going wrong here.
> >> >>
> >> >> Maybe you can make a test case / give more details?
> >> >>
> >> >> What assertions are broken? Why is it bad if SMS does a merge before
> >> >> you reopen? Why are you using SMS :)
> >> >>
> >> >> Mike McCandless
> >> >>
> >> >> http://blog.mikemccandless.com
> >> >>
> >> >> On Mon, Sep 22, 2014 at 6:00 PM, Mikhail Khludnev
> >> >> <mk...@griddynamics.com> wrote:
> >> >> > Hello!
> >> >> > I'm in trouble with Lucene Index Writer. I'm benchmarking some
> >> >> > algorithm
> >> >> > which might seem like NRT-case, but I'm not sure that I need it
> >> >> > particularly. The overall problem is to writing "join index"
> (column
> >> >> > holds
> >> >> > docnums) via updating binary docvalues after commit. i.e.:
> >> >> > - update docs
> >> >> > - commit
> >> >> > - read docs (openIfChanged() before )
> >> >> > - updateDocVals
> >> >> > - commit
> >> >> >
> >> >> > It's clunky but it works, until guess what happens... merge.Oh my.
> >> >> >
> >> >> > Once a time I have segments
> >> >> > segments_ec:2090 _7c(5.0):C117/8:delGen=8:....
> >> >> > _7j(5.0):C1:fieldInfosGen=1:dvGen=1 _7k(5.0):C1)
> >> >> >
> >> >> > I apply one update and trigger commit, as a result I have:
> >> >> > segments_ee:2102 _7c(5.0):C117/9:delGen=9:..
> >> >> > _7k(5.0):C1:fieldInfosGen=1:dvGen=1 _7l(5.0):C1)
> >> >> >
> >> >> > however, somewhere inside of the this commit call, pretty
> >> >> > SerialMergeScheduler bakes the single solid segment
> >> >> > _7m(5.0):C117
> >> >> > however, it wasn't exposed in via any segments file so far.
> >> >> >
> >> >> > And now I get into trouble:
> >> >> > if I call DR.openIfChanged(segments_ec) (even after
> IW.waitMerges()),
> >> >> > I've
> >> >> > got segments_ee that's fairly reasonable, to keep it incremental
> and
> >> >> > fast.
> >> >> > but if I use that IndexWriter, it applies new updates on top of
> that
> >> >> > merged
> >> >> > one (_7m(5.0):C117), not on segments_ee. And it broke my
> assertions.
> >> >> > I
> >> >> > rather need to open reader of that merged _7m(5.0):C117, which IW
> >> >> > keeps
> >> >> > somewhere internally, and it's better to do if fancy&incremental.
> If
> >> >> > you
> >> >> > can
> >> >> > point me on how NRT can solve I'd happy to switch on it.
> >> >> >
> >> >> > Incredibly thank you for your time!!!
> >> >> >
> >> >> > --
> >> >> > Sincerely yours
> >> >> > Mikhail Khludnev
> >> >> > Principal Engineer,
> >> >> > Grid Dynamics
> >> >> >
> >> >> >
> >> >>
> >> >> ---------------------------------------------------------------------
> >> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Sincerely yours
> >> > Mikhail Khludnev
> >> > Principal Engineer,
> >> > Grid Dynamics
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: How to openIfChanged the most recent merge?
Posted by Michael McCandless <lu...@mikemccandless.com>.
OK I ran the test and saw the failure, thank you!
I think I understand why you are seeing what you are seeing.
First off, you are not actually using an NRT reader when
hardReopenBeforeDVUpdate is false, because in readerReopenIfChanged,
when oldReader == null, you must do:
return DirectoryReader.open(writer2, true);
so that your initial reader is in fact NRT. All subsequent reopens
from then on will then be NRT.
When I make that change to your test, it seems to pass (or at least
run for much longer than it did before...).
However, if I remove the writer.commit() before the reopen, the test
fails. The reason is that IW commit and NRT reader reopen do not
reflect merges "just kicked off" by that flush, even when using SMS.
So, there will always be this "off by 1", in that you'll get a reader
with 10 segments (pre-merge) not 1 segment (post-merge).
One possible workaround here w/o having to call crazy-expensive commit
would be to call reopenIfChanged twice in a row (and fix your reopen
method to properly handle null return from openIfChanged); when I
tried that in your test, it also seemed to run forever...
Mike McCandless
http://blog.mikemccandless.com
On Fri, Sep 26, 2014 at 2:44 PM, Mikhail Khludnev
<mk...@griddynamics.com> wrote:
>
>
> On Fri, Sep 26, 2014 at 7:07 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>>
>> Sorry I can't make heads or tails of what you are saying here ... can
>> you maybe make a small test case that fails with "ant test"? Boil it
>> down as much as possible...
>
>
> Sure. I'm really sorry for being so confusing.
> I changed constant
> https://github.com/m-khl/lucene-merge-visibility/commit/a4a01c2c91d9c30850602b8dddf23de5363c4851#diff-86ebfbf440fe69ee36a52705cb92b824R44
> to make it fail.
> the branch reader-vs-merge at
> https://github.com/m-khl/lucene-merge-visibility/tree/reader-vs-merge
> in lucene/core there is a failed test
> $> ant test -Dtestcase=TestNumDValUpdVsReaderVisibility
>
> it's verbose, because it uses sysout as infostream.
> [junit4] FAILURE 2.40s | TestNumDValUpdVsReaderVisibility.testSimple <<<
> [junit4] > Throwable #1: java.lang.AssertionError: failed on id:doc-18
> expected:<17> but was:<18>
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([73A18231908F4ADC:4B12A6CFB77C9E0D]:0)
> [junit4] > at
> org.apache.lucene.index.TestNumDValUpdVsReaderVisibility.testSimple(TestNumDValUpdVsReaderVisibility.java:134)
>
>
>>
>>
>> The gist seems to be if you use an NRT reader something fails, but if
>> you instead open a new reader, that something passes?
>
> I don't use NTR, and perhaps it's a solution. I just don't know how to do
> that.
> Note: closing writer, open reader - works (but I suppose it's slow); just
> committing and reopening reader - it fails;
>>
>> But what
>> exactly is failing?
>
> - let I have merge factor 10 and SerialMergeSceduler.
> - I did 9 commits already and have 9 segments in the index
> - I add a few docs and commit
> - 10th commit triggers merge synchronously, it's done.
> - now if I reopen reader it see 10 unmerged segments (merged single segment
> index, isn't visible for reopen) /*test FAILS*/
> - but if I fully close writer&reader and open reader, I've got single
> segment merged index. /*test PASS */
>
> - usually such behavior gets no probs, it's reasonable, and fine.
> - but I do a mad thing
> - I use that reader (with 10 segments) to get docnum and write it as a
> docvalue;
> - after I commit only docvalues update (no docs update) and reopen reader,
> I've got single segment index, which was already written by merge at the
> previous commit.
> - and here is a problem because a docnum obtained at 10 segments index,
> doesn't match to docnum at single segment index (there was a deletion)
>
>>
>> And what is a "solid" segment here?
>
> I meant an index contains of single segment, at contrast from index contains
> of many ones.
>
> Thank you!
>>
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Thu, Sep 25, 2014 at 6:00 PM, Mikhail Khludnev
>> <mk...@griddynamics.com> wrote:
>> > Hello Mike!
>> >
>> > Thanks for your attention.
>> > I pushed the mad case at
>> >
>> > https://github.com/m-khl/lucene-merge-visibility/commit/fa2d60be5b13eb57e0527c843119cf62cfa83a7d#diff-86ebfbf440fe69ee36a52705cb92b824R120
>> >
>> > it does the following
>> >
>> > - writes a pair of doc
>> > - commit
>> > - reopen reader, searches for one of them
>> > - update this doc with its' docnum (I know it's weird, but I should work
>> > if
>> > reopened reader sees that update)
>> > - commit this DV update
>> > - search that doc and check the written doc val.
>> > it passes if hardReopenBeforeDVUpdate=true and fails otherwise
>> >
>> > I know that changing docnum is natural, but I expect it doesnt change
>> > while
>> > I update docvals.
>> > here how it flips:
>> > at the commit after doc update we have many segments
>> >
>> > now checkpoint "_0(6.0.0):C2/1:delGen=1:fieldInfosGen=1:dvGen=1
>> > _1(6.0.0):C2:fieldInfosGen=1:dvGen=1 _2(6.0.0):C2:
>> > commit: wrote segments file "segments_j"
>> >
>> > but also there is a solid segment, which is merged but haven't
>> > committed/published
>> > after commitMerge: _a(6.0.0):c19
>> >
>> > and after DV update commit we have that solid segment visible
>> >
>> > now checkpoint "_a(6.0.0):c19:fieldInfosGen=1:dvGen=1" [1 segments ;
>> > isCommit = true]
>> > IFD 0 [Thu Sep 25 23:56:22 SAST 2014;
>> >
>> > TEST-TestNumDValUpdVsReaderVisibility.testSimple-seed#[6131CF35B3A45FC3]]:
>> > deleteCommits: now decRef commit "segments_j"
>> > ...
>> > wrote segments file "segments_k"
>> >
>> > I'm using SerialMergeScheduler, and expect to see single solid segment
>> > after
>> > I commit document updates and it triggers the merge.
>> > How I can reopen reader which sees it?
>> > Thanks
>> >
>> >
>> > On Wed, Sep 24, 2014 at 10:07 PM, Michael McCandless
>> > <lu...@mikemccandless.com> wrote:
>> >>
>> >> I don't understand what's actually happening / going wrong here.
>> >>
>> >> Maybe you can make a test case / give more details?
>> >>
>> >> What assertions are broken? Why is it bad if SMS does a merge before
>> >> you reopen? Why are you using SMS :)
>> >>
>> >> Mike McCandless
>> >>
>> >> http://blog.mikemccandless.com
>> >>
>> >> On Mon, Sep 22, 2014 at 6:00 PM, Mikhail Khludnev
>> >> <mk...@griddynamics.com> wrote:
>> >> > Hello!
>> >> > I'm in trouble with Lucene Index Writer. I'm benchmarking some
>> >> > algorithm
>> >> > which might seem like NRT-case, but I'm not sure that I need it
>> >> > particularly. The overall problem is to writing "join index" (column
>> >> > holds
>> >> > docnums) via updating binary docvalues after commit. i.e.:
>> >> > - update docs
>> >> > - commit
>> >> > - read docs (openIfChanged() before )
>> >> > - updateDocVals
>> >> > - commit
>> >> >
>> >> > It's clunky but it works, until guess what happens... merge.Oh my.
>> >> >
>> >> > Once a time I have segments
>> >> > segments_ec:2090 _7c(5.0):C117/8:delGen=8:....
>> >> > _7j(5.0):C1:fieldInfosGen=1:dvGen=1 _7k(5.0):C1)
>> >> >
>> >> > I apply one update and trigger commit, as a result I have:
>> >> > segments_ee:2102 _7c(5.0):C117/9:delGen=9:..
>> >> > _7k(5.0):C1:fieldInfosGen=1:dvGen=1 _7l(5.0):C1)
>> >> >
>> >> > however, somewhere inside of the this commit call, pretty
>> >> > SerialMergeScheduler bakes the single solid segment
>> >> > _7m(5.0):C117
>> >> > however, it wasn't exposed in via any segments file so far.
>> >> >
>> >> > And now I get into trouble:
>> >> > if I call DR.openIfChanged(segments_ec) (even after IW.waitMerges()),
>> >> > I've
>> >> > got segments_ee that's fairly reasonable, to keep it incremental and
>> >> > fast.
>> >> > but if I use that IndexWriter, it applies new updates on top of that
>> >> > merged
>> >> > one (_7m(5.0):C117), not on segments_ee. And it broke my assertions.
>> >> > I
>> >> > rather need to open reader of that merged _7m(5.0):C117, which IW
>> >> > keeps
>> >> > somewhere internally, and it's better to do if fancy&incremental. If
>> >> > you
>> >> > can
>> >> > point me on how NRT can solve I'd happy to switch on it.
>> >> >
>> >> > Incredibly thank you for your time!!!
>> >> >
>> >> > --
>> >> > Sincerely yours
>> >> > Mikhail Khludnev
>> >> > Principal Engineer,
>> >> > Grid Dynamics
>> >> >
>> >> >
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: dev-help@lucene.apache.org
>> >>
>> >
>> >
>> >
>> > --
>> > Sincerely yours
>> > Mikhail Khludnev
>> > Principal Engineer,
>> > Grid Dynamics
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: How to openIfChanged the most recent merge?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
On Fri, Sep 26, 2014 at 7:07 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> Sorry I can't make heads or tails of what you are saying here ... can
> you maybe make a small test case that fails with "ant test"? Boil it
> down as much as possible...
>
Sure. I'm really sorry for being so confusing.
I changed constant
https://github.com/m-khl/lucene-merge-visibility/commit/a4a01c2c91d9c30850602b8dddf23de5363c4851#diff-86ebfbf440fe69ee36a52705cb92b824R44
to make it fail.
the branch *reader-vs-merge
<https://github.com/m-khl/lucene-merge-visibility/tree/reader-vs-merge> *at
https://github.com/m-khl/lucene-merge-visibility/tree/reader-vs-merge
in lucene/core there is a failed test
$> ant test -Dtestcase=TestNumDValUpdVsReaderVisibility
it's verbose, because it uses sysout as infostream.
[junit4] FAILURE 2.40s | TestNumDValUpdVsReaderVisibility.testSimple <<<
[junit4] > Throwable #1: java.lang.AssertionError: failed on
id:doc-18 expected:<17> but was:<18>
[junit4] > at
__randomizedtesting.SeedInfo.seed([73A18231908F4ADC:4B12A6CFB77C9E0D]:0)
[junit4] > at
org.apache.lucene.index.TestNumDValUpdVsReaderVisibility.testSimple(TestNumDValUpdVsReaderVisibility.java:134)
>
> The gist seems to be if you use an NRT reader something fails, but if
> you instead open a new reader, that something passes?
I don't use NTR, and perhaps it's a solution. I just don't know how to do
that.
Note: closing writer, open reader - works (but I suppose it's slow); just
committing and reopening reader - it fails;
> But what
> exactly is failing?
>
- let I have merge factor 10 and SerialMergeSceduler.
- I did 9 commits already and have 9 segments in the index
- I add a few docs and commit
- 10th commit triggers merge synchronously, it's done.
- now if I reopen reader it see 10 unmerged segments (merged single segment
index, isn't visible for reopen) /*test FAILS*/
- but if I fully close writer&reader and open reader, I've got single
segment merged index. /*test PASS */
- usually such behavior gets no probs, it's reasonable, and fine.
- but I do a mad thing
- I use that reader (with 10 segments) to get docnum and write it as a
docvalue;
- after I commit only docvalues update (no docs update) and reopen reader,
I've got single segment index, which was already written by merge at the
previous commit.
- and here is a problem because a docnum obtained at 10 segments index,
doesn't match to docnum at single segment index (there was a deletion)
> And what is a "solid" segment here?
>
I meant an index contains of single segment, at contrast from index
contains of many ones.
Thank you!
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Thu, Sep 25, 2014 at 6:00 PM, Mikhail Khludnev
> <mk...@griddynamics.com> wrote:
> > Hello Mike!
> >
> > Thanks for your attention.
> > I pushed the mad case at
> >
> https://github.com/m-khl/lucene-merge-visibility/commit/fa2d60be5b13eb57e0527c843119cf62cfa83a7d#diff-86ebfbf440fe69ee36a52705cb92b824R120
> >
> > it does the following
> >
> > - writes a pair of doc
> > - commit
> > - reopen reader, searches for one of them
> > - update this doc with its' docnum (I know it's weird, but I should work
> if
> > reopened reader sees that update)
> > - commit this DV update
> > - search that doc and check the written doc val.
> > it passes if hardReopenBeforeDVUpdate=true and fails otherwise
> >
> > I know that changing docnum is natural, but I expect it doesnt change
> while
> > I update docvals.
> > here how it flips:
> > at the commit after doc update we have many segments
> >
> > now checkpoint "_0(6.0.0):C2/1:delGen=1:fieldInfosGen=1:dvGen=1
> > _1(6.0.0):C2:fieldInfosGen=1:dvGen=1 _2(6.0.0):C2:
> > commit: wrote segments file "segments_j"
> >
> > but also there is a solid segment, which is merged but haven't
> > committed/published
> > after commitMerge: _a(6.0.0):c19
> >
> > and after DV update commit we have that solid segment visible
> >
> > now checkpoint "_a(6.0.0):c19:fieldInfosGen=1:dvGen=1" [1 segments ;
> > isCommit = true]
> > IFD 0 [Thu Sep 25 23:56:22 SAST 2014;
> >
> TEST-TestNumDValUpdVsReaderVisibility.testSimple-seed#[6131CF35B3A45FC3]]:
> > deleteCommits: now decRef commit "segments_j"
> > ...
> > wrote segments file "segments_k"
> >
> > I'm using SerialMergeScheduler, and expect to see single solid segment
> after
> > I commit document updates and it triggers the merge.
> > How I can reopen reader which sees it?
> > Thanks
> >
> >
> > On Wed, Sep 24, 2014 at 10:07 PM, Michael McCandless
> > <lu...@mikemccandless.com> wrote:
> >>
> >> I don't understand what's actually happening / going wrong here.
> >>
> >> Maybe you can make a test case / give more details?
> >>
> >> What assertions are broken? Why is it bad if SMS does a merge before
> >> you reopen? Why are you using SMS :)
> >>
> >> Mike McCandless
> >>
> >> http://blog.mikemccandless.com
> >>
> >> On Mon, Sep 22, 2014 at 6:00 PM, Mikhail Khludnev
> >> <mk...@griddynamics.com> wrote:
> >> > Hello!
> >> > I'm in trouble with Lucene Index Writer. I'm benchmarking some
> algorithm
> >> > which might seem like NRT-case, but I'm not sure that I need it
> >> > particularly. The overall problem is to writing "join index" (column
> >> > holds
> >> > docnums) via updating binary docvalues after commit. i.e.:
> >> > - update docs
> >> > - commit
> >> > - read docs (openIfChanged() before )
> >> > - updateDocVals
> >> > - commit
> >> >
> >> > It's clunky but it works, until guess what happens... merge.Oh my.
> >> >
> >> > Once a time I have segments
> >> > segments_ec:2090 _7c(5.0):C117/8:delGen=8:....
> >> > _7j(5.0):C1:fieldInfosGen=1:dvGen=1 _7k(5.0):C1)
> >> >
> >> > I apply one update and trigger commit, as a result I have:
> >> > segments_ee:2102 _7c(5.0):C117/9:delGen=9:..
> >> > _7k(5.0):C1:fieldInfosGen=1:dvGen=1 _7l(5.0):C1)
> >> >
> >> > however, somewhere inside of the this commit call, pretty
> >> > SerialMergeScheduler bakes the single solid segment
> >> > _7m(5.0):C117
> >> > however, it wasn't exposed in via any segments file so far.
> >> >
> >> > And now I get into trouble:
> >> > if I call DR.openIfChanged(segments_ec) (even after IW.waitMerges()),
> >> > I've
> >> > got segments_ee that's fairly reasonable, to keep it incremental and
> >> > fast.
> >> > but if I use that IndexWriter, it applies new updates on top of that
> >> > merged
> >> > one (_7m(5.0):C117), not on segments_ee. And it broke my assertions. I
> >> > rather need to open reader of that merged _7m(5.0):C117, which IW
> keeps
> >> > somewhere internally, and it's better to do if fancy&incremental. If
> you
> >> > can
> >> > point me on how NRT can solve I'd happy to switch on it.
> >> >
> >> > Incredibly thank you for your time!!!
> >> >
> >> > --
> >> > Sincerely yours
> >> > Mikhail Khludnev
> >> > Principal Engineer,
> >> > Grid Dynamics
> >> >
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: dev-help@lucene.apache.org
> >>
> >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: How to openIfChanged the most recent merge?
Posted by Michael McCandless <lu...@mikemccandless.com>.
Sorry I can't make heads or tails of what you are saying here ... can
you maybe make a small test case that fails with "ant test"? Boil it
down as much as possible...
The gist seems to be if you use an NRT reader something fails, but if
you instead open a new reader, that something passes? But what
exactly is failing?
And what is a "solid" segment here?
Mike McCandless
http://blog.mikemccandless.com
On Thu, Sep 25, 2014 at 6:00 PM, Mikhail Khludnev
<mk...@griddynamics.com> wrote:
> Hello Mike!
>
> Thanks for your attention.
> I pushed the mad case at
> https://github.com/m-khl/lucene-merge-visibility/commit/fa2d60be5b13eb57e0527c843119cf62cfa83a7d#diff-86ebfbf440fe69ee36a52705cb92b824R120
>
> it does the following
>
> - writes a pair of doc
> - commit
> - reopen reader, searches for one of them
> - update this doc with its' docnum (I know it's weird, but I should work if
> reopened reader sees that update)
> - commit this DV update
> - search that doc and check the written doc val.
> it passes if hardReopenBeforeDVUpdate=true and fails otherwise
>
> I know that changing docnum is natural, but I expect it doesnt change while
> I update docvals.
> here how it flips:
> at the commit after doc update we have many segments
>
> now checkpoint "_0(6.0.0):C2/1:delGen=1:fieldInfosGen=1:dvGen=1
> _1(6.0.0):C2:fieldInfosGen=1:dvGen=1 _2(6.0.0):C2:
> commit: wrote segments file "segments_j"
>
> but also there is a solid segment, which is merged but haven't
> committed/published
> after commitMerge: _a(6.0.0):c19
>
> and after DV update commit we have that solid segment visible
>
> now checkpoint "_a(6.0.0):c19:fieldInfosGen=1:dvGen=1" [1 segments ;
> isCommit = true]
> IFD 0 [Thu Sep 25 23:56:22 SAST 2014;
> TEST-TestNumDValUpdVsReaderVisibility.testSimple-seed#[6131CF35B3A45FC3]]:
> deleteCommits: now decRef commit "segments_j"
> ...
> wrote segments file "segments_k"
>
> I'm using SerialMergeScheduler, and expect to see single solid segment after
> I commit document updates and it triggers the merge.
> How I can reopen reader which sees it?
> Thanks
>
>
> On Wed, Sep 24, 2014 at 10:07 PM, Michael McCandless
> <lu...@mikemccandless.com> wrote:
>>
>> I don't understand what's actually happening / going wrong here.
>>
>> Maybe you can make a test case / give more details?
>>
>> What assertions are broken? Why is it bad if SMS does a merge before
>> you reopen? Why are you using SMS :)
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Mon, Sep 22, 2014 at 6:00 PM, Mikhail Khludnev
>> <mk...@griddynamics.com> wrote:
>> > Hello!
>> > I'm in trouble with Lucene Index Writer. I'm benchmarking some algorithm
>> > which might seem like NRT-case, but I'm not sure that I need it
>> > particularly. The overall problem is to writing "join index" (column
>> > holds
>> > docnums) via updating binary docvalues after commit. i.e.:
>> > - update docs
>> > - commit
>> > - read docs (openIfChanged() before )
>> > - updateDocVals
>> > - commit
>> >
>> > It's clunky but it works, until guess what happens... merge.Oh my.
>> >
>> > Once a time I have segments
>> > segments_ec:2090 _7c(5.0):C117/8:delGen=8:....
>> > _7j(5.0):C1:fieldInfosGen=1:dvGen=1 _7k(5.0):C1)
>> >
>> > I apply one update and trigger commit, as a result I have:
>> > segments_ee:2102 _7c(5.0):C117/9:delGen=9:..
>> > _7k(5.0):C1:fieldInfosGen=1:dvGen=1 _7l(5.0):C1)
>> >
>> > however, somewhere inside of the this commit call, pretty
>> > SerialMergeScheduler bakes the single solid segment
>> > _7m(5.0):C117
>> > however, it wasn't exposed in via any segments file so far.
>> >
>> > And now I get into trouble:
>> > if I call DR.openIfChanged(segments_ec) (even after IW.waitMerges()),
>> > I've
>> > got segments_ee that's fairly reasonable, to keep it incremental and
>> > fast.
>> > but if I use that IndexWriter, it applies new updates on top of that
>> > merged
>> > one (_7m(5.0):C117), not on segments_ee. And it broke my assertions. I
>> > rather need to open reader of that merged _7m(5.0):C117, which IW keeps
>> > somewhere internally, and it's better to do if fancy&incremental. If you
>> > can
>> > point me on how NRT can solve I'd happy to switch on it.
>> >
>> > Incredibly thank you for your time!!!
>> >
>> > --
>> > Sincerely yours
>> > Mikhail Khludnev
>> > Principal Engineer,
>> > Grid Dynamics
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: How to openIfChanged the most recent merge?
Posted by Mikhail Khludnev <mk...@griddynamics.com>.
Hello Mike!
Thanks for your attention.
I pushed the mad case at
https://github.com/m-khl/lucene-merge-visibility/commit/fa2d60be5b13eb57e0527c843119cf62cfa83a7d#diff-86ebfbf440fe69ee36a52705cb92b824R120
it does the following
- writes a pair of doc
- commit
- reopen reader, searches for one of them
- update this doc with its' docnum (I know it's weird, but I should work if
reopened reader sees that update)
- commit this DV update
- search that doc and check the written doc val.
it passes if hardReopenBeforeDVUpdate=true and fails otherwise
I know that changing docnum is natural, but I expect it doesnt change while
I update docvals.
here how it flips:
at the commit after doc update we have many segments
now checkpoint "_0(6.0.0):C2/1:delGen=1:fieldInfosGen=1:dvGen=1
_1(6.0.0):C2:fieldInfosGen=1:dvGen=1 _2(6.0.0):C2:
commit: wrote segments file "segments_j"
but also there is a solid segment, which is merged but haven't
committed/published
after commitMerge: _a(6.0.0):c19
and after DV update commit we have that solid segment visible
now checkpoint "_a(6.0.0):c19:fieldInfosGen=1:dvGen=1" [1 segments ;
isCommit = true]
IFD 0 [Thu Sep 25 23:56:22 SAST 2014;
TEST-TestNumDValUpdVsReaderVisibility.testSimple-seed#[6131CF35B3A45FC3]]:
deleteCommits: now decRef commit "segments_j"
...
wrote segments file "segments_k"
I'm using SerialMergeScheduler, and expect to see single solid segment
after I commit document updates and it triggers the merge.
How I can reopen reader which sees it?
Thanks
On Wed, Sep 24, 2014 at 10:07 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:
> I don't understand what's actually happening / going wrong here.
>
> Maybe you can make a test case / give more details?
>
> What assertions are broken? Why is it bad if SMS does a merge before
> you reopen? Why are you using SMS :)
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Mon, Sep 22, 2014 at 6:00 PM, Mikhail Khludnev
> <mk...@griddynamics.com> wrote:
> > Hello!
> > I'm in trouble with Lucene Index Writer. I'm benchmarking some algorithm
> > which might seem like NRT-case, but I'm not sure that I need it
> > particularly. The overall problem is to writing "join index" (column
> holds
> > docnums) via updating binary docvalues after commit. i.e.:
> > - update docs
> > - commit
> > - read docs (openIfChanged() before )
> > - updateDocVals
> > - commit
> >
> > It's clunky but it works, until guess what happens... merge.Oh my.
> >
> > Once a time I have segments
> > segments_ec:2090 _7c(5.0):C117/8:delGen=8:....
> > _7j(5.0):C1:fieldInfosGen=1:dvGen=1 _7k(5.0):C1)
> >
> > I apply one update and trigger commit, as a result I have:
> > segments_ee:2102 _7c(5.0):C117/9:delGen=9:..
> > _7k(5.0):C1:fieldInfosGen=1:dvGen=1 _7l(5.0):C1)
> >
> > however, somewhere inside of the this commit call, pretty
> > SerialMergeScheduler bakes the single solid segment
> > _7m(5.0):C117
> > however, it wasn't exposed in via any segments file so far.
> >
> > And now I get into trouble:
> > if I call DR.openIfChanged(segments_ec) (even after IW.waitMerges()),
> I've
> > got segments_ee that's fairly reasonable, to keep it incremental and
> fast.
> > but if I use that IndexWriter, it applies new updates on top of that
> merged
> > one (_7m(5.0):C117), not on segments_ee. And it broke my assertions. I
> > rather need to open reader of that merged _7m(5.0):C117, which IW keeps
> > somewhere internally, and it's better to do if fancy&incremental. If you
> can
> > point me on how NRT can solve I'd happy to switch on it.
> >
> > Incredibly thank you for your time!!!
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics
<http://www.griddynamics.com>
<mk...@griddynamics.com>
Re: How to openIfChanged the most recent merge?
Posted by Michael McCandless <lu...@mikemccandless.com>.
I don't understand what's actually happening / going wrong here.
Maybe you can make a test case / give more details?
What assertions are broken? Why is it bad if SMS does a merge before
you reopen? Why are you using SMS :)
Mike McCandless
http://blog.mikemccandless.com
On Mon, Sep 22, 2014 at 6:00 PM, Mikhail Khludnev
<mk...@griddynamics.com> wrote:
> Hello!
> I'm in trouble with Lucene Index Writer. I'm benchmarking some algorithm
> which might seem like NRT-case, but I'm not sure that I need it
> particularly. The overall problem is to writing "join index" (column holds
> docnums) via updating binary docvalues after commit. i.e.:
> - update docs
> - commit
> - read docs (openIfChanged() before )
> - updateDocVals
> - commit
>
> It's clunky but it works, until guess what happens... merge.Oh my.
>
> Once a time I have segments
> segments_ec:2090 _7c(5.0):C117/8:delGen=8:....
> _7j(5.0):C1:fieldInfosGen=1:dvGen=1 _7k(5.0):C1)
>
> I apply one update and trigger commit, as a result I have:
> segments_ee:2102 _7c(5.0):C117/9:delGen=9:..
> _7k(5.0):C1:fieldInfosGen=1:dvGen=1 _7l(5.0):C1)
>
> however, somewhere inside of the this commit call, pretty
> SerialMergeScheduler bakes the single solid segment
> _7m(5.0):C117
> however, it wasn't exposed in via any segments file so far.
>
> And now I get into trouble:
> if I call DR.openIfChanged(segments_ec) (even after IW.waitMerges()), I've
> got segments_ee that's fairly reasonable, to keep it incremental and fast.
> but if I use that IndexWriter, it applies new updates on top of that merged
> one (_7m(5.0):C117), not on segments_ee. And it broke my assertions. I
> rather need to open reader of that merged _7m(5.0):C117, which IW keeps
> somewhere internally, and it's better to do if fancy&incremental. If you can
> point me on how NRT can solve I'd happy to switch on it.
>
> Incredibly thank you for your time!!!
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org