You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Yonghui Zhao <zh...@gmail.com> on 2017/09/28 12:49:41 UTC
TieredMergePolicy disrupts doc id order after merge
Hi,
It is easier to elaborate my question with an example.
My lucene version is 4.10.4
I use
SortField sortField = new SortField(null, SortField.Type.DOC, true);
sort = new Sort(sortField);
return new SortingMergePolicy(new TieredMergePolicy(), sort);
to make sure my index merger will make fresh documents in the beginning of
merged segment.
TieredMergePolicy will chose biggest segments to merge, so the merge order
should be size descending order.
Say we have 2 segments, segment 0 size is smaller than segment 1, after
size sorting merge readers are (segment 1, segment 0), after merge all
docs in segment 1 are in front of segment 0.
This doesn't satisfy my requirement.
If merge readers will be restored to origin order after TieredMergePolicy
findMerges, then this problem will be fixed.
Any problem of this solution?
Re: TieredMergePolicy disrupts doc id order after merge
Posted by Yonghui Zhao <zh...@gmail.com>.
Got it, thanks Uwe!
2017-09-29 1:39 GMT+08:00 Uwe Schindler <uw...@thetaphi.de>:
> Hi,
>
> Use another merge policy, see LogMergePolicy subclasses! Those preserve
> order, but are not merging in ideal ways.
>
> In general: Relying on internal Lucene DocIDs is not guaranteed to work,
> this is only an implementation detail. The internal IDs are also not
> stable!!!
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Yonghui Zhao [mailto:zhaoyonghui@gmail.com]
> > Sent: Thursday, September 28, 2017 2:50 PM
> > To: java-user@lucene.apache.org
> > Subject: TieredMergePolicy disrupts doc id order after merge
> >
> > Hi,
> >
> > It is easier to elaborate my question with an example.
> >
> > My lucene version is 4.10.4
> >
> > I use
> >
> > SortField sortField = new SortField(null, SortField.Type.DOC, true);
> > sort = new Sort(sortField);
> > return new SortingMergePolicy(new TieredMergePolicy(), sort);
> >
> >
> > to make sure my index merger will make fresh documents in the beginning
> of
> > merged segment.
> >
> > TieredMergePolicy will chose biggest segments to merge, so the merge
> order
> > should be size descending order.
> >
> > Say we have 2 segments, segment 0 size is smaller than segment 1, after
> > size sorting merge readers are (segment 1, segment 0), after merge all
> > docs in segment 1 are in front of segment 0.
> >
> > This doesn't satisfy my requirement.
> >
> > If merge readers will be restored to origin order after TieredMergePolicy
> > findMerges, then this problem will be fixed.
> >
> > Any problem of this solution?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: TieredMergePolicy disrupts doc id order after merge
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
Use another merge policy, see LogMergePolicy subclasses! Those preserve order, but are not merging in ideal ways.
In general: Relying on internal Lucene DocIDs is not guaranteed to work, this is only an implementation detail. The internal IDs are also not stable!!!
Uwe
-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Yonghui Zhao [mailto:zhaoyonghui@gmail.com]
> Sent: Thursday, September 28, 2017 2:50 PM
> To: java-user@lucene.apache.org
> Subject: TieredMergePolicy disrupts doc id order after merge
>
> Hi,
>
> It is easier to elaborate my question with an example.
>
> My lucene version is 4.10.4
>
> I use
>
> SortField sortField = new SortField(null, SortField.Type.DOC, true);
> sort = new Sort(sortField);
> return new SortingMergePolicy(new TieredMergePolicy(), sort);
>
>
> to make sure my index merger will make fresh documents in the beginning of
> merged segment.
>
> TieredMergePolicy will chose biggest segments to merge, so the merge order
> should be size descending order.
>
> Say we have 2 segments, segment 0 size is smaller than segment 1, after
> size sorting merge readers are (segment 1, segment 0), after merge all
> docs in segment 1 are in front of segment 0.
>
> This doesn't satisfy my requirement.
>
> If merge readers will be restored to origin order after TieredMergePolicy
> findMerges, then this problem will be fixed.
>
> Any problem of this solution?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org