You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Earwin Burrfoot <ea...@gmail.com> on 2009/05/17 14:37:12 UTC
Re(opening) (Multi)SegmentReaders
While experimenting with indexReader 'components', I've got this thought:
What if we always create MultiSegmentReader when (re)opening an index,
even if index contains a single segment?
Using unwrapped SegmentReader for single-segment case was a valid
optimization for the times when Lucene did collection over a single
reader, but now it goes down to SegmentReaders anyway.
If we do that, (re)open logic will become a lot more transparent. (And
consequently, hooking into it)
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re(opening) (Multi)SegmentReaders
Posted by Earwin Burrfoot <ea...@gmail.com>.
Okay, I've got some more time on my hands.
While fixing the tests, I found a reopen() bug on trunk, which was
previously hidden from tests by SR-as-toplevel-reader optimization.
(LUCENE-1645)
Can post the first experimental patch tomorrow, with this test failing.
On Mon, May 18, 2009 at 16:06, Michael McCandless
<lu...@mikemccandless.com> wrote:
> On Mon, May 18, 2009 at 7:56 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
>> Will post one soon.
>>
>> Had to slightly modify a whole bunch of tests relying on
>> IndexReader.open(dir) returning SegmentReader instance for
>> single-segment indexes.
>> Currently struggling with Clone/CloneNorms/Reopen tests that actually
>> verify old behaviour.
>>
>> The fun part is - SR no longer has to pose as toplevel
>> directory-owning reader, so most of stuff moved from DIR to MSR and
>> now DIR is almost empty, save for static factory methods and a pair of
>> flags.
>> I'm going to remove DIR completely and make MSR/SR inherit directly from IR.
>
> Sounds good! Nice simplifications...
>
>> All affected classes/methods are package-private, so I bet that
>> doesn't hamper backwards compatibility?
>
> Run ant test-tag to see if the machine agrees with you?
>
> Likely tests will fail, eg at least those verifying old behavior, but
> if it's because these tests were relying on package-private APIs or
> old behavior, you should fix the tests (on the back-compat branch) and
> include that in your patch.
>
> Yonik is there anything in Solr that might not like this change?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re(opening) (Multi)SegmentReaders
Posted by Yonik Seeley <ys...@gmail.com>.
On Mon, May 18, 2009 at 8:06 AM, Michael McCandless
<lu...@mikemccandless.com> wrote:
> Yonik is there anything in Solr that might not like this change?
Yep, there is :-) Should be very easy to work around though.
-Yonik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re(opening) (Multi)SegmentReaders
Posted by Michael McCandless <lu...@mikemccandless.com>.
On Mon, May 18, 2009 at 7:56 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> Will post one soon.
>
> Had to slightly modify a whole bunch of tests relying on
> IndexReader.open(dir) returning SegmentReader instance for
> single-segment indexes.
> Currently struggling with Clone/CloneNorms/Reopen tests that actually
> verify old behaviour.
>
> The fun part is - SR no longer has to pose as toplevel
> directory-owning reader, so most of stuff moved from DIR to MSR and
> now DIR is almost empty, save for static factory methods and a pair of
> flags.
> I'm going to remove DIR completely and make MSR/SR inherit directly from IR.
Sounds good! Nice simplifications...
> All affected classes/methods are package-private, so I bet that
> doesn't hamper backwards compatibility?
Run ant test-tag to see if the machine agrees with you?
Likely tests will fail, eg at least those verifying old behavior, but
if it's because these tests were relying on package-private APIs or
old behavior, you should fix the tests (on the back-compat branch) and
include that in your patch.
Yonik is there anything in Solr that might not like this change?
Mike
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re(opening) (Multi)SegmentReaders
Posted by Earwin Burrfoot <ea...@gmail.com>.
Will post one soon.
Had to slightly modify a whole bunch of tests relying on
IndexReader.open(dir) returning SegmentReader instance for
single-segment indexes.
Currently struggling with Clone/CloneNorms/Reopen tests that actually
verify old behaviour.
The fun part is - SR no longer has to pose as toplevel
directory-owning reader, so most of stuff moved from DIR to MSR and
now DIR is almost empty, save for static factory methods and a pair of
flags.
I'm going to remove DIR completely and make MSR/SR inherit directly from IR.
All affected classes/methods are package-private, so I bet that
doesn't hamper backwards compatibility?
2009/5/17 Michael McCandless <lu...@mikemccandless.com>:
> I tentatively think that's a good idea. The reopen logic is quite hairy...
>
> Wanna make a separate patch for that?
>
> Mike
>
> On Sun, May 17, 2009 at 8:37 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
>> While experimenting with indexReader 'components', I've got this thought:
>>
>> What if we always create MultiSegmentReader when (re)opening an index,
>> even if index contains a single segment?
>> Using unwrapped SegmentReader for single-segment case was a valid
>> optimization for the times when Lucene did collection over a single
>> reader, but now it goes down to SegmentReaders anyway.
>>
>> If we do that, (re)open logic will become a lot more transparent. (And
>> consequently, hooking into it)
>>
>> --
>> Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
>> Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
>> ICQ: 104465785
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re(opening) (Multi)SegmentReaders
Posted by Michael McCandless <lu...@mikemccandless.com>.
I tentatively think that's a good idea. The reopen logic is quite hairy...
Wanna make a separate patch for that?
Mike
On Sun, May 17, 2009 at 8:37 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> While experimenting with indexReader 'components', I've got this thought:
>
> What if we always create MultiSegmentReader when (re)opening an index,
> even if index contains a single segment?
> Using unwrapped SegmentReader for single-segment case was a valid
> optimization for the times when Lucene did collection over a single
> reader, but now it goes down to SegmentReaders anyway.
>
> If we do that, (re)open logic will become a lot more transparent. (And
> consequently, hooking into it)
>
> --
> Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
> Home / Mobile: +7 (495) 683-567-4 / +7 (903) 5-888-423
> ICQ: 104465785
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org