You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2009/07/31 00:29:36 UTC
SegmentReader field cache merging?
I know this has been somewhat stuck in LUCENE-831 which seems to
have blown up quite a bit over time and is untouched of late?
Perhaps in separate patch that's limited to field cache merging
we can simply modify our existing field cache code (i.e. not
rewrite field caching in general) (in conjunction with
IW.getReader and segment merging) to automatically (or with a
settings callback in IW for which fields should be auto merged)?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SegmentReader field cache merging?
Posted by Michael McCandless <lu...@mikemccandless.com>.
+1
We're already able to warm a reader in the the newly merged segment,
in the BG and not delaying any near real-time reopens in the
meantime... but if we can merge FieldCache entries directly in RAM
this ought to be a good speedup over re-un-inverting the merged
segment.
Mike
On Thu, Jul 30, 2009 at 6:35 PM, Mark Miller<ma...@gmail.com> wrote:
>>> Perhaps in separate patch that's limited to field cache merging
>>> we can simply modify our existing field cache code (i.e. not
>>> rewrite field caching in general) (in conjunction with
>>> IW.getReader and segment merging) to automatically (or with a
>>> settings callback in IW for which fields should be auto merged)?
>
> +1 - especially if you able to work on it soon rather than later. 831 won't
> be a reality for some time would be my prediction.
> It doesn't seem like you would need it to support merging FieldCaches
> either. You would just prepopulate the new merged segment with values from
> the previous segments and then those would get garbage collected when no
> longer referenced right?
>
> Jason Rutherglen wrote:
>>
>> I know this has been somewhat stuck in LUCENE-831 which seems to
>> have blown up quite a bit over time and is untouched of late?
>>
>> Perhaps in separate patch that's limited to field cache merging
>> we can simply modify our existing field cache code (i.e. not
>> rewrite field caching in general) (in conjunction with
>> IW.getReader and segment merging) to automatically (or with a
>> settings callback in IW for which fields should be auto merged)?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: SegmentReader field cache merging?
Posted by Mark Miller <ma...@gmail.com>.
>>Perhaps in separate patch that's limited to field cache merging
>>we can simply modify our existing field cache code (i.e. not
>>rewrite field caching in general) (in conjunction with
>>IW.getReader and segment merging) to automatically (or with a
>>settings callback in IW for which fields should be auto merged)?
+1 - especially if you able to work on it soon rather than later. 831
won't be a reality for some time would be my prediction.
It doesn't seem like you would need it to support merging FieldCaches
either. You would just prepopulate the new merged segment with values
from the previous segments and then those would get garbage collected
when no longer referenced right?
Jason Rutherglen wrote:
> I know this has been somewhat stuck in LUCENE-831 which seems to
> have blown up quite a bit over time and is untouched of late?
>
> Perhaps in separate patch that's limited to field cache merging
> we can simply modify our existing field cache code (i.e. not
> rewrite field caching in general) (in conjunction with
> IW.getReader and segment merging) to automatically (or with a
> settings callback in IW for which fields should be auto merged)?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
--
- Mark
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org