You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Jason Rutherglen <ja...@gmail.com> on 2009/07/18 16:37:15 UTC
Throttling merges
It may be useful to allow users to throttle merges. A callback
that IW passes into SegmentMerger would suffice where individual
SM methods make use of the callback. I suppose this could slow
down overall merging by adding a potentially useless method
call. However if merging typically consumes IO resources for an
extended period of time, this offers a way for the user to tune
IO consumption and at preferred times free up IO for other
tasks.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Throttling merges
Posted by Jason Rutherglen <ja...@gmail.com>.
Yep, pretty much like "ionice" for Linux.
At the Directory level makes the most sense with the context passed in.
> then somehow add throttling in there.
Yeah I'm not sure how exactly that will work yet.
On Sat, Jul 18, 2009 at 9:12 AM, Michael
McCandless<lu...@mikemccandless.com> wrote:
> The goal is to be like "ionice" right? Meaning, lower the priority of
> IO caused by merging? I agree that makes sense.
>
> I wonder if we could implement it in the Directory level, so that when
> openInput/createOutput is called we can optionally specify the
> "context" (reader, merging, writer, etc.), and then somehow add
> throttling in there.
>
> Mike
>
> On Sat, Jul 18, 2009 at 10:37 AM, Jason
> Rutherglen<ja...@gmail.com> wrote:
>> It may be useful to allow users to throttle merges. A callback
>> that IW passes into SegmentMerger would suffice where individual
>> SM methods make use of the callback. I suppose this could slow
>> down overall merging by adding a potentially useless method
>> call. However if merging typically consumes IO resources for an
>> extended period of time, this offers a way for the user to tune
>> IO consumption and at preferred times free up IO for other
>> tasks.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Throttling merges
Posted by Michael McCandless <lu...@mikemccandless.com>.
The goal is to be like "ionice" right? Meaning, lower the priority of
IO caused by merging? I agree that makes sense.
I wonder if we could implement it in the Directory level, so that when
openInput/createOutput is called we can optionally specify the
"context" (reader, merging, writer, etc.), and then somehow add
throttling in there.
Mike
On Sat, Jul 18, 2009 at 10:37 AM, Jason
Rutherglen<ja...@gmail.com> wrote:
> It may be useful to allow users to throttle merges. A callback
> that IW passes into SegmentMerger would suffice where individual
> SM methods make use of the callback. I suppose this could slow
> down overall merging by adding a potentially useless method
> call. However if merging typically consumes IO resources for an
> extended period of time, this offers a way for the user to tune
> IO consumption and at preferred times free up IO for other
> tasks.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org