You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2014/12/19 11:01:14 UTC
[jira] [Commented] (LUCENE-6119) Add
AdaptiveRateLimitedDirectoryWrapper
[ https://issues.apache.org/jira/browse/LUCENE-6119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14253197#comment-14253197 ]
Michael McCandless commented on LUCENE-6119:
--------------------------------------------
I ran some tests with this approach and I think it's no good.
This creates a tricky feedback system, where both CMS (via hard stalling of incoming threads) and this directory attempt to make change to let merges catch up. When CMS's hard stalls kick in, this lowers the indexing byte/sec rate, which causes this directory to (over simplistically) lower the merge IO throttling, which causes the merges to take longer.
I think it's better if all throttling efforts happen in one place, e.g. CMS. I'l think about it ...
> Add AdaptiveRateLimitedDirectoryWrapper
> ---------------------------------------
>
> Key: LUCENE-6119
> URL: https://issues.apache.org/jira/browse/LUCENE-6119
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: 5.0, Trunk
>
> Attachments: LUCENE-6119.patch, LUCENE-6119.patch
>
>
> This method returns number of "incoming" bytes IW has written since it
> was opened, excluding merging.
> It tracks flushed segments, new commits (segments_N), incoming
> files/segments by addIndexes, newly written live docs / doc values
> updates files.
> It's an easy statistic for IW to track and should be useful to help
> applications more intelligently set defaults for IO throttling
> (RateLimiter).
> For example, an application that does hardly any indexing but finally
> triggered a large merge can afford to heavily throttle that large
> merge so it won't interfere with ongoing searches.
> But an application that's causing IW to write new bytes at 50 MB/sec
> must set a correspondingly higher IO throttling otherwise merges will
> clearly fall behind.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org