You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Michael McCandless (JIRA)" <ji...@apache.org> on 2011/06/14 19:23:47 UTC
[jira] [Created] (LUCENE-3203) Rate-limit IO used by merging
Rate-limit IO used by merging
-----------------------------
Key: LUCENE-3203
URL: https://issues.apache.org/jira/browse/LUCENE-3203
Project: Lucene - Java
Issue Type: Improvement
Components: core/store
Reporter: Michael McCandless
Assignee: Michael McCandless
Priority: Minor
Fix For: 3.3, 4.0
Large merges can mess up searches and increase NRT reopen time (see
http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.html).
A simple rate limiter improves the spikey NRT reopen times during big
merges, so I think we should somehow make this possible. Likely this
would reduce impact on searches as well.
Typically apps that do indexing and searching on same box are in no
rush to see the merges complete so this is a good tradeoff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3203) Rate-limit IO used by merging
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3203:
---------------------------------------
Attachment: LUCENE-3203.patch
Patch, with a hacked up a prototype impl, but I don't think we should
commit it like this. Instead, I think we should wait for IOContext,
and then Dir impls can allow app to specify max merge write rate.
> Rate-limit IO used by merging
> -----------------------------
>
> Key: LUCENE-3203
> URL: https://issues.apache.org/jira/browse/LUCENE-3203
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/store
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.3, 4.0
>
> Attachments: LUCENE-3203.patch
>
>
> Large merges can mess up searches and increase NRT reopen time (see
> http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.html).
> A simple rate limiter improves the spikey NRT reopen times during big
> merges, so I think we should somehow make this possible. Likely this
> would reduce impact on searches as well.
> Typically apps that do indexing and searching on same box are in no
> rush to see the merges complete so this is a good tradeoff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3203) Rate-limit IO used by merging
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-3203:
---------------------------------------
Attachment: LUCENE-3203.patch
New patch, applies to the IOContext branch. I think it's committable!
It adds set/getMaxMergeWriteMBPerSec methods to FSDirectory.
> Rate-limit IO used by merging
> -----------------------------
>
> Key: LUCENE-3203
> URL: https://issues.apache.org/jira/browse/LUCENE-3203
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/store
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.3, 4.0
>
> Attachments: LUCENE-3203.patch, LUCENE-3203.patch
>
>
> Large merges can mess up searches and increase NRT reopen time (see
> http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.html).
> A simple rate limiter improves the spikey NRT reopen times during big
> merges, so I think we should somehow make this possible. Likely this
> would reduce impact on searches as well.
> Typically apps that do indexing and searching on same box are in no
> rush to see the merges complete so this is a good tradeoff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3203) Rate-limit IO used by merging
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-3203.
----------------------------------------
Resolution: Fixed
Fix Version/s: (was: 3.3)
(was: 4.0)
IOContext branch
> Rate-limit IO used by merging
> -----------------------------
>
> Key: LUCENE-3203
> URL: https://issues.apache.org/jira/browse/LUCENE-3203
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/store
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: IOContext branch
>
> Attachments: LUCENE-3203.patch, LUCENE-3203.patch
>
>
> Large merges can mess up searches and increase NRT reopen time (see
> http://blog.mikemccandless.com/2011/06/lucenes-near-real-time-search-is-fast.html).
> A simple rate limiter improves the spikey NRT reopen times during big
> merges, so I think we should somehow make this possible. Likely this
> would reduce impact on searches as well.
> Typically apps that do indexing and searching on same box are in no
> rush to see the merges complete so this is a good tradeoff.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org