You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Radim Kolar (JIRA)" <ji...@apache.org> on 2012/11/05 00:32:12 UTC

[jira] [Created] (SOLR-4030) Use Lucene segment merge throttling

Radim Kolar created SOLR-4030:
---------------------------------

             Summary: Use Lucene segment merge throttling
                 Key: SOLR-4030
                 URL: https://issues.apache.org/jira/browse/SOLR-4030
             Project: Solr
          Issue Type: Improvement
    Affects Versions: 5.0
            Reporter: Radim Kolar
         Attachments: solr-mergeratelimit.txt

add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505790#comment-13505790 ] 

Radim Kolar commented on SOLR-4030:
-----------------------------------

it still merges into trunk cleanly. just do git pull
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Assigned] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller reassigned SOLR-4030:
---------------------------------

    Assignee: Mark Miller
    
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Assigned] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller reassigned SOLR-4030:
---------------------------------

    Assignee:     (was: Mark Miller)
    
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505866#comment-13505866 ] 

Radim Kolar commented on SOLR-4030:
-----------------------------------

LUCENE-4537 is cause. I will rework it but its not clear if new rate limiter is used for ordinary writes (not just merges) as well.
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mark Miller updated SOLR-4030:
------------------------------

             Priority: Minor  (was: Major)
    Affects Version/s:     (was: 5.0)
        Fix Version/s: 5.0
    
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505161#comment-13505161 ] 

Mark Miller commented on SOLR-4030:
-----------------------------------

This would be nice to add - could you update this patch to 5x trunk Radmin?
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508813#comment-13508813 ] 

Radim Kolar commented on SOLR-4030:
-----------------------------------

any progress on this? its trivial patch.
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Radim Kolar updated SOLR-4030:
------------------------------

    Attachment: solr-ratelimit2.txt
    
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505892#comment-13505892 ] 

Radim Kolar commented on SOLR-4030:
-----------------------------------

added limits for reading, merging and writing.
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Radim Kolar updated SOLR-4030:
------------------------------

    Attachment: solr-mergeratelimit.txt
    
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>    Affects Versions: 5.0
>            Reporter: Radim Kolar
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505855#comment-13505855 ] 

Radim Kolar commented on SOLR-4030:
-----------------------------------

but it failed to build with 5.x in jenkins. i will take a more detailed look on it
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Radim Kolar updated SOLR-4030:
------------------------------

    Fix Version/s: 4.1
    
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>    Affects Versions: 5.0
>            Reporter: Radim Kolar
>              Labels: patch
>             Fix For: 4.1
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Radim Kolar (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Radim Kolar updated SOLR-4030:
------------------------------

    Labels: patch  (was: )
    
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>    Affects Versions: 5.0
>            Reporter: Radim Kolar
>              Labels: patch
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505840#comment-13505840 ] 

Mark Miller commented on SOLR-4030:
-----------------------------------

bq. it still merges into trunk cleanly. 

Sure, it applies - it does not compile.

bq.  just do git pull

I'm using your patch and an svn checkout - no git to be found.
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (SOLR-4030) Use Lucene segment merge throttling

Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/SOLR-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505908#comment-13505908 ] 

Mark Miller commented on SOLR-4030:
-----------------------------------

Looks good. I'll look at adding a unit test. I'd also rather detect when no settings have been entered (all unlimited) and not wrap the dir with a rate limiting wrapper in that case.

We also do an instance of check in 4x against fsdir - so we should make that work with this.
                
> Use Lucene segment merge throttling
> -----------------------------------
>
>                 Key: SOLR-4030
>                 URL: https://issues.apache.org/jira/browse/SOLR-4030
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Radim Kolar
>            Assignee: Mark Miller
>            Priority: Minor
>              Labels: patch
>             Fix For: 4.1, 5.0
>
>         Attachments: solr-mergeratelimit.txt, solr-ratelimit2.txt
>
>
> add argument "maxMergeWriteMBPerSec" to Solr directory factories.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org