[jira] [Updated] (LUCENE-5641) SimpleRateLimiter is too simple

["Michael McCandless (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LUCENE-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael McCandless updated LUCENE-5641:
---------------------------------------

    Attachment: LUCENE-5641.patch

Here's a simple patch: it just pushes responsibility of adding up
pending (thread-private) bytes to the caller, i.e. caller should not
call RateLimiter.pause until it has written more than 5 msec worth of
bytes.

I played with another approach of doing aggregation inside
RateLimiter.pause using an AtomicLong, but it quickly gets hairy, and
at least the solution I iterated too was not scheduler-friendly (it
would pause all threads once the IO rate was exceeded), vs this patch
which will "take turns" pausing the threads.


> SimpleRateLimiter is too simple
> -------------------------------
>
>                 Key: LUCENE-5641
>                 URL: https://issues.apache.org/jira/browse/LUCENE-5641
>             Project: Lucene - Core
>          Issue Type: Bug
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 4.9, 5.0
>
>         Attachments: LUCENE-5641.patch
>
>
> I was playing with merge throttling and discovered that our
> SimpleRateLimiter is throttling far more than requested; e.g. I asked
> for 50 MB/sec merge throttling, but it throttled at more like 8
> MB/sec.
> The problem is we are calling Thread.sleep on too-small (< 1 msec)
> times; on ordinary (non-real-time) JVMs, anything less than 1 msec is
> rounded up to 1 msec.  Also, System.nanoTime() is somewhat costly.
> To fix this, I think we should aggregate the incoming byte count,
> until it crosses a threshold of enough bytes to warrant pausing.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org