[jira] [Created] (LUCENE-10265) IO write throttle rate will beyond the Ceiling(1024MB/s) in the merge

["kkewwei (Jira)" <ji...@apache.org>]

kkewwei created LUCENE-10265:
--------------------------------

             Summary: IO write throttle rate will beyond the Ceiling(1024MB/s) in the merge
                 Key: LUCENE-10265
                 URL: https://issues.apache.org/jira/browse/LUCENE-10265
             Project: Lucene - Core
          Issue Type: Bug
          Components: core/other
    Affects Versions: 8.6.2
            Reporter: kkewwei


It's known that merge io write throttle rate is under the control  of `targetMBPerSec` In ConcurrentMergeSchedule, it should beyond the Ceiling(1024MB/s).

`targetMBPerSec` is shared by many merge threads, it will be changed by the way:
```
if (newBacklog) {
      // This new merge adds to the backlog: increase IO throttle by 20%
      targetMBPerSec *= 1.20; 
      if (targetMBPerSec > MAX_MERGE_MB_PER_SEC) {
        targetMBPerSec = MAX_MERGE_MB_PER_SEC;
      }
      ......
} else {
      // We are not falling behind: decrease IO throttle by 10%
      targetMBPerSec /= 1.10;
      if (targetMBPerSec < MIN_MERGE_MB_PER_SEC) {
        targetMBPerSec = MIN_MERGE_MB_PER_SEC;
      }
     ......
}
```
The modification process is not a atomic operation:
1.`targetMBPerSec` is changed by the first merge thread from 1024 to 1024*1.2
2.other merge thread will read the new value(1024*1.2).
3.the first merge thread limit the value to be 1024.
The bad case will happen.

In product, we do find that IO write throttle rate is beyond the Ceiling(1024MB/s) in the merge.
```
[2021-11-26T15:27:19,861][TRACE][o.e.i.e.E.MS             ] [data1] [test1][25] elasticsearch[data1][refresh][T#5] MS: io throttle: current merge backlog; leave IO rate at 3589.1 MB/sec
[2021-11-26T15:27:20,304][TRACE][o.e.i.e.E.MS             ] [data1] [test1][13] elasticsearch[data1][write][T#3] MS: io throttle: current merge backlog; leave IO rate at 192.4 MB/sec
[2021-11-26T15:27:25,330][TRACE][o.e.i.e.E.MS             ] [data1] [test1][22] elasticsearch[data1][[test1][22]: Lucene Merge Thread #1026] MS: io throttle: current merge backlog; leave IO rate at 96.3 MB/sec
[2021-11-26T15:27:25,995][TRACE][o.e.i.e.E.MS             ] [data1] [test1][16] elasticsearch[data1][[test1][16]: Lucene Merge Thread #1063] MS: io throttle: current merge backlog; leave IO rate at 419.2 MB/sec
[2021-11-26T15:27:38,335][TRACE][o.e.i.e.E.MS             ] [data1] [test1][19] elasticsearch[data1][write][T#2] MS: io throttle: current merge backlog; leave IO rate at 3051.5 MB/sec
```

If we shoud do the following:
1. change it by the atomic operation.
2. add the `volatile` attribute to `targetMBPerSec`.




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org