[GitHub] [incubator-pegasus] Shuo-Jia opened a new issue #802: perf: seperate copy checkpoint from `learn` flow

[GitBox <gi...@apache.org>]

Shuo-Jia opened a new issue #802:
URL: https://github.com/apache/incubator-pegasus/issues/802


   Copy large checkpoint when learn has huge impact, In the past, we only think it because of the increase of `io write load`. However, I did two comparative tests and find the design of `nfs copy checkpoint` may be  unreasonable.
   
   ### Test One: `restore` vs `learn` with same rate
   In theroy,  if io write rate is same, `restore` should has same impact with `learn`, but the test show different result:
   | Rate(MB/s)   | case     | io util    |  P99   |
   | --------   | ----- | -----:   | :----: |
   | 400 | restore      | 20%      |   50ms    |
   | 400  | learn      | 80%      |   1000ms    |
   
   From above, I found same io rate has different IO util and cause different latency. But if you think if only IO utill casue the result, please see the next test
   
   ### Test Two:  `learn` vs `no learn` but with same write rate and same io utils:
   In this test, I let the util keep same, the only difference is another test has no `learn` flow, we get the result:
    | Rate(MB/s)   | case     | io util    |  P99   |
   | --------   | ----- | -----:   | :----: |
   | 400 | no learn      | 80%      |   200ms    |
   | 400  | learn      | 80%      |   1000ms    |
   
   From above, I found that the latency of `io util load`caused  only account for a small part
   
   ## Conclusion
   Write latency increase reason has two part:
   
   - io load increase: 200ms
   - learn thread task queue: 800ms
   
   So if we re-implement `learn checkpoint`, surpose current latency is 1000ms, and then:
   
   1. if seprate copy checkpoint from `learn` flow in `write thread pool` so as to it won't competing thread resources, latency can decrease 1000ms to 200ms
   2. if improve copy checkpoint has same io utils impact with `restore`, latecny can decrease from 200ms to 50ms
   
   Actually, `copy checkpoint` should be a independent task like `restore`download file from
   remote filesystem and only cause small io utils and little latency impact
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org