[GitHub] [incubator-doris] vagetablechicken edited a comment on issue #3143: Non blocking OlapTableSink

[GitBox <gi...@apache.org>]

vagetablechicken edited a comment on issue #3143: Non blocking OlapTableSink
URL: https://github.com/apache/incubator-doris/pull/3143#issuecomment-611406948
 
 
   @imay @morningman 
   Sorry about my change is a bit late.
   I've changed  yeild() to sleep(). And the default interval time is 10ms.
   Why 10ms, it's calculated through the avg time of add batch RPCs.
   
   If the case is large, it get faster obviously, so I focus on little cases this time.
   I tested two little cases, and got the logs about add batch time. e.g.
   `node add batch time(ms)/wait lock time(ms)/num: {10003:(76703)(0)(4336)} {10002:(87961)(0)(4921)} {10006:(80490)(0)(4570)} {10005:(71098)(0)(3984)} {10004:(87736)(0)(4686)}`
   
   So `avg time` = `add batch time` / `num`. And if you think the network time delay is not negligible, you can add it.
   In my tests, the avg times are tens of milliseconds. So for the faster sink, I set it to 10ms.
   
   And the test result(interval=10ms) is
   | test sample | origin ver(avg) | nonblocking ver(avg) |
   |-|-|-|
   | {"ScannedRows":28582315,"TaskNumber":1,"FileNumber":3,"FileSize":569042681} | 4min37s | 3min27s |
   |{"ScannedRows":100967318,"TaskNumber":1,"FileNumber":9,"FileSize":3096583144} | 12min27s | 8min27s |
    
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org