You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/02/19 10:43:49 UTC
[GitHub] [incubator-doris] vagetablechicken commented on issue #2780:
OlapTableSink::send is low efficient?
vagetablechicken commented on issue #2780: OlapTableSink::send is low efficient?
URL: https://github.com/apache/incubator-doris/issues/2780#issuecomment-588156273
# OlapTableSink Multithreading Solution Design
## single-thread model (original model)
## multi-thread model
When we create an OlapTableSink, prepare N buffers and N send threads.
If a row needs to be added to NodeChannel(node_id=A, only consider node_id), the main thread copy it to buffer A%N.
We can limit the buffer size(use MemTracker), the mem_limit is configurable. If a buffer is full, we block adding rows to this buffer , until all rows in buffer have been consumed.
### Extra cost of multi-thread ver
thread: buffer_num
mem: buffer_num*(mem_limit+buffer_running_mem)
### single/multi version switch
The acceleration effect is evident for large data imports. But no need for every OlapTableSink.
So I design it:
We can select origin(single-thread) version or multi-thread version by set broker load property, and configure the buffer_num & mem_limit_per_buffer.
We add fields to TOlapTableSink, like
```
struct TOlapTableSink {
...
14: optional i64 load_channel_timeout_s // the timeout of load channels in second
15: optional i32 buffer_num
16: optional i64 mem_limit_per_buf
17: optional i64 size_limit_per_buf
}
```
`buffer_num > 0` means using "multi-thread & buffer" mode.
`buffer_num = 0` or not set means using "single-thread" mode(the origin mode).
(`buffer_num = 1` is hard to define. We should avoid setting buffer_num to 1.)
So we can use
```
LOAD LABEL ...
PROPERTIES
(
"buffer_num"="5",
"mem_limit_per_buf"="5368709120",
"size_limit_per_buf"="62914560"
);
```
# A Test Case
cluster: 5 be
{"ScannedRows":895737370,"TaskNumber":1,"FileNumber":300,"FileSize":60216822679} // 56G
origin ver
| LoadStartTime | LoadFinishTime |
|----|----|
- [ ] TODO
multi-thread ver
buffer_num/mem_limit_per_buf/size_limit_per_buf = 5/1G/30M
| LoadStartTime | LoadFinishTime |
|----|----|
| 2020-02-18 17:57:38 | 2020-02-18 20:05:13 |
buffer_num/mem_limit_per_buf/size_limit_per_buf = 5/5G/30M
| LoadStartTime | LoadFinishTime |
|----|----|
| 2020-02-18 20:14:58 | 2020-02-18 22:19:54 |
buffer_num/mem_limit_per_buf/size_limit_per_buf = 5/5G/60M
| LoadStartTime | LoadFinishTime |
|----|----|
| 2020-02-18 22:28:08 | 2020-02-19 00:36:28 |
- [ ] TODO test analysis
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org