You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/03/10 09:00:51 UTC
[GitHub] [incubator-doris] vagetablechicken commented on issue #2780:
OlapTableSink::send is low efficient?
vagetablechicken commented on issue #2780: OlapTableSink::send is low efficient?
URL: https://github.com/apache/incubator-doris/issues/2780#issuecomment-596974767
https://github.com/apache/incubator-doris/pull/2956#issuecomment-596889947
As mentioned, here‘s the new design of OlapTableSink--add one sender thread to do non-blocking sending.
Let me explain the meaning of non-blocking.
The origin version of OlapTableSink can be abstracted as one queue(contains all batches in all node channels). One thread consumes the queue's items, one by one. When it wants to send a batch of the NodeChannel which has a in flight packet(rpc hasn’t returned response), it must wait(rpc join).
For example:
The batch which index id=0 && node id=2, is denoted by "B1(0-2)".
The Abstract queue as shown below.
B0(0-1) | B1(0-2) | B2(0-1) | B3(1-4) | ... | ...
-- | -- | -- | -- | -- | --
When we are sending B2(0-1), we must wait for the B0 response. But if we set aside B2(0-1) and send the next item "B3(1-4)", it won't be blocked.
So I used to split the one queue into multi queues(abandoned), as follows.( For details, see https://github.com/apache/incubator-doris/issues/2780#issuecomment-588156273)
Batches queue0:
B0(0-0) | B1(0-0) | B2(2-0) | B3(4-0) | ... | ...
-- | -- | -- | -- | -- | --
Batches queue1:
B0(2-1) | B1(0-1) | B2(0-1) | B3(1-1) | ... | ...
-- | -- | -- | -- | -- | --
Batches queue2:
B0(0-2) | B1(0-2) | B2(3-2) | B3(1-2) | ... | ...
-- | -- | -- | -- | -- | --
Each queue needs one thread to consume items. Block time is shared by multiple queues. But it's still a block way.
### The new design is non-blocking.
We can save batches in NodeChannels(pending batches), and **try to send** a pending batch. If the current channel has a in flight packet, we just skip sending in this round.
The implementation is coming soon.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org