You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@doris.apache.org by weizuo <we...@apache.org> on 2022/05/30 04:37:01 UTC
[Discuss][DSIP] Support single replica load for stream load
Hi all,
During load process, the same operation are performed on all replicas such as sort and aggregation which are resource-intensive. Concurrent data load would consume much CPU and memory resources. I plan to support the capacity of single replica load to reduce the usage of CPU and Memory for load. Performing write process (writing data into MemTable and then data flush) on single replica and synchronizing data files to other replicas before transaction finished.
The associated issue: https://github.com/apache/incubator-doris/issues/9847
Zuo Wei
Email: weizuo@apache.org
Re:Re:[Discuss][DSIP] Support single replica load for stream load
Posted by weizuo <we...@apache.org>.
I have the write permission of DSIP WIKI, and I will create a new DSIP for this feature.
Zuo Wei
At 2022-05-30 12:46:55, "陈明雨" <mo...@163.com> wrote:
>Good to see that!
>As you already have the DSIP WIKI write permission, you can create a new DSIP for this and show us some detail design about it.
>
>
>
>
>--
>
>此致！Best Regards
>陈明雨 Mingyu Chen
>
>Email:
>chenmingyu@apache.org
>
>
>
>
>
>At 2022-05-30 12:37:01, "weizuo" <we...@apache.org> wrote:
>>Hi all,
>>During load process, the same operation are performed on all replicas such as sort and aggregation which are resource-intensive. Concurrent data load would consume much CPU and memory resources. I plan to support the capacity of single replica load to reduce the usage of CPU and Memory for load. Performing write process (writing data into MemTable and then data flush) on single replica and synchronizing data files to other replicas before transaction finished.
>>The associated issue: https://github.com/apache/incubator-doris/issues/9847
>>
>>
>>Zuo Wei
>>Email: weizuo@apache.org
Re:[Discuss][DSIP] Support single replica load for stream load
Posted by 陈明雨 <mo...@163.com>.
Good to see that!
As you already have the DSIP WIKI write permission, you can create a new DSIP for this and show us some detail design about it.
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmingyu@apache.org
At 2022-05-30 12:37:01, "weizuo" <we...@apache.org> wrote:
>Hi all,
>During load process, the same operation are performed on all replicas such as sort and aggregation which are resource-intensive. Concurrent data load would consume much CPU and memory resources. I plan to support the capacity of single replica load to reduce the usage of CPU and Memory for load. Performing write process (writing data into MemTable and then data flush) on single replica and synchronizing data files to other replicas before transaction finished.
>The associated issue: https://github.com/apache/incubator-doris/issues/9847
>
>
>Zuo Wei
>Email: weizuo@apache.org