Re: [DISCUSS] Unify the load framework in Doris

[Zhao Chun <bu...@gmail.com>]

It will be a good job for Doris.

Here are some code to do same thing, for example, the CSVScanNode can be
replaced by BrokerScanNode.
And BrokerScanNode seems to be changed another name, because that it not
only can scan from broker, but also from local file or memory sink.
Can you change it to another name?

Could you create an issue in GitHub to record this work, and add more
detail to it.

Thanks,
ZHAO Chun



Miao,Ling <mi...@baidu.com> 于2019年4月9日周二 下午4:40写道：

> Hi all,
>
> I prepare to unify the load framework in Doris.
>
> The background is following:
> There are different load frameworks belong to different load methods in
> Doris. On the one hand, the performance of broker and mini load which use
> the old framework is lower than stream load. On the other hand, the loading
> of large file is not supported by the framework of broker load which
> includes non-streaming sink.
>
> The purposes of ‘Unify the Load Framework’ are following:
>
>   1.  Improve the performance of broker and mini load by using the
> framework of stream load.
>   2.  The loading of large file is supported by the unified framework.
>
> I am looking forward to your suggestions of ‘Unify the Load Framework'.
>
> Best Regards,
> Emmy Miao
>