You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@doris.apache.org by 晖 <82...@qq.com.INVALID> on 2022/03/29 10:17:02 UTC
[Discuss][DSIP] Support load binlog from MySQL directly without relying other service
As issue #8025(https://github.com/apache/incubator-doris/issues/8025) mentioned, the current binlog load is relying on Canal service. We can integrate the binlog aquire and parse function into Doris, so that user can be more convinience to use it.Please determine the final DSIP number for it.
回复：[Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by 晖 <82...@qq.com.INVALID>.
Hi,
I have registered my account, and my cwiki account is `tianhui`. Thanks!
--
Hui Tian
------------------ 原始邮件 ------------------
发件人: "dev" <morningman@163.com>;
发送时间: 2022年3月29日(星期二) 晚上7:45
收件人: "dev"<dev@doris.apache.org>;
主题: Re:[Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Hi Hui,
Thanks for it. Could you register a cwiki acount and tell me your account id? So that I can create DSIP and grant write priv for you.
https://cwiki.apache.org/confluence/signup.action
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmingyu@apache.org
At 2022-03-29 18:17:02, "晖" <827677355@qq.com.INVALID> wrote:
>As issue #8025(https://github.com/apache/incubator-doris/issues/8025)&nbsp;mentioned, the current binlog load is relying on Canal service. We can integrate the binlog aquire and parse function into Doris, so that user can be more convinience to use it.Please determine the final DSIP number for it.
Re:Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by 蔡聪辉 <ca...@163.com>.
I think one of the elegant implementations may be something like routine load
在 2022-03-30 10:47:39，"ling miao" <li...@apache.org> 写道：
>> Do you mean 'canal' component?
>Yes.
>
>It sounds like you mean a new way about binlog load?
>You can show the architecture diagram and general design on the wiki, so
>that I can better understand your intentions.[1]
>
>Ling Miao
>[
>https://cwiki.apache.org/confluence/display/DORIS/DSIP-003%3A+Directly+binlog+consumption
>]
>
>晖 <82...@qq.com.invalid> 于2022年3月29日周二 20:56写道：
>
>> Hi Ling,
>> Do you mean 'canal' component? I have not cancelled any component,
>> only extended current binlog load with a new type of 'debezium'.
>>
>>
>> Because the Canal service is not provided in my company's product
>> environment,
>> users can't use the current binlog load. I think the new type simplifies
>> the usage
>> of binlog load, and shortens the data flow.
>>
>>
>> The similiar function is provided by ClickHouse. I think it's a brilliant
>> experience to
>> use a MPP database as a MySQL's slave. It's not the same background with
>> Flink
>> Connector.
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> 发件人:
>> "dev"
>> <
>> lingmiao@apache.org>;
>> 发送时间: 2022年3月29日(星期二) 晚上8:13
>> 收件人: "dev"<dev@doris.apache.org>;
>>
>> 主题: Re: [Discuss][DSIP] Support load binlog from MySQL directly
>> without relying other service
>>
>>
>>
>> Hi hui,
>>
>> I want to know what is the reason for canceling the cancel component? Is
>> there any problem in use? What are the benefits ?
>>
>> Parsing binlog is not actually the function of Doris itself. Or is it
>> better to appear as a connector? For example, like Flink connector.
>>
>> Ling Miao
>>
>> 陈明雨 <morningman@163.com> 于2022年3月29日周二 19:46写道：
>>
>> > Hi Hui,
>> > Thanks for it. Could you register a cwiki acount and tell me your
>> account
>> > id? So that I can create DSIP and grant write priv for you.
>> > https://cwiki.apache.org/confluence/signup.action
>> >
>> >
>> >
>> >
>> > --
>> >
>> > 此致！Best Regards
>> > 陈明雨 Mingyu Chen
>> >
>> > Email:
>> > chenmingyu@apache.org
>> >
>> >
>> >
>> >
>> >
>> > At 2022-03-29 18:17:02, "晖" <827677355@qq.com.INVALID> wrote:
>> > >As issue #8025(
>> >
>> https://github.com/apache/incubator-doris/issues/8025)&nbsp;mentioned,
>> > the current binlog load is relying on Canal service. We can integrate
>> the
>> > binlog aquire and parse function into Doris, so that user can be more
>> > convinience to use it.Please determine the final DSIP number for it.
>> >
Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by ling miao <li...@apache.org>.
> Do you mean 'canal' component?
Yes.
It sounds like you mean a new way about binlog load?
You can show the architecture diagram and general design on the wiki, so
that I can better understand your intentions.[1]
Ling Miao
[
https://cwiki.apache.org/confluence/display/DORIS/DSIP-003%3A+Directly+binlog+consumption
]
晖 <82...@qq.com.invalid> 于2022年3月29日周二 20:56写道：
> Hi Ling,
> Do you mean 'canal' component? I have not cancelled any component,
> only extended current binlog load with a new type of 'debezium'.
>
>
> Because the Canal service is not provided in my company's product
> environment,
> users can't use the current binlog load. I think the new type simplifies
> the usage
> of binlog load, and shortens the data flow.
>
>
> The similiar function is provided by ClickHouse. I think it's a brilliant
> experience to
> use a MPP database as a MySQL's slave. It's not the same background with
> Flink
> Connector.
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人:
> "dev"
> <
> lingmiao@apache.org>;
> 发送时间: 2022年3月29日(星期二) 晚上8:13
> 收件人: "dev"<dev@doris.apache.org>;
>
> 主题: Re: [Discuss][DSIP] Support load binlog from MySQL directly
> without relying other service
>
>
>
> Hi hui,
>
> I want to know what is the reason for canceling the cancel component? Is
> there any problem in use? What are the benefits ?
>
> Parsing binlog is not actually the function of Doris itself. Or is it
> better to appear as a connector? For example, like Flink connector.
>
> Ling Miao
>
> 陈明雨 <morningman@163.com> 于2022年3月29日周二 19:46写道：
>
> > Hi Hui,
> > Thanks for it. Could you register a cwiki acount and tell me your
> account
> > id? So that I can create DSIP and grant write priv for you.
> > https://cwiki.apache.org/confluence/signup.action
> >
> >
> >
> >
> > --
> >
> > 此致！Best Regards
> > 陈明雨 Mingyu Chen
> >
> > Email:
> > chenmingyu@apache.org
> >
> >
> >
> >
> >
> > At 2022-03-29 18:17:02, "晖" <827677355@qq.com.INVALID> wrote:
> > >As issue #8025(
> >
> https://github.com/apache/incubator-doris/issues/8025)&nbsp;mentioned,
> > the current binlog load is relying on Canal service. We can integrate
> the
> > binlog aquire and parse function into Doris, so that user can be more
> > convinience to use it.Please determine the final DSIP number for it.
> >
回复： [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by 晖 <82...@qq.com.INVALID>.
Hi Ling,
Do you mean 'canal' component? I have not cancelled any component,
only extended current binlog load with a new type of 'debezium'.
Because the Canal service is not provided in my company's product environment,
users can't use the current binlog load. I think the new type simplifies the usage
of binlog load, and shortens the data flow.
The similiar function is provided by ClickHouse. I think it's a brilliant experience to
use a MPP database as a MySQL's slave. It's not the same background with Flink
Connector.
------------------ 原始邮件 ------------------
发件人: "dev" <lingmiao@apache.org>;
发送时间: 2022年3月29日(星期二) 晚上8:13
收件人: "dev"<dev@doris.apache.org>;
主题: Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Hi hui,
I want to know what is the reason for canceling the cancel component? Is
there any problem in use? What are the benefits ?
Parsing binlog is not actually the function of Doris itself. Or is it
better to appear as a connector? For example, like Flink connector.
Ling Miao
陈明雨 <morningman@163.com> 于2022年3月29日周二 19:46写道：
> Hi Hui,
> Thanks for it. Could you register a cwiki acount and tell me your account
> id? So that I can create DSIP and grant write priv for you.
> https://cwiki.apache.org/confluence/signup.action
>
>
>
>
> --
>
> 此致！Best Regards
> 陈明雨 Mingyu Chen
>
> Email:
> chenmingyu@apache.org
>
>
>
>
>
> At 2022-03-29 18:17:02, "晖" <827677355@qq.com.INVALID> wrote:
> >As issue #8025(
> https://github.com/apache/incubator-doris/issues/8025)&nbsp;mentioned,
> the current binlog load is relying on Canal service. We can integrate the
> binlog aquire and parse function into Doris, so that user can be more
> convinience to use it.Please determine the final DSIP number for it.
>
Re:回复： [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by 陈明雨 <mo...@163.com>.
Hi hui,
I created DSIP-003 for you: https://cwiki.apache.org/confluence/display/DORIS/DSIP-003%3A+Directly+binlog+consumption
And I have some questions:
1.
Using debezium can indeed simplify the analysis difficulty of binglog, but the disadvantage is that it occupies the computing resources of FE.
This is also the disadvantage of the current Canal solution. So we need some test data to illustrate the resource overhead of this scheme
and what synchronization rate can be achieved.
2. What are the advantages and disadvantages of this scheme compared to Canal solution? Is it possible to completely replace Canal's solution?
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmingyu@apache.org
在 2022-03-29 21:05:08，"晖" <82...@qq.com.INVALID> 写道：
>Hi Yang,
>
>
>I have not found any library we can use. Maybe it's a lot of work to put it on BE.
>
>
>I think it's not necessary to put it in BE, Because we can distribute the sync job into
>other FE instead of Master, and user can use Flink Connector if there is still too much data.
>
>
>
>
>------------------ 原始邮件 ------------------
>发件人: "dev" <xuyang25@baidu.com>;
>发送时间: 2022年3月29日(星期二) 晚上8:38
>收件人: "dev@doris.apache.org"<dev@doris.apache.org>;
>
>主题: Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
>
>
>
>Hi hui,
>Thanks for the pr.
>Is there any way to integrate the binlog aquire and parse function into Be, instead of Fe?
>
>在 2022/3/29 下午8:13，“ling miao”<lingmiao@apache.org> 写入:
>
> Hi hui,
>
> I want to know what is the reason for canceling the cancel component? Is
> there any problem in use? What are the benefits ?
>
> Parsing binlog is not actually the function of Doris itself. Or is it
> better to appear as a connector? For example, like Flink connector.
>
> Ling Miao
>
> 陈明雨 <morningman@163.com> 于2022年3月29日周二 19:46写道：
>
> > Hi Hui,
> > Thanks for it. Could you register a cwiki acount and tell me your account
> > id? So that I can create DSIP and grant write priv for you.
> > https://cwiki.apache.org/confluence/signup.action
> >
> >
> >
> >
> > --
> >
> > 此致！Best Regards
> > 陈明雨 Mingyu Chen
> >
> > Email:
> > chenmingyu@apache.org
> >
> >
> >
> >
> >
> > At 2022-03-29 18:17:02, "晖" <827677355@qq.com.INVALID> wrote:
> > >As issue #8025(
> > https://github.com/apache/incubator-doris/issues/8025)&nbsp;mentioned,
> > the current binlog load is relying on Canal service. We can integrate the
> > binlog aquire and parse function into Doris, so that user can be more
> > convinience to use it.Please determine the final DSIP number for it.
> >
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
>For additional commands, e-mail: dev-help@doris.apache.org
回复： [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by 晖 <82...@qq.com.INVALID>.
Hi Yang,
I have not found any library we can use. Maybe it's a lot of work to put it on BE.
I think it's not necessary to put it in BE, Because we can distribute the sync job into
other FE instead of Master, and user can use Flink Connector if there is still too much data.
------------------ 原始邮件 ------------------
发件人: "dev" <xuyang25@baidu.com>;
发送时间: 2022年3月29日(星期二) 晚上8:38
收件人: "dev@doris.apache.org"<dev@doris.apache.org>;
主题: Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Hi hui,
Thanks for the pr.
Is there any way to integrate the binlog aquire and parse function into Be, instead of Fe?
在 2022/3/29 下午8:13，“ling miao”<lingmiao@apache.org> 写入:
Hi hui,
I want to know what is the reason for canceling the cancel component? Is
there any problem in use? What are the benefits ?
Parsing binlog is not actually the function of Doris itself. Or is it
better to appear as a connector? For example, like Flink connector.
Ling Miao
陈明雨 <morningman@163.com> 于2022年3月29日周二 19:46写道：
> Hi Hui,
> Thanks for it. Could you register a cwiki acount and tell me your account
> id? So that I can create DSIP and grant write priv for you.
> https://cwiki.apache.org/confluence/signup.action
>
>
>
>
> --
>
> 此致！Best Regards
> 陈明雨 Mingyu Chen
>
> Email:
> chenmingyu@apache.org
>
>
>
>
>
> At 2022-03-29 18:17:02, "晖" <827677355@qq.com.INVALID> wrote:
> >As issue #8025(
> https://github.com/apache/incubator-doris/issues/8025)&nbsp;mentioned,
> the current binlog load is relying on Canal service. We can integrate the
> binlog aquire and parse function into Doris, so that user can be more
> convinience to use it.Please determine the final DSIP number for it.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by "Xu,Yang(INF)" <xu...@baidu.com>.
Hi hui,
Thanks for the pr.
Is there any way to integrate the binlog aquire and parse function into Be, instead of Fe?
在 2022/3/29 下午8:13，“ling miao”<li...@apache.org> 写入:
Hi hui,
I want to know what is the reason for canceling the cancel component? Is
there any problem in use? What are the benefits ?
Parsing binlog is not actually the function of Doris itself. Or is it
better to appear as a connector? For example, like Flink connector.
Ling Miao
陈明雨 <mo...@163.com> 于2022年3月29日周二 19:46写道：
> Hi Hui,
> Thanks for it. Could you register a cwiki acount and tell me your account
> id? So that I can create DSIP and grant write priv for you.
> https://cwiki.apache.org/confluence/signup.action
>
>
>
>
> --
>
> 此致！Best Regards
> 陈明雨 Mingyu Chen
>
> Email:
> chenmingyu@apache.org
>
>
>
>
>
> At 2022-03-29 18:17:02, "晖" <82...@qq.com.INVALID> wrote:
> >As issue #8025(
> https://github.com/apache/incubator-doris/issues/8025) mentioned,
> the current binlog load is relying on Canal service. We can integrate the
> binlog aquire and parse function into Doris, so that user can be more
> convinience to use it.Please determine the final DSIP number for it.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re: [Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by ling miao <li...@apache.org>.
Hi hui,
I want to know what is the reason for canceling the cancel component? Is
there any problem in use? What are the benefits ?
Parsing binlog is not actually the function of Doris itself. Or is it
better to appear as a connector? For example, like Flink connector.
Ling Miao
陈明雨 <mo...@163.com> 于2022年3月29日周二 19:46写道：
> Hi Hui,
> Thanks for it. Could you register a cwiki acount and tell me your account
> id? So that I can create DSIP and grant write priv for you.
> https://cwiki.apache.org/confluence/signup.action
>
>
>
>
> --
>
> 此致！Best Regards
> 陈明雨 Mingyu Chen
>
> Email:
> chenmingyu@apache.org
>
>
>
>
>
> At 2022-03-29 18:17:02, "晖" <82...@qq.com.INVALID> wrote:
> >As issue #8025(
> https://github.com/apache/incubator-doris/issues/8025) mentioned,
> the current binlog load is relying on Canal service. We can integrate the
> binlog aquire and parse function into Doris, so that user can be more
> convinience to use it.Please determine the final DSIP number for it.
>
Re:[Discuss][DSIP] Support load binlog from MySQL directly without relying other service
Posted by 陈明雨 <mo...@163.com>.
Hi Hui,
Thanks for it. Could you register a cwiki acount and tell me your account id? So that I can create DSIP and grant write priv for you.
https://cwiki.apache.org/confluence/signup.action
--
此致！Best Regards
陈明雨 Mingyu Chen
Email:
chenmingyu@apache.org
At 2022-03-29 18:17:02, "晖" <82...@qq.com.INVALID> wrote:
>As issue #8025(https://github.com/apache/incubator-doris/issues/8025) mentioned, the current binlog load is relying on Canal service. We can integrate the binlog aquire and parse function into Doris, so that user can be more convinience to use it.Please determine the final DSIP number for it.