You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by 孙家宝 <ji...@xtransfer.cn.INVALID> on 2022/09/22 16:01:56 UTC
[DISCUSS] FLIP-262 MongoDB Connector
Hi everyone,
I would like to start a discussion thread on FLIP-262 for a MongoDB Connector that will support both
DataStream, Table API and SQL implementations with Lookup ability.
This would be a new connector in its own external repository being
https://www.github.com/apache/flink-connector-mongodb <https://www.github.com/apache/flink-connector-mongodb >.
This repository doesn't exist yet, so we created a FLIP to discuss whether we need to create it.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector >
Looking forward to any comments or feedback.
Best Regards,
Jiabao
Re: [DISCUSS] FLIP-262 MongoDB Connector
Posted by Martijn Visser <ma...@apache.org>.
Hi Jiabao,
Thanks for creating the FLIP. It looks like there's is enthusiasm for a
MongoDB connector so that's good.
What is also good to mention is that you have already prepared a PR at
https://github.com/apache/flink/pull/20848 - That would need to move to the
dedicated repository.
Overall +1 from my end
Thanks, Martijn
Op do 22 sep. 2022 om 18:02 schreef 孙家宝 <ji...@xtransfer.cn.invalid>
> Hi everyone,
> I would like to start a discussion thread on FLIP-262 for a MongoDB
> Connector that will support both
> DataStream, Table API and SQL implementations with Lookup ability.
> This would be a new connector in its own external repository being
> https://www.github.com/apache/flink-connector-mongodb <
> https://www.github.com/apache/flink-connector-mongodb >.
> This repository doesn't exist yet, so we created a FLIP to discuss whether
> we need to create it.
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector
> >
> Looking forward to any comments or feedback.
> Best Regards,
> Jiabao
>
Re: [DISCUSS] FLIP-262 MongoDB Connector
Posted by Jiabao Sun <ji...@xtransfer.cn.INVALID>.
Hi Qingsheng,
Thanks For the feedback.
Other metrics mentioned in FLIP-33, we will also implement it.
Best,
Jiabao
------------------------------------------------------------------
From:Qingsheng Ren <re...@gmail.com>
Send Time:2022年9月28日(星期三) 18:36
To:孙家宝 <ji...@xtransfer.cn>
Cc:dev <de...@flink.apache.org>
Subject:Re: [DISCUSS] FLIP-262 MongoDB Connector
Thanks for the update Jiabao.
c. It’ll be nice to have all metrics defined in FLIP-33 in the connector. Also it’s better to have some MongoDB specific metrics, which will be very helpful for users to monitor their jobs.
Best,
Qingsheng
On Sep 27, 2022, at 11:59, Jiabao Sun <jiabao.sun@xtransfer.cn <mailto:jiabao.sun@xtransfer.cn >> wrote:
Hi Qingsheng,
Thanks for the great suggestion.
a. Currently we have implemented the Source of FLIP-177 in https://github.com/apache/flink/pull/20848 <https://github.com/apache/flink/pull/20848 >.
Sorry for some mistakes in the Public Interface of FLIP-262, the interface of Sink has been adjusted from FLIP-143 to FLIP-177.
b. The AsyncWriter mentioned by FLIP-171 looks helpful, I'll try to implement it.
c. We have provided some common metrics mentioned by FLIP-33.
Source: numRecordsSend
Sink: numRecordsSend/currentSendTime
Best,
Jiabao
------------------------------------------------------------------
From:Qingsheng Ren <renqs@apache.org <mailto:renqs@apache.org >>
Send Time:2022年9月27日(星期二) 10:43
To:dev <dev@flink.apache.org <mailto:dev@flink.apache.org >>
Cc:孙家宝 <jiabao.sun@xtransfer.cn <mailto:jiabao.sun@xtransfer.cn >>
Subject:Re: [DISCUSS] FLIP-262 MongoDB Connector
Hi Jiabao,
Thanks for the FLIP! Very glad to see having a new connector in Flink eco-system, so overall +1 from my side.
In addition to those FLIPs you mention in the proposal, here’s some other FLIPs that you may be interested in:
a. FLIP-177 [1] extended Sink API by Sink API v2 with some additional features, so it’ll be great to implement the sink directly on the latest interface.
b. FLIP-171 [2] introduced async sink that you can to reuse if at-least-once semantic could fulfill the requirement of MongoDB sink.
c. FLIP-33 [3] defined general source and sink metrics that connectors should report.
Looking forward to the new connector!
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API <https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API >
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink <https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink >
[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics <https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics >
Best,
Qingsheng
On Sep 23, 2022, at 00:01, 孙家宝 <jiabao.sun@xtransfer.cn.INVALID <mailto:jiabao.sun@xtransfer.cn.INVALID >> wrote:
Hi everyone,
I would like to start a discussion thread on FLIP-262 for a MongoDB Connector that will support both
DataStream, Table API and SQL implementations with Lookup ability.
This would be a new connector in its own external repository being
https://www.github.com/apache/flink-connector-mongodb <https://www.github.com/apache/flink-connector-mongodb > <https://www.github.com/apache/flink-connector-mongodb <https://www.github.com/apache/flink-connector-mongodb > >.
This repository doesn't exist yet, so we created a FLIP to discuss whether we need to create it.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector > <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector > >
Looking forward to any comments or feedback.
Best Regards,
Jiabao
Re: [DISCUSS] FLIP-262 MongoDB Connector
Posted by Qingsheng Ren <re...@gmail.com>.
Thanks for the update Jiabao.
c. It’ll be nice to have all metrics defined in FLIP-33 in the connector. Also it’s better to have some MongoDB specific metrics, which will be very helpful for users to monitor their jobs.
Best,
Qingsheng
> On Sep 27, 2022, at 11:59, Jiabao Sun <jiabao.sun@xtransfer.cn <ma...@xtransfer.cn>> wrote:
>
> Hi Qingsheng,
>
> Thanks for the great suggestion.
>
> a. Currently we have implemented the Source of FLIP-177 in https://github.com/apache/flink/pull/20848 <https://github.com/apache/flink/pull/20848>.
> Sorry for some mistakes in the Public Interface of FLIP-262, the interface of Sink has been adjusted from FLIP-143 to FLIP-177.
>
> b. The AsyncWriter mentioned by FLIP-171 looks helpful, I'll try to implement it.
>
> c. We have provided some common metrics mentioned by FLIP-33.
> Source: numRecordsSend
> Sink: numRecordsSend/currentSendTime
>
> Best,
> Jiabao
>
> ------------------------------------------------------------------
> From:Qingsheng Ren <renqs@apache.org <ma...@apache.org>>
> Send Time:2022年9月27日(星期二) 10:43
> To:dev <dev@flink.apache.org <ma...@flink.apache.org>>
> Cc:孙家宝 <jiabao.sun@xtransfer.cn <ma...@xtransfer.cn>>
> Subject:Re: [DISCUSS] FLIP-262 MongoDB Connector
>
> Hi Jiabao,
>
> Thanks for the FLIP! Very glad to see having a new connector in Flink eco-system, so overall +1 from my side.
>
> In addition to those FLIPs you mention in the proposal, here’s some other FLIPs that you may be interested in:
>
> a. FLIP-177 [1] extended Sink API by Sink API v2 with some additional features, so it’ll be great to implement the sink directly on the latest interface.
> b. FLIP-171 [2] introduced async sink that you can to reuse if at-least-once semantic could fulfill the requirement of MongoDB sink.
> c. FLIP-33 [3] defined general source and sink metrics that connectors should report.
>
> Looking forward to the new connector!
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API <https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API>
> [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink <https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink>
> [3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics <https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics>
>
> Best,
> Qingsheng
>
>
>> On Sep 23, 2022, at 00:01, 孙家宝 <jiabao.sun@xtransfer.cn.INVALID <ma...@xtransfer.cn.INVALID>> wrote:
>>
>> Hi everyone,
>> I would like to start a discussion thread on FLIP-262 for a MongoDB Connector that will support both
>> DataStream, Table API and SQL implementations with Lookup ability.
>> This would be a new connector in its own external repository being
>> https://www.github.com/apache/flink-connector-mongodb <https://www.github.com/apache/flink-connector-mongodb> <https://www.github.com/apache/flink-connector-mongodb <https://www.github.com/apache/flink-connector-mongodb> >.
>> This repository doesn't exist yet, so we created a FLIP to discuss whether we need to create it.
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector> >
>> Looking forward to any comments or feedback.
>> Best Regards,
>> Jiabao
>
Re: [DISCUSS] FLIP-262 MongoDB Connector
Posted by Jiabao Sun <ji...@xtransfer.cn.INVALID>.
Hi Qingsheng,
Thanks for the great suggestion.
a. Currently we have implemented the Source of FLIP-177 in https://github.com/apache/flink/pull/20848.
Sorry for some mistakes in the Public Interface of FLIP-262, the interface of Sink has been adjusted from FLIP-143 to FLIP-177.
b. The AsyncWriter mentioned by FLIP-171 looks helpful, I'll try to implement it.
c. We have provided some common metrics mentioned by FLIP-33.
Source: numRecordsSend
Sink: numRecordsSend/currentSendTime
Best,
Jiabao
------------------------------------------------------------------
From:Qingsheng Ren <re...@apache.org>
Send Time:2022年9月27日(星期二) 10:43
To:dev <de...@flink.apache.org>
Cc:孙家宝 <ji...@xtransfer.cn>
Subject:Re: [DISCUSS] FLIP-262 MongoDB Connector
Hi Jiabao,
Thanks for the FLIP! Very glad to see having a new connector in Flink eco-system, so overall +1 from my side.
In addition to those FLIPs you mention in the proposal, here’s some other FLIPs that you may be interested in:
a. FLIP-177 [1] extended Sink API by Sink API v2 with some additional features, so it’ll be great to implement the sink directly on the latest interface.
b. FLIP-171 [2] introduced async sink that you can to reuse if at-least-once semantic could fulfill the requirement of MongoDB sink.
c. FLIP-33 [3] defined general source and sink metrics that connectors should report.
Looking forward to the new connector!
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
Best,
Qingsheng
> On Sep 23, 2022, at 00:01, 孙家宝 <ji...@xtransfer.cn.INVALID> wrote:
>
> Hi everyone,
> I would like to start a discussion thread on FLIP-262 for a MongoDB Connector that will support both
> DataStream, Table API and SQL implementations with Lookup ability.
> This would be a new connector in its own external repository being
> https://www.github.com/apache/flink-connector-mongodb <https://www.github.com/apache/flink-connector-mongodb >.
> This repository doesn't exist yet, so we created a FLIP to discuss whether we need to create it.
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector >
> Looking forward to any comments or feedback.
> Best Regards,
> Jiabao
Re: [DISCUSS] FLIP-262 MongoDB Connector
Posted by Qingsheng Ren <re...@apache.org>.
Hi Jiabao,
Thanks for the FLIP! Very glad to see having a new connector in Flink eco-system, so overall +1 from my side.
In addition to those FLIPs you mention in the proposal, here’s some other FLIPs that you may be interested in:
a. FLIP-177 [1] extended Sink API by Sink API v2 with some additional features, so it’ll be great to implement the sink directly on the latest interface.
b. FLIP-171 [2] introduced async sink that you can to reuse if at-least-once semantic could fulfill the requirement of MongoDB sink.
c. FLIP-33 [3] defined general source and sink metrics that connectors should report.
Looking forward to the new connector!
[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API
[2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
[3] https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
Best,
Qingsheng
> On Sep 23, 2022, at 00:01, 孙家宝 <ji...@xtransfer.cn.INVALID> wrote:
>
> Hi everyone,
> I would like to start a discussion thread on FLIP-262 for a MongoDB Connector that will support both
> DataStream, Table API and SQL implementations with Lookup ability.
> This would be a new connector in its own external repository being
> https://www.github.com/apache/flink-connector-mongodb <https://www.github.com/apache/flink-connector-mongodb >.
> This repository doesn't exist yet, so we created a FLIP to discuss whether we need to create it.
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector <https://cwiki.apache.org/confluence/display/FLINK/FLIP-262%3A+Introduce+MongoDB+connector >
> Looking forward to any comments or feedback.
> Best Regards,
> Jiabao