You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by 赵 建云 <zh...@outlook.com> on 2020/12/23 09:37:06 UTC
[DISCUSS][FLINK-20726] Introduce Pulsar connector
We are now done with the Flink Pulsar connector. It is ready to be merged.
Now our plan is
1. merge Source and Sink based on Function, because it has been verified by users and is reliable.
2. Next is to merge the DynamicTable feature. table implementation depends on Step 1.
3. merge Pulsar Catalog
4. merge the PulsarSource based on FLIP-27: Refactor Source Interface<https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface> of PulsarSource
5. merge PulsarSink based on FLIP-143: Unified Sink API<https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API>
6. merge upsert-pulsar
About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>, but I don't have access to it.
StreamNative/Pulsar Flink Connector: https://github.com/streamnative/pulsar-flink
Looking forward to your feedback~
Jianyun Zhao
Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector
Posted by Till Rohrmann <tr...@apache.org>.
Hi Jianyun,
I've assigned you to the respective tickets.
Cheers,
Till
On Thu, Jan 14, 2021 at 9:07 PM 赵 建云 <zh...@outlook.com> wrote:
> Hi Arvid,
>
> Thank you very much for being a mentor to the Flink community, I will
> finish submitting all the content as soon as possible.
>
> The PR submission will be supported by me and weijie Guo(
> weijie@streamnative.io).
> I have submitted PRs for SourceFunction
> <https://github.com/apache/flink/pull/14622> and SinkFunction
> <https://github.com/apache/flink/pull/14623> now, but I see from
> flink-bot that I need to ticket and assign tasks in jira first. I need your
> help here to handle it.
>
>
> Thanks for your help!
>
> Jianyun Zhao
>
> 2021年1月6日 下午6:50，Arvid Heise <ar...@ververica.com> 写道：
>
> Hi Till,
>
> 1) Who from the Flink community will mentor this effort and could take
>
> responsibility for it?
>
>
> I'd be happy to mentor the transition. It remains to be seen who is doing
> mainly the maintenance in the long run. If all fails, I can also take that
> over but I was hoping that the respective connector authors would also
> become Flink contributors/committers in the long run.
>
> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini
>
> cluster for IT tests)?
>
>
> We use testcontainer for running a Pulsar standalone cluster to do the
> integration tests. If it is not suitable for Flink project, we are happy to
> adopt any technologies the Flink community prefers.
>
>
> We currently try to establish testcontainers for some external systems, so
> this sounds perfect to me.
>
> 3) Which Pulsar versions will/can be supported?
>
>
>
> Pulsar has a pretty good backward compatibility between versions. In our
> experience, people have been running the connector with Pulsar clusters
> from 2.5.x to 2.7.x in production.
>
>
> I must admit that I didn't think too much about versions yet. Pulsar has a
> fast development pace, so I'm assuming that we also need to cut support for
> older versions rather quickly. We should avoid supporting several major
> versions like we do with other connectors. Maybe we can phase out old
> versions into flink-packages.org?
>
> If the Pulsar community is willing to help with some of these tasks, then
>
> this should be much easier but we need buy in from someone in the Flink
> community to help with this effort.
>
>
> I hope that Sijie and I addressed your concerns.
>
> Best,
>
> Arvid
>
>
> On Wed, Jan 6, 2021 at 11:28 AM Sijie Guo <sg...@streamnative.io> wrote:
>
> Hi Till,
>
> Thank you for your email! Please find my comments inline.
>
> On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
> Hi Jianyun,
>
> Thanks a lot for reviving this discussion. I think it would be great to
> have a well working Pulsar connector for Flink. Before diving into the
> detailed plan on how to do it technically, I think we should discuss where
> exactly the connector should live. From the Flink community's perspective
> every additional connector adds a considerable maintenance burden and,
> thus, needs to be carefully considered.
>
> 1) Who from the Flink community will mentor this effort and could take
> responsibility for it?
>
>
> We had a conversation with Stephen and Arvid about this. I think Arvid
> Heise was willing to mentor this effort.
>
>
> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini
> cluster for IT tests)?
>
>
> We use testcontainer for running a Pulsar standalone cluster to do the
> integration tests. If it is not suitable for Flink project, we are happy to
> adopt any technologies the Flink community prefers.
>
>
> 2.1) Do we need additional e2e tests?
>
>
> The Pulsar-Flink connector repository contains pretty good test coverage.
> If it is not sufficient, we are happy to add more to follow Flink's best
> practices.
>
>
> 3) Which Pulsar versions will/can be supported?
>
>
> Pulsar has a pretty good backward compatibility between versions. In our
> experience, people have been running the connector with Pulsar clusters
> from 2.5.x to 2.7.x in production.
>
>
>
> If the Pulsar community is willing to help with some of these tasks, then
> this should be much easier but we need buy in from someone in the Flink
> community to help with this effort.
>
>
> I am one of the Pulsar PMC members. Some of our Pulsar PMC members and
> committers are committed to helping maintain this Pulsar Flink connector.
> Hope this helps eliminate some concerns here.
>
>
> The Flink community might be a bit unresponsive in the next couple of
> days because of Christmas and New Year. Beginning of January the discussion
> should see a couple of additional comments.
>
> Cheers,
> Till
>
> On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zh...@outlook.com> wrote:
>
> We are now done with the Flink Pulsar connector. It is ready to be
> merged.
>
> Now our plan is
>
> 1. merge Source and Sink based on Function, because it has been
> verified by users and is reliable.
> 2. Next is to merge the DynamicTable feature. table implementation
> depends on Step 1.
> 3. merge Pulsar Catalog
> 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface<
>
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-27%253A%2BRefactor%2BSource%2BInterface&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293093069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=z9qIAy47xgEXOJpPDLuj%2FXi%2B72mzym8mcO9%2FJg5HEZA%3D&reserved=0
> >
> of PulsarSource
> 5. merge PulsarSink based on FLIP-143: Unified Sink API<
>
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-143%253A%2BUnified%2BSink%2BAPI&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uBvfbkA0%2Bi0z2evYt%2FcSAqpV64Dn6ZaqLzLgREQ%2BKbo%3D&reserved=0
>
>
> 6. merge upsert-pulsar
>
> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<
>
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-72%253A%2BIntroduce%2BPulsar%2BConnector&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KyUsD5EzruT0TxHlJogV1V2FQb%2B42kP7I0eUd9Eb1LI%3D&reserved=0
> >,
> but I don't have access to it.
>
> StreamNative/Pulsar Flink Connector:
>
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fstreamnative%2Fpulsar-flink&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2oWFw2BaNYSn9kR5ZUIHWLAHUzpGoIJSeOW%2FAdw7SIY%3D&reserved=0
>
>
> Looking forward to your feedback~
>
> Jianyun Zhao
>
>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ververica.com%2F&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aLxoBwbE%2BRhC8pkQkq%2BkFwmEsq08LLRxPUgGb6C7CxM%3D&reserved=0
> >
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <
> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fflink-forward.org%2F&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MBd4VnU4DVqoDEy3iT2voj2QzqsLg2Ouf0TOuBO2%2BaE%3D&reserved=0>
> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
>
>
Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector
Posted by 赵 建云 <zh...@outlook.com>.
Hi Arvid,
Thank you very much for being a mentor to the Flink community, I will finish submitting all the content as soon as possible.
The PR submission will be supported by me and weijie Guo(weijie@streamnative.io<ma...@streamnative.io>).
I have submitted PRs for SourceFunction<https://github.com/apache/flink/pull/14622> and SinkFunction<https://github.com/apache/flink/pull/14623> now, but I see from flink-bot that I need to ticket and assign tasks in jira first. I need your help here to handle it.
Thanks for your help!
Jianyun Zhao
2021年1月6日 下午6:50，Arvid Heise <ar...@ververica.com>> 写道：
Hi Till,
1) Who from the Flink community will mentor this effort and could take
responsibility for it?
I'd be happy to mentor the transition. It remains to be seen who is doing
mainly the maintenance in the long run. If all fails, I can also take that
over but I was hoping that the respective connector authors would also
become Flink contributors/committers in the long run.
2) How can Pulsar be tested together with Flink (is there a Pulsar mini
cluster for IT tests)?
We use testcontainer for running a Pulsar standalone cluster to do the
integration tests. If it is not suitable for Flink project, we are happy to
adopt any technologies the Flink community prefers.
We currently try to establish testcontainers for some external systems, so
this sounds perfect to me.
3) Which Pulsar versions will/can be supported?
Pulsar has a pretty good backward compatibility between versions. In our
experience, people have been running the connector with Pulsar clusters
from 2.5.x to 2.7.x in production.
I must admit that I didn't think too much about versions yet. Pulsar has a
fast development pace, so I'm assuming that we also need to cut support for
older versions rather quickly. We should avoid supporting several major
versions like we do with other connectors. Maybe we can phase out old
versions into flink-packages.org<http://flink-packages.org>?
If the Pulsar community is willing to help with some of these tasks, then
this should be much easier but we need buy in from someone in the Flink
community to help with this effort.
I hope that Sijie and I addressed your concerns.
Best,
Arvid
On Wed, Jan 6, 2021 at 11:28 AM Sijie Guo <sg...@streamnative.io>> wrote:
Hi Till,
Thank you for your email! Please find my comments inline.
On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <tr...@apache.org>>
wrote:
Hi Jianyun,
Thanks a lot for reviving this discussion. I think it would be great to
have a well working Pulsar connector for Flink. Before diving into the
detailed plan on how to do it technically, I think we should discuss where
exactly the connector should live. From the Flink community's perspective
every additional connector adds a considerable maintenance burden and,
thus, needs to be carefully considered.
1) Who from the Flink community will mentor this effort and could take
responsibility for it?
We had a conversation with Stephen and Arvid about this. I think Arvid
Heise was willing to mentor this effort.
2) How can Pulsar be tested together with Flink (is there a Pulsar mini
cluster for IT tests)?
We use testcontainer for running a Pulsar standalone cluster to do the
integration tests. If it is not suitable for Flink project, we are happy to
adopt any technologies the Flink community prefers.
2.1) Do we need additional e2e tests?
The Pulsar-Flink connector repository contains pretty good test coverage.
If it is not sufficient, we are happy to add more to follow Flink's best
practices.
3) Which Pulsar versions will/can be supported?
Pulsar has a pretty good backward compatibility between versions. In our
experience, people have been running the connector with Pulsar clusters
from 2.5.x to 2.7.x in production.
If the Pulsar community is willing to help with some of these tasks, then
this should be much easier but we need buy in from someone in the Flink
community to help with this effort.
I am one of the Pulsar PMC members. Some of our Pulsar PMC members and
committers are committed to helping maintain this Pulsar Flink connector.
Hope this helps eliminate some concerns here.
The Flink community might be a bit unresponsive in the next couple of
days because of Christmas and New Year. Beginning of January the discussion
should see a couple of additional comments.
Cheers,
Till
On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zh...@outlook.com>> wrote:
We are now done with the Flink Pulsar connector. It is ready to be
merged.
Now our plan is
1. merge Source and Sink based on Function, because it has been
verified by users and is reliable.
2. Next is to merge the DynamicTable feature. table implementation
depends on Step 1.
3. merge Pulsar Catalog
4. merge the PulsarSource based on FLIP-27: Refactor Source Interface<
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-27%253A%2BRefactor%2BSource%2BInterface&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293093069%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=z9qIAy47xgEXOJpPDLuj%2FXi%2B72mzym8mcO9%2FJg5HEZA%3D&reserved=0>
of PulsarSource
5. merge PulsarSink based on FLIP-143: Unified Sink API<
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-143%253A%2BUnified%2BSink%2BAPI&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=uBvfbkA0%2Bi0z2evYt%2FcSAqpV64Dn6ZaqLzLgREQ%2BKbo%3D&reserved=0
6. merge upsert-pulsar
About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-72%253A%2BIntroduce%2BPulsar%2BConnector&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KyUsD5EzruT0TxHlJogV1V2FQb%2B42kP7I0eUd9Eb1LI%3D&reserved=0>,
but I don't have access to it.
StreamNative/Pulsar Flink Connector:
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fstreamnative%2Fpulsar-flink&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2oWFw2BaNYSn9kR5ZUIHWLAHUzpGoIJSeOW%2FAdw7SIY%3D&reserved=0
Looking forward to your feedback~
Jianyun Zhao
--
Arvid Heise | Senior Java Developer
<https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ververica.com%2F&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aLxoBwbE%2BRhC8pkQkq%2BkFwmEsq08LLRxPUgGb6C7CxM%3D&reserved=0>
Follow us @VervericaData
--
Join Flink Forward <https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fflink-forward.org%2F&data=04%7C01%7C%7Cf98034bf575f420dea5608d8b230e15f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637455270293103030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MBd4VnU4DVqoDEy3iT2voj2QzqsLg2Ouf0TOuBO2%2BaE%3D&reserved=0> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng
Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector
Posted by Till Rohrmann <tr...@apache.org>.
Seeing strong support from the Pulsar community and Arvid volunteering to
mentor this effort resolved my concerns. I think the Pulsar connector will
be a good addition to the project as it opens up more integrations.
Cheers,
Till
On Wed, Jan 6, 2021 at 11:50 AM Arvid Heise <ar...@ververica.com> wrote:
> Hi Till,
>
> 1) Who from the Flink community will mentor this effort and could take
> > responsibility for it?
> >
>
> I'd be happy to mentor the transition. It remains to be seen who is doing
> mainly the maintenance in the long run. If all fails, I can also take that
> over but I was hoping that the respective connector authors would also
> become Flink contributors/committers in the long run.
>
> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini
> >> cluster for IT tests)?
> >>
> >
> > We use testcontainer for running a Pulsar standalone cluster to do the
> > integration tests. If it is not suitable for Flink project, we are happy
> to
> > adopt any technologies the Flink community prefers.
> >
>
> We currently try to establish testcontainers for some external systems, so
> this sounds perfect to me.
>
> 3) Which Pulsar versions will/can be supported?
> >>
> >
> > Pulsar has a pretty good backward compatibility between versions. In our
> > experience, people have been running the connector with Pulsar clusters
> > from 2.5.x to 2.7.x in production.
> >
>
> I must admit that I didn't think too much about versions yet. Pulsar has a
> fast development pace, so I'm assuming that we also need to cut support for
> older versions rather quickly. We should avoid supporting several major
> versions like we do with other connectors. Maybe we can phase out old
> versions into flink-packages.org?
>
> If the Pulsar community is willing to help with some of these tasks, then
> > this should be much easier but we need buy in from someone in the Flink
> > community to help with this effort.
> >
>
> I hope that Sijie and I addressed your concerns.
>
> Best,
>
> Arvid
>
>
> On Wed, Jan 6, 2021 at 11:28 AM Sijie Guo <sg...@streamnative.io> wrote:
>
> > Hi Till,
> >
> > Thank you for your email! Please find my comments inline.
> >
> > On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <tr...@apache.org>
> > wrote:
> >
> >> Hi Jianyun,
> >>
> >> Thanks a lot for reviving this discussion. I think it would be great to
> >> have a well working Pulsar connector for Flink. Before diving into the
> >> detailed plan on how to do it technically, I think we should discuss
> where
> >> exactly the connector should live. From the Flink community's
> perspective
> >> every additional connector adds a considerable maintenance burden and,
> >> thus, needs to be carefully considered.
> >>
> >> 1) Who from the Flink community will mentor this effort and could take
> >> responsibility for it?
> >>
> >
> > We had a conversation with Stephen and Arvid about this. I think Arvid
> > Heise was willing to mentor this effort.
> >
> >
> >> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini
> >> cluster for IT tests)?
> >>
> >
> > We use testcontainer for running a Pulsar standalone cluster to do the
> > integration tests. If it is not suitable for Flink project, we are happy
> to
> > adopt any technologies the Flink community prefers.
> >
> >
> >> 2.1) Do we need additional e2e tests?
> >>
> >
> > The Pulsar-Flink connector repository contains pretty good test coverage.
> > If it is not sufficient, we are happy to add more to follow Flink's best
> > practices.
> >
> >
> >> 3) Which Pulsar versions will/can be supported?
> >>
> >
> > Pulsar has a pretty good backward compatibility between versions. In our
> > experience, people have been running the connector with Pulsar clusters
> > from 2.5.x to 2.7.x in production.
> >
> >
> >>
> >> If the Pulsar community is willing to help with some of these tasks,
> then
> >> this should be much easier but we need buy in from someone in the Flink
> >> community to help with this effort.
> >>
> >
> > I am one of the Pulsar PMC members. Some of our Pulsar PMC members and
> > committers are committed to helping maintain this Pulsar Flink connector.
> > Hope this helps eliminate some concerns here.
> >
> >>
> >> The Flink community might be a bit unresponsive in the next couple of
> >> days because of Christmas and New Year. Beginning of January the
> discussion
> >> should see a couple of additional comments.
> >>
> >> Cheers,
> >> Till
> >>
> >> On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zh...@outlook.com>
> wrote:
> >>
> >>> We are now done with the Flink Pulsar connector. It is ready to be
> >>> merged.
> >>>
> >>> Now our plan is
> >>>
> >>> 1. merge Source and Sink based on Function, because it has been
> >>> verified by users and is reliable.
> >>> 2. Next is to merge the DynamicTable feature. table implementation
> >>> depends on Step 1.
> >>> 3. merge Pulsar Catalog
> >>> 4. merge the PulsarSource based on FLIP-27: Refactor Source
> Interface<
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
> >
> >>> of PulsarSource
> >>> 5. merge PulsarSink based on FLIP-143: Unified Sink API<
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> >>> >
> >>> 6. merge upsert-pulsar
> >>>
> >>> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector
> >,
> >>> but I don't have access to it.
> >>>
> >>> StreamNative/Pulsar Flink Connector:
> >>> https://github.com/streamnative/pulsar-flink
> >>>
> >>>
> >>> Looking forward to your feedback~
> >>>
> >>> Jianyun Zhao
> >>>
> >>>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector
Posted by Arvid Heise <ar...@ververica.com>.
Hi Till,
1) Who from the Flink community will mentor this effort and could take
> responsibility for it?
>
I'd be happy to mentor the transition. It remains to be seen who is doing
mainly the maintenance in the long run. If all fails, I can also take that
over but I was hoping that the respective connector authors would also
become Flink contributors/committers in the long run.
2) How can Pulsar be tested together with Flink (is there a Pulsar mini
>> cluster for IT tests)?
>>
>
> We use testcontainer for running a Pulsar standalone cluster to do the
> integration tests. If it is not suitable for Flink project, we are happy to
> adopt any technologies the Flink community prefers.
>
We currently try to establish testcontainers for some external systems, so
this sounds perfect to me.
3) Which Pulsar versions will/can be supported?
>>
>
> Pulsar has a pretty good backward compatibility between versions. In our
> experience, people have been running the connector with Pulsar clusters
> from 2.5.x to 2.7.x in production.
>
I must admit that I didn't think too much about versions yet. Pulsar has a
fast development pace, so I'm assuming that we also need to cut support for
older versions rather quickly. We should avoid supporting several major
versions like we do with other connectors. Maybe we can phase out old
versions into flink-packages.org?
If the Pulsar community is willing to help with some of these tasks, then
> this should be much easier but we need buy in from someone in the Flink
> community to help with this effort.
>
I hope that Sijie and I addressed your concerns.
Best,
Arvid
On Wed, Jan 6, 2021 at 11:28 AM Sijie Guo <sg...@streamnative.io> wrote:
> Hi Till,
>
> Thank you for your email! Please find my comments inline.
>
> On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Jianyun,
>>
>> Thanks a lot for reviving this discussion. I think it would be great to
>> have a well working Pulsar connector for Flink. Before diving into the
>> detailed plan on how to do it technically, I think we should discuss where
>> exactly the connector should live. From the Flink community's perspective
>> every additional connector adds a considerable maintenance burden and,
>> thus, needs to be carefully considered.
>>
>> 1) Who from the Flink community will mentor this effort and could take
>> responsibility for it?
>>
>
> We had a conversation with Stephen and Arvid about this. I think Arvid
> Heise was willing to mentor this effort.
>
>
>> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini
>> cluster for IT tests)?
>>
>
> We use testcontainer for running a Pulsar standalone cluster to do the
> integration tests. If it is not suitable for Flink project, we are happy to
> adopt any technologies the Flink community prefers.
>
>
>> 2.1) Do we need additional e2e tests?
>>
>
> The Pulsar-Flink connector repository contains pretty good test coverage.
> If it is not sufficient, we are happy to add more to follow Flink's best
> practices.
>
>
>> 3) Which Pulsar versions will/can be supported?
>>
>
> Pulsar has a pretty good backward compatibility between versions. In our
> experience, people have been running the connector with Pulsar clusters
> from 2.5.x to 2.7.x in production.
>
>
>>
>> If the Pulsar community is willing to help with some of these tasks, then
>> this should be much easier but we need buy in from someone in the Flink
>> community to help with this effort.
>>
>
> I am one of the Pulsar PMC members. Some of our Pulsar PMC members and
> committers are committed to helping maintain this Pulsar Flink connector.
> Hope this helps eliminate some concerns here.
>
>>
>> The Flink community might be a bit unresponsive in the next couple of
>> days because of Christmas and New Year. Beginning of January the discussion
>> should see a couple of additional comments.
>>
>> Cheers,
>> Till
>>
>> On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zh...@outlook.com> wrote:
>>
>>> We are now done with the Flink Pulsar connector. It is ready to be
>>> merged.
>>>
>>> Now our plan is
>>>
>>> 1. merge Source and Sink based on Function, because it has been
>>> verified by users and is reliable.
>>> 2. Next is to merge the DynamicTable feature. table implementation
>>> depends on Step 1.
>>> 3. merge Pulsar Catalog
>>> 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface<
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface>
>>> of PulsarSource
>>> 5. merge PulsarSink based on FLIP-143: Unified Sink API<
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
>>> >
>>> 6. merge upsert-pulsar
>>>
>>> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>,
>>> but I don't have access to it.
>>>
>>> StreamNative/Pulsar Flink Connector:
>>> https://github.com/streamnative/pulsar-flink
>>>
>>>
>>> Looking forward to your feedback~
>>>
>>> Jianyun Zhao
>>>
>>>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng
Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector
Posted by Sijie Guo <sg...@streamnative.io.INVALID>.
Hi Till,
Thank you for your email! Please find my comments inline.
On Mon, Dec 28, 2020 at 5:50 AM Till Rohrmann <tr...@apache.org> wrote:
> Hi Jianyun,
>
> Thanks a lot for reviving this discussion. I think it would be great to
> have a well working Pulsar connector for Flink. Before diving into the
> detailed plan on how to do it technically, I think we should discuss where
> exactly the connector should live. From the Flink community's perspective
> every additional connector adds a considerable maintenance burden and,
> thus, needs to be carefully considered.
>
> 1) Who from the Flink community will mentor this effort and could take
> responsibility for it?
>
We had a conversation with Stephen and Arvid about this. I think Arvid
Heise was willing to mentor this effort.
> 2) How can Pulsar be tested together with Flink (is there a Pulsar mini
> cluster for IT tests)?
>
We use testcontainer for running a Pulsar standalone cluster to do the
integration tests. If it is not suitable for Flink project, we are happy to
adopt any technologies the Flink community prefers.
> 2.1) Do we need additional e2e tests?
>
The Pulsar-Flink connector repository contains pretty good test coverage.
If it is not sufficient, we are happy to add more to follow Flink's best
practices.
> 3) Which Pulsar versions will/can be supported?
>
Pulsar has a pretty good backward compatibility between versions. In our
experience, people have been running the connector with Pulsar clusters
from 2.5.x to 2.7.x in production.
>
> If the Pulsar community is willing to help with some of these tasks, then
> this should be much easier but we need buy in from someone in the Flink
> community to help with this effort.
>
I am one of the Pulsar PMC members. Some of our Pulsar PMC members and
committers are committed to helping maintain this Pulsar Flink connector.
Hope this helps eliminate some concerns here.
>
> The Flink community might be a bit unresponsive in the next couple of days
> because of Christmas and New Year. Beginning of January the discussion
> should see a couple of additional comments.
>
> Cheers,
> Till
>
> On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zh...@outlook.com> wrote:
>
>> We are now done with the Flink Pulsar connector. It is ready to be merged.
>>
>> Now our plan is
>>
>> 1. merge Source and Sink based on Function, because it has been
>> verified by users and is reliable.
>> 2. Next is to merge the DynamicTable feature. table implementation
>> depends on Step 1.
>> 3. merge Pulsar Catalog
>> 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface<
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface>
>> of PulsarSource
>> 5. merge PulsarSink based on FLIP-143: Unified Sink API<
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
>> >
>> 6. merge upsert-pulsar
>>
>> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>,
>> but I don't have access to it.
>>
>> StreamNative/Pulsar Flink Connector:
>> https://github.com/streamnative/pulsar-flink
>>
>>
>> Looking forward to your feedback~
>>
>> Jianyun Zhao
>>
>>
Re: [DISCUSS][FLINK-20726] Introduce Pulsar connector
Posted by Till Rohrmann <tr...@apache.org>.
Hi Jianyun,
Thanks a lot for reviving this discussion. I think it would be great to
have a well working Pulsar connector for Flink. Before diving into the
detailed plan on how to do it technically, I think we should discuss where
exactly the connector should live. From the Flink community's perspective
every additional connector adds a considerable maintenance burden and,
thus, needs to be carefully considered.
1) Who from the Flink community will mentor this effort and could take
responsibility for it?
2) How can Pulsar be tested together with Flink (is there a Pulsar mini
cluster for IT tests)?
2.1) Do we need additional e2e tests?
3) Which Pulsar versions will/can be supported?
If the Pulsar community is willing to help with some of these tasks, then
this should be much easier but we need buy in from someone in the Flink
community to help with this effort.
The Flink community might be a bit unresponsive in the next couple of days
because of Christmas and New Year. Beginning of January the discussion
should see a couple of additional comments.
Cheers,
Till
On Wed, Dec 23, 2020 at 10:37 AM 赵 建云 <zh...@outlook.com> wrote:
> We are now done with the Flink Pulsar connector. It is ready to be merged.
>
> Now our plan is
>
> 1. merge Source and Sink based on Function, because it has been
> verified by users and is reliable.
> 2. Next is to merge the DynamicTable feature. table implementation
> depends on Step 1.
> 3. merge Pulsar Catalog
> 4. merge the PulsarSource based on FLIP-27: Refactor Source Interface<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface>
> of PulsarSource
> 5. merge PulsarSink based on FLIP-143: Unified Sink API<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> >
> 6. merge upsert-pulsar
>
> About wiki, I need to update the FLIP-72: Introduce Pulsar Connector<
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-72%3A+Introduce+Pulsar+Connector>,
> but I don't have access to it.
>
> StreamNative/Pulsar Flink Connector:
> https://github.com/streamnative/pulsar-flink
>
>
> Looking forward to your feedback~
>
> Jianyun Zhao
>
>