You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Ran Tao <ch...@gmail.com> on 2023/07/03 13:35:51 UTC
[DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
I have an issue here that needs to upgrade commons-collections[1] (this is
an example), but PR ci fails because flink-python test cases depend on
flink-sql-connector-kafka, but kafka-sql-connector is a small jar, does not
include this dependency, so flink ci cause exception[2]. Current my
solution is [3]. But even if this PR is done, the upgrade of flink still
requires kafka-connector released.
This issue leads to deeper problems. Although the connectors have been
externalized, many UTs of flink-python depend on these connectors, and a
basic agreement of externalized connectors is that other dependencies
cannot be introduced explicitly, which means the externalized connectors
use dependencies inherited from flink. In this way, when flink main
upgrades some dependencies, it is easy to fail when executing flink-python
test cases，because flink no longer has this class, and the connector does
not contain it. It's circular problem.
Unless, the connector self-consistently includes all dependencies, which is
uncontrollable.
(only a few connectors include all jars in shade phase)
In short, the current flink-python module's dependencies on the connector
leads to an incomplete process of externalization and decoupling, which
will lead to circular dependencies when flink upgrade or change some
dependencies.
I don't know if I made it clear. I hope to get everyone's opinions on what
better solutions we should adopt for similar problems in the future.
[1] https://issues.apache.org/jira/browse/FLINK-30274
[2]
https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
[3] https://github.com/apache/flink-connector-kafka/pull/38
Best Regards,
Ran Tao
https://github.com/chucheng92
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Jing Ge <ji...@ververica.com.INVALID>.
Hi Ran Tao,
What is the current status? @Dian There were many options. Which one is the
most feasible one you prefer?
Best regards,
Jing
On Fri, Jul 7, 2023 at 2:37 PM Mason Chen <ma...@gmail.com> wrote:
> Hi all,
>
> I also agree with what's been said above.
>
> +1, I think the Table API delegation is a good suggestion--it essentially
> allows a connector to get Python support for free. We've seen that
> Table/SQL and Python APIs complement each other well and are ideal for data
> scientists. With respect to unaligned functionalities, I think that also
> holds true for other APIs, e.g. Datastream and Table/SQL since there is
> functionality that is not natural to represent as a configuration/SQL.
>
> Best,
> Mason
>
> On Wed, Jul 5, 2023 at 10:14 PM Dian Fu <di...@gmail.com> wrote:
>
> > Hi Chesnay,
> >
> > >> The wrapping of connectors is a bit of a maintenance nightmare and
> > doesn't really work with external/custom connectors.
> >
> > Cannot agree with you more.
> >
> > >> Has there ever been thoughts about changing flink-pythons connector
> > setup to use the table api connectors underneath?
> >
> > I'm still not sure if this is feasible for all connectors, however,
> > this may be a good idea. The concern is that the DataStream API
> > connectors functionalities may be unaligned between Java and Python
> > connectors. Besides, there are still a few connectors which only have
> > DataStream API connectors, e.g. Google PubSub, RabbitMQ, Cassandra,
> > Pulsar, Hybrid Source, etc. Besides, it currently already supports
> > Table API connectors in PyFlink and if we take this way, maybe we
> > could just tell users to use Table API connector directly.
> >
> > Another option in my head before is to provide an API which allows
> > configuring the behavior via key/value pairs in both the Java & Python
> > DataStream API connectors.
> >
> > Regards,
> > Dian
> >
> > On Wed, Jul 5, 2023 at 6:34 PM Chesnay Schepler <ch...@apache.org>
> > wrote:
> > >
> > > Has there ever been thoughts about changing flink-pythons connector
> > > setup to use the table api connectors underneath?
> > >
> > > The wrapping of connectors is a bit of a maintenance nightmare and
> > > doesn't really work with external/custom connectors.
> > >
> > > On 04/07/2023 13:35, Dian Fu wrote:
> > > > Thanks Ran Tao for proposing this discussion and Martijn for sharing
> > > > the thought.
> > > >
> > > >> While flink-python now fails the CI, it shouldn't actually depend
> > on the
> > > > externalized connectors. I'm not sure what PyFlink does with it, but
> if
> > > > belongs to the connector code,
> > > >
> > > > For each DataStream connector, there is a corresponding Python
> wrapper
> > > > and also some test cases in PyFlink. In theory, we should move that
> > > > wrapper into each connector repository. In the past, we have not done
> > > > that when externalizing the connectors since it may introduce some
> > > > burden when releasing since it means that we have to publish each
> > > > connector to PyPI separately.
> > > >
> > > > To resolve this problem, I guess we can move the connector support in
> > > > PyFlink into the external connector repository.
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > >
> > > > On Mon, Jul 3, 2023 at 11:08 PM Ran Tao <ch...@gmail.com>
> wrote:
> > > >> @Martijn
> > > >> thanks for clear explanations.
> > > >>
> > > >> If we follow the line you specified (Connectors shouldn't rely on
> > > >> dependencies that may or may not be
> > > >> available in Flink itself)
> > > >> It seems that we should add a certain dependency if we need(such as
> > > >> commons-io, commons-collection) in connector pom explicitly.
> > > >> And bundle it in sql-connector uber jar.
> > > >>
> > > >> Then there is only one thing left that we need to make flink-python
> > test
> > > >> not depend on the released flink-connector.
> > > >> Maybe we should check it out and decouple it like you suggested.
> > > >>
> > > >> Best Regards,
> > > >> Ran Tao
> > > >> https://github.com/chucheng92
> > > >>
> > > >>
> > > >> Martijn Visser <ma...@apache.org> 于2023年7月3日周一 22:06写道：
> > > >>
> > > >>> Hi Ran Tao,
> > > >>>
> > > >>> Thanks for opening this topic. I think there's a couple of things
> at
> > hand:
> > > >>> 1. Connectors shouldn't rely on dependencies that may or may not be
> > > >>> available in Flink itself, like we've seen with flink-shaded. That
> > avoids a
> > > >>> tight coupling between Flink and connectors, which is exactly what
> > we try
> > > >>> to avoid.
> > > >>> 2. When following that line, that would also be applicable for
> > things like
> > > >>> commons-collections and commons-io. If a connector wants to use
> > them, it
> > > >>> should make sure that it bundles those artifacts itself.
> > > >>> 3. While flink-python now fails the CI, it shouldn't actually
> depend
> > on the
> > > >>> externalized connectors. I'm not sure what PyFlink does with it,
> but
> > if
> > > >>> belongs to the connector code, that code should also be moved to
> the
> > > >>> individual connector repo. If it's just a generic test, we could
> > consider
> > > >>> creating a generic test against released connector versions to
> > determine
> > > >>> compatibility.
> > > >>>
> > > >>> I'm curious about the opinions of others as well.
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> Martijn
> > > >>>
> > > >>> On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com>
> > wrote:
> > > >>>
> > > >>>> I have an issue here that needs to upgrade commons-collections[1]
> > (this
> > > >>> is
> > > >>>> an example), but PR ci fails because flink-python test cases
> depend
> > on
> > > >>>> flink-sql-connector-kafka, but kafka-sql-connector is a small jar,
> > does
> > > >>> not
> > > >>>> include this dependency, so flink ci cause exception[2]. Current
> my
> > > >>>> solution is [3]. But even if this PR is done, the upgrade of flink
> > still
> > > >>>> requires kafka-connector released.
> > > >>>>
> > > >>>> This issue leads to deeper problems. Although the connectors have
> > been
> > > >>>> externalized, many UTs of flink-python depend on these connectors,
> > and a
> > > >>>> basic agreement of externalized connectors is that other
> > dependencies
> > > >>>> cannot be introduced explicitly, which means the externalized
> > connectors
> > > >>>> use dependencies inherited from flink. In this way, when flink
> main
> > > >>>> upgrades some dependencies, it is easy to fail when executing
> > > >>> flink-python
> > > >>>> test cases，because flink no longer has this class, and the
> > connector does
> > > >>>> not contain it. It's circular problem.
> > > >>>>
> > > >>>> Unless, the connector self-consistently includes all dependencies,
> > which
> > > >>> is
> > > >>>> uncontrollable.
> > > >>>> (only a few connectors include all jars in shade phase)
> > > >>>>
> > > >>>> In short, the current flink-python module's dependencies on the
> > connector
> > > >>>> leads to an incomplete process of externalization and decoupling,
> > which
> > > >>>> will lead to circular dependencies when flink upgrade or change
> some
> > > >>>> dependencies.
> > > >>>>
> > > >>>> I don't know if I made it clear. I hope to get everyone's opinions
> > on
> > > >>> what
> > > >>>> better solutions we should adopt for similar problems in the
> future.
> > > >>>>
> > > >>>> [1] https://issues.apache.org/jira/browse/FLINK-30274
> > > >>>> [2]
> > > >>>>
> > > >>>>
> > > >>>
> >
> https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
> > > >>>>
> > > >>>>
> > > >>>
> >
> https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> > > >>>> [3] https://github.com/apache/flink-connector-kafka/pull/38
> > > >>>>
> > > >>>> Best Regards,
> > > >>>> Ran Tao
> > > >>>> https://github.com/chucheng92
> > > >>>>
> > >
> >
>
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Mason Chen <ma...@gmail.com>.
Hi all,
I also agree with what's been said above.
+1, I think the Table API delegation is a good suggestion--it essentially
allows a connector to get Python support for free. We've seen that
Table/SQL and Python APIs complement each other well and are ideal for data
scientists. With respect to unaligned functionalities, I think that also
holds true for other APIs, e.g. Datastream and Table/SQL since there is
functionality that is not natural to represent as a configuration/SQL.
Best,
Mason
On Wed, Jul 5, 2023 at 10:14 PM Dian Fu <di...@gmail.com> wrote:
> Hi Chesnay,
>
> >> The wrapping of connectors is a bit of a maintenance nightmare and
> doesn't really work with external/custom connectors.
>
> Cannot agree with you more.
>
> >> Has there ever been thoughts about changing flink-pythons connector
> setup to use the table api connectors underneath?
>
> I'm still not sure if this is feasible for all connectors, however,
> this may be a good idea. The concern is that the DataStream API
> connectors functionalities may be unaligned between Java and Python
> connectors. Besides, there are still a few connectors which only have
> DataStream API connectors, e.g. Google PubSub, RabbitMQ, Cassandra,
> Pulsar, Hybrid Source, etc. Besides, it currently already supports
> Table API connectors in PyFlink and if we take this way, maybe we
> could just tell users to use Table API connector directly.
>
> Another option in my head before is to provide an API which allows
> configuring the behavior via key/value pairs in both the Java & Python
> DataStream API connectors.
>
> Regards,
> Dian
>
> On Wed, Jul 5, 2023 at 6:34 PM Chesnay Schepler <ch...@apache.org>
> wrote:
> >
> > Has there ever been thoughts about changing flink-pythons connector
> > setup to use the table api connectors underneath?
> >
> > The wrapping of connectors is a bit of a maintenance nightmare and
> > doesn't really work with external/custom connectors.
> >
> > On 04/07/2023 13:35, Dian Fu wrote:
> > > Thanks Ran Tao for proposing this discussion and Martijn for sharing
> > > the thought.
> > >
> > >> While flink-python now fails the CI, it shouldn't actually depend
> on the
> > > externalized connectors. I'm not sure what PyFlink does with it, but if
> > > belongs to the connector code,
> > >
> > > For each DataStream connector, there is a corresponding Python wrapper
> > > and also some test cases in PyFlink. In theory, we should move that
> > > wrapper into each connector repository. In the past, we have not done
> > > that when externalizing the connectors since it may introduce some
> > > burden when releasing since it means that we have to publish each
> > > connector to PyPI separately.
> > >
> > > To resolve this problem, I guess we can move the connector support in
> > > PyFlink into the external connector repository.
> > >
> > > Regards,
> > > Dian
> > >
> > >
> > > On Mon, Jul 3, 2023 at 11:08 PM Ran Tao <ch...@gmail.com> wrote:
> > >> @Martijn
> > >> thanks for clear explanations.
> > >>
> > >> If we follow the line you specified (Connectors shouldn't rely on
> > >> dependencies that may or may not be
> > >> available in Flink itself)
> > >> It seems that we should add a certain dependency if we need(such as
> > >> commons-io, commons-collection) in connector pom explicitly.
> > >> And bundle it in sql-connector uber jar.
> > >>
> > >> Then there is only one thing left that we need to make flink-python
> test
> > >> not depend on the released flink-connector.
> > >> Maybe we should check it out and decouple it like you suggested.
> > >>
> > >> Best Regards,
> > >> Ran Tao
> > >> https://github.com/chucheng92
> > >>
> > >>
> > >> Martijn Visser <ma...@apache.org> 于2023年7月3日周一 22:06写道：
> > >>
> > >>> Hi Ran Tao,
> > >>>
> > >>> Thanks for opening this topic. I think there's a couple of things at
> hand:
> > >>> 1. Connectors shouldn't rely on dependencies that may or may not be
> > >>> available in Flink itself, like we've seen with flink-shaded. That
> avoids a
> > >>> tight coupling between Flink and connectors, which is exactly what
> we try
> > >>> to avoid.
> > >>> 2. When following that line, that would also be applicable for
> things like
> > >>> commons-collections and commons-io. If a connector wants to use
> them, it
> > >>> should make sure that it bundles those artifacts itself.
> > >>> 3. While flink-python now fails the CI, it shouldn't actually depend
> on the
> > >>> externalized connectors. I'm not sure what PyFlink does with it, but
> if
> > >>> belongs to the connector code, that code should also be moved to the
> > >>> individual connector repo. If it's just a generic test, we could
> consider
> > >>> creating a generic test against released connector versions to
> determine
> > >>> compatibility.
> > >>>
> > >>> I'm curious about the opinions of others as well.
> > >>>
> > >>> Best regards,
> > >>>
> > >>> Martijn
> > >>>
> > >>> On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com>
> wrote:
> > >>>
> > >>>> I have an issue here that needs to upgrade commons-collections[1]
> (this
> > >>> is
> > >>>> an example), but PR ci fails because flink-python test cases depend
> on
> > >>>> flink-sql-connector-kafka, but kafka-sql-connector is a small jar,
> does
> > >>> not
> > >>>> include this dependency, so flink ci cause exception[2]. Current my
> > >>>> solution is [3]. But even if this PR is done, the upgrade of flink
> still
> > >>>> requires kafka-connector released.
> > >>>>
> > >>>> This issue leads to deeper problems. Although the connectors have
> been
> > >>>> externalized, many UTs of flink-python depend on these connectors,
> and a
> > >>>> basic agreement of externalized connectors is that other
> dependencies
> > >>>> cannot be introduced explicitly, which means the externalized
> connectors
> > >>>> use dependencies inherited from flink. In this way, when flink main
> > >>>> upgrades some dependencies, it is easy to fail when executing
> > >>> flink-python
> > >>>> test cases，because flink no longer has this class, and the
> connector does
> > >>>> not contain it. It's circular problem.
> > >>>>
> > >>>> Unless, the connector self-consistently includes all dependencies,
> which
> > >>> is
> > >>>> uncontrollable.
> > >>>> (only a few connectors include all jars in shade phase)
> > >>>>
> > >>>> In short, the current flink-python module's dependencies on the
> connector
> > >>>> leads to an incomplete process of externalization and decoupling,
> which
> > >>>> will lead to circular dependencies when flink upgrade or change some
> > >>>> dependencies.
> > >>>>
> > >>>> I don't know if I made it clear. I hope to get everyone's opinions
> on
> > >>> what
> > >>>> better solutions we should adopt for similar problems in the future.
> > >>>>
> > >>>> [1] https://issues.apache.org/jira/browse/FLINK-30274
> > >>>> [2]
> > >>>>
> > >>>>
> > >>>
> https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
> > >>>>
> > >>>>
> > >>>
> https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> > >>>> [3] https://github.com/apache/flink-connector-kafka/pull/38
> > >>>>
> > >>>> Best Regards,
> > >>>> Ran Tao
> > >>>> https://github.com/chucheng92
> > >>>>
> >
>
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Dian Fu <di...@gmail.com>.
Hi Chesnay,
>> The wrapping of connectors is a bit of a maintenance nightmare and
doesn't really work with external/custom connectors.
Cannot agree with you more.
>> Has there ever been thoughts about changing flink-pythons connector
setup to use the table api connectors underneath?
I'm still not sure if this is feasible for all connectors, however,
this may be a good idea. The concern is that the DataStream API
connectors functionalities may be unaligned between Java and Python
connectors. Besides, there are still a few connectors which only have
DataStream API connectors, e.g. Google PubSub, RabbitMQ, Cassandra,
Pulsar, Hybrid Source, etc. Besides, it currently already supports
Table API connectors in PyFlink and if we take this way, maybe we
could just tell users to use Table API connector directly.
Another option in my head before is to provide an API which allows
configuring the behavior via key/value pairs in both the Java & Python
DataStream API connectors.
Regards,
Dian
On Wed, Jul 5, 2023 at 6:34 PM Chesnay Schepler <ch...@apache.org> wrote:
>
> Has there ever been thoughts about changing flink-pythons connector
> setup to use the table api connectors underneath?
>
> The wrapping of connectors is a bit of a maintenance nightmare and
> doesn't really work with external/custom connectors.
>
> On 04/07/2023 13:35, Dian Fu wrote:
> > Thanks Ran Tao for proposing this discussion and Martijn for sharing
> > the thought.
> >
> >> While flink-python now fails the CI, it shouldn't actually depend on the
> > externalized connectors. I'm not sure what PyFlink does with it, but if
> > belongs to the connector code,
> >
> > For each DataStream connector, there is a corresponding Python wrapper
> > and also some test cases in PyFlink. In theory, we should move that
> > wrapper into each connector repository. In the past, we have not done
> > that when externalizing the connectors since it may introduce some
> > burden when releasing since it means that we have to publish each
> > connector to PyPI separately.
> >
> > To resolve this problem, I guess we can move the connector support in
> > PyFlink into the external connector repository.
> >
> > Regards,
> > Dian
> >
> >
> > On Mon, Jul 3, 2023 at 11:08 PM Ran Tao <ch...@gmail.com> wrote:
> >> @Martijn
> >> thanks for clear explanations.
> >>
> >> If we follow the line you specified (Connectors shouldn't rely on
> >> dependencies that may or may not be
> >> available in Flink itself)
> >> It seems that we should add a certain dependency if we need(such as
> >> commons-io, commons-collection) in connector pom explicitly.
> >> And bundle it in sql-connector uber jar.
> >>
> >> Then there is only one thing left that we need to make flink-python test
> >> not depend on the released flink-connector.
> >> Maybe we should check it out and decouple it like you suggested.
> >>
> >> Best Regards,
> >> Ran Tao
> >> https://github.com/chucheng92
> >>
> >>
> >> Martijn Visser <ma...@apache.org> 于2023年7月3日周一 22:06写道：
> >>
> >>> Hi Ran Tao,
> >>>
> >>> Thanks for opening this topic. I think there's a couple of things at hand:
> >>> 1. Connectors shouldn't rely on dependencies that may or may not be
> >>> available in Flink itself, like we've seen with flink-shaded. That avoids a
> >>> tight coupling between Flink and connectors, which is exactly what we try
> >>> to avoid.
> >>> 2. When following that line, that would also be applicable for things like
> >>> commons-collections and commons-io. If a connector wants to use them, it
> >>> should make sure that it bundles those artifacts itself.
> >>> 3. While flink-python now fails the CI, it shouldn't actually depend on the
> >>> externalized connectors. I'm not sure what PyFlink does with it, but if
> >>> belongs to the connector code, that code should also be moved to the
> >>> individual connector repo. If it's just a generic test, we could consider
> >>> creating a generic test against released connector versions to determine
> >>> compatibility.
> >>>
> >>> I'm curious about the opinions of others as well.
> >>>
> >>> Best regards,
> >>>
> >>> Martijn
> >>>
> >>> On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com> wrote:
> >>>
> >>>> I have an issue here that needs to upgrade commons-collections[1] (this
> >>> is
> >>>> an example), but PR ci fails because flink-python test cases depend on
> >>>> flink-sql-connector-kafka, but kafka-sql-connector is a small jar, does
> >>> not
> >>>> include this dependency, so flink ci cause exception[2]. Current my
> >>>> solution is [3]. But even if this PR is done, the upgrade of flink still
> >>>> requires kafka-connector released.
> >>>>
> >>>> This issue leads to deeper problems. Although the connectors have been
> >>>> externalized, many UTs of flink-python depend on these connectors, and a
> >>>> basic agreement of externalized connectors is that other dependencies
> >>>> cannot be introduced explicitly, which means the externalized connectors
> >>>> use dependencies inherited from flink. In this way, when flink main
> >>>> upgrades some dependencies, it is easy to fail when executing
> >>> flink-python
> >>>> test cases，because flink no longer has this class, and the connector does
> >>>> not contain it. It's circular problem.
> >>>>
> >>>> Unless, the connector self-consistently includes all dependencies, which
> >>> is
> >>>> uncontrollable.
> >>>> (only a few connectors include all jars in shade phase)
> >>>>
> >>>> In short, the current flink-python module's dependencies on the connector
> >>>> leads to an incomplete process of externalization and decoupling, which
> >>>> will lead to circular dependencies when flink upgrade or change some
> >>>> dependencies.
> >>>>
> >>>> I don't know if I made it clear. I hope to get everyone's opinions on
> >>> what
> >>>> better solutions we should adopt for similar problems in the future.
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/FLINK-30274
> >>>> [2]
> >>>>
> >>>>
> >>> https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
> >>>>
> >>>>
> >>> https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> >>>> [3] https://github.com/apache/flink-connector-kafka/pull/38
> >>>>
> >>>> Best Regards,
> >>>> Ran Tao
> >>>> https://github.com/chucheng92
> >>>>
>
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Chesnay Schepler <ch...@apache.org>.
Has there ever been thoughts about changing flink-pythons connector
setup to use the table api connectors underneath?
The wrapping of connectors is a bit of a maintenance nightmare and
doesn't really work with external/custom connectors.
On 04/07/2023 13:35, Dian Fu wrote:
> Thanks Ran Tao for proposing this discussion and Martijn for sharing
> the thought.
>
>> While flink-python now fails the CI, it shouldn't actually depend on the
> externalized connectors. I'm not sure what PyFlink does with it, but if
> belongs to the connector code,
>
> For each DataStream connector, there is a corresponding Python wrapper
> and also some test cases in PyFlink. In theory, we should move that
> wrapper into each connector repository. In the past, we have not done
> that when externalizing the connectors since it may introduce some
> burden when releasing since it means that we have to publish each
> connector to PyPI separately.
>
> To resolve this problem, I guess we can move the connector support in
> PyFlink into the external connector repository.
>
> Regards,
> Dian
>
>
> On Mon, Jul 3, 2023 at 11:08 PM Ran Tao <ch...@gmail.com> wrote:
>> @Martijn
>> thanks for clear explanations.
>>
>> If we follow the line you specified (Connectors shouldn't rely on
>> dependencies that may or may not be
>> available in Flink itself)
>> It seems that we should add a certain dependency if we need(such as
>> commons-io, commons-collection) in connector pom explicitly.
>> And bundle it in sql-connector uber jar.
>>
>> Then there is only one thing left that we need to make flink-python test
>> not depend on the released flink-connector.
>> Maybe we should check it out and decouple it like you suggested.
>>
>> Best Regards,
>> Ran Tao
>> https://github.com/chucheng92
>>
>>
>> Martijn Visser <ma...@apache.org> 于2023年7月3日周一 22:06写道：
>>
>>> Hi Ran Tao,
>>>
>>> Thanks for opening this topic. I think there's a couple of things at hand:
>>> 1. Connectors shouldn't rely on dependencies that may or may not be
>>> available in Flink itself, like we've seen with flink-shaded. That avoids a
>>> tight coupling between Flink and connectors, which is exactly what we try
>>> to avoid.
>>> 2. When following that line, that would also be applicable for things like
>>> commons-collections and commons-io. If a connector wants to use them, it
>>> should make sure that it bundles those artifacts itself.
>>> 3. While flink-python now fails the CI, it shouldn't actually depend on the
>>> externalized connectors. I'm not sure what PyFlink does with it, but if
>>> belongs to the connector code, that code should also be moved to the
>>> individual connector repo. If it's just a generic test, we could consider
>>> creating a generic test against released connector versions to determine
>>> compatibility.
>>>
>>> I'm curious about the opinions of others as well.
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>> On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com> wrote:
>>>
>>>> I have an issue here that needs to upgrade commons-collections[1] (this
>>> is
>>>> an example), but PR ci fails because flink-python test cases depend on
>>>> flink-sql-connector-kafka, but kafka-sql-connector is a small jar, does
>>> not
>>>> include this dependency, so flink ci cause exception[2]. Current my
>>>> solution is [3]. But even if this PR is done, the upgrade of flink still
>>>> requires kafka-connector released.
>>>>
>>>> This issue leads to deeper problems. Although the connectors have been
>>>> externalized, many UTs of flink-python depend on these connectors, and a
>>>> basic agreement of externalized connectors is that other dependencies
>>>> cannot be introduced explicitly, which means the externalized connectors
>>>> use dependencies inherited from flink. In this way, when flink main
>>>> upgrades some dependencies, it is easy to fail when executing
>>> flink-python
>>>> test cases，because flink no longer has this class, and the connector does
>>>> not contain it. It's circular problem.
>>>>
>>>> Unless, the connector self-consistently includes all dependencies, which
>>> is
>>>> uncontrollable.
>>>> (only a few connectors include all jars in shade phase)
>>>>
>>>> In short, the current flink-python module's dependencies on the connector
>>>> leads to an incomplete process of externalization and decoupling, which
>>>> will lead to circular dependencies when flink upgrade or change some
>>>> dependencies.
>>>>
>>>> I don't know if I made it clear. I hope to get everyone's opinions on
>>> what
>>>> better solutions we should adopt for similar problems in the future.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-30274
>>>> [2]
>>>>
>>>>
>>> https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
>>>>
>>>>
>>> https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
>>>> [3] https://github.com/apache/flink-connector-kafka/pull/38
>>>>
>>>> Best Regards,
>>>> Ran Tao
>>>> https://github.com/chucheng92
>>>>
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Ran Tao <ch...@gmail.com>.
Thanks Dian Fu for sharing the externalized connector usage within
flink-python.
Looking forward to the solution to this problem, and if possible, can
participate in the contribution of this problem,
and thank you Martijn for the dependency usage that the externalized
connector should follow.
Best Regards,
Ran Tao
https://github.com/chucheng92
Dian Fu <di...@gmail.com> 于2023年7月4日周二 19:37写道：
> Thanks Ran Tao for proposing this discussion and Martijn for sharing
> the thought.
>
> > While flink-python now fails the CI, it shouldn't actually depend on the
> externalized connectors. I'm not sure what PyFlink does with it, but if
> belongs to the connector code,
>
> For each DataStream connector, there is a corresponding Python wrapper
> and also some test cases in PyFlink. In theory, we should move that
> wrapper into each connector repository. In the past, we have not done
> that when externalizing the connectors since it may introduce some
> burden when releasing since it means that we have to publish each
> connector to PyPI separately.
>
> To resolve this problem, I guess we can move the connector support in
> PyFlink into the external connector repository.
>
> Regards,
> Dian
>
>
> On Mon, Jul 3, 2023 at 11:08 PM Ran Tao <ch...@gmail.com> wrote:
> >
> > @Martijn
> > thanks for clear explanations.
> >
> > If we follow the line you specified (Connectors shouldn't rely on
> > dependencies that may or may not be
> > available in Flink itself)
> > It seems that we should add a certain dependency if we need(such as
> > commons-io, commons-collection) in connector pom explicitly.
> > And bundle it in sql-connector uber jar.
> >
> > Then there is only one thing left that we need to make flink-python test
> > not depend on the released flink-connector.
> > Maybe we should check it out and decouple it like you suggested.
> >
> > Best Regards,
> > Ran Tao
> > https://github.com/chucheng92
> >
> >
> > Martijn Visser <ma...@apache.org> 于2023年7月3日周一 22:06写道：
> >
> > > Hi Ran Tao,
> > >
> > > Thanks for opening this topic. I think there's a couple of things at
> hand:
> > > 1. Connectors shouldn't rely on dependencies that may or may not be
> > > available in Flink itself, like we've seen with flink-shaded. That
> avoids a
> > > tight coupling between Flink and connectors, which is exactly what we
> try
> > > to avoid.
> > > 2. When following that line, that would also be applicable for things
> like
> > > commons-collections and commons-io. If a connector wants to use them,
> it
> > > should make sure that it bundles those artifacts itself.
> > > 3. While flink-python now fails the CI, it shouldn't actually depend
> on the
> > > externalized connectors. I'm not sure what PyFlink does with it, but if
> > > belongs to the connector code, that code should also be moved to the
> > > individual connector repo. If it's just a generic test, we could
> consider
> > > creating a generic test against released connector versions to
> determine
> > > compatibility.
> > >
> > > I'm curious about the opinions of others as well.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com> wrote:
> > >
> > > > I have an issue here that needs to upgrade commons-collections[1]
> (this
> > > is
> > > > an example), but PR ci fails because flink-python test cases depend
> on
> > > > flink-sql-connector-kafka, but kafka-sql-connector is a small jar,
> does
> > > not
> > > > include this dependency, so flink ci cause exception[2]. Current my
> > > > solution is [3]. But even if this PR is done, the upgrade of flink
> still
> > > > requires kafka-connector released.
> > > >
> > > > This issue leads to deeper problems. Although the connectors have
> been
> > > > externalized, many UTs of flink-python depend on these connectors,
> and a
> > > > basic agreement of externalized connectors is that other dependencies
> > > > cannot be introduced explicitly, which means the externalized
> connectors
> > > > use dependencies inherited from flink. In this way, when flink main
> > > > upgrades some dependencies, it is easy to fail when executing
> > > flink-python
> > > > test cases，because flink no longer has this class, and the connector
> does
> > > > not contain it. It's circular problem.
> > > >
> > > > Unless, the connector self-consistently includes all dependencies,
> which
> > > is
> > > > uncontrollable.
> > > > (only a few connectors include all jars in shade phase)
> > > >
> > > > In short, the current flink-python module's dependencies on the
> connector
> > > > leads to an incomplete process of externalization and decoupling,
> which
> > > > will lead to circular dependencies when flink upgrade or change some
> > > > dependencies.
> > > >
> > > > I don't know if I made it clear. I hope to get everyone's opinions on
> > > what
> > > > better solutions we should adopt for similar problems in the future.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-30274
> > > > [2]
> > > >
> > > >
> > >
> https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
> > > >
> > > >
> > > >
> > >
> https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> > > > [3] https://github.com/apache/flink-connector-kafka/pull/38
> > > >
> > > > Best Regards,
> > > > Ran Tao
> > > > https://github.com/chucheng92
> > > >
> > >
>
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Dian Fu <di...@gmail.com>.
Thanks Ran Tao for proposing this discussion and Martijn for sharing
the thought.
> While flink-python now fails the CI, it shouldn't actually depend on the
externalized connectors. I'm not sure what PyFlink does with it, but if
belongs to the connector code,
For each DataStream connector, there is a corresponding Python wrapper
and also some test cases in PyFlink. In theory, we should move that
wrapper into each connector repository. In the past, we have not done
that when externalizing the connectors since it may introduce some
burden when releasing since it means that we have to publish each
connector to PyPI separately.
To resolve this problem, I guess we can move the connector support in
PyFlink into the external connector repository.
Regards,
Dian
On Mon, Jul 3, 2023 at 11:08 PM Ran Tao <ch...@gmail.com> wrote:
>
> @Martijn
> thanks for clear explanations.
>
> If we follow the line you specified (Connectors shouldn't rely on
> dependencies that may or may not be
> available in Flink itself)
> It seems that we should add a certain dependency if we need(such as
> commons-io, commons-collection) in connector pom explicitly.
> And bundle it in sql-connector uber jar.
>
> Then there is only one thing left that we need to make flink-python test
> not depend on the released flink-connector.
> Maybe we should check it out and decouple it like you suggested.
>
> Best Regards,
> Ran Tao
> https://github.com/chucheng92
>
>
> Martijn Visser <ma...@apache.org> 于2023年7月3日周一 22:06写道：
>
> > Hi Ran Tao,
> >
> > Thanks for opening this topic. I think there's a couple of things at hand:
> > 1. Connectors shouldn't rely on dependencies that may or may not be
> > available in Flink itself, like we've seen with flink-shaded. That avoids a
> > tight coupling between Flink and connectors, which is exactly what we try
> > to avoid.
> > 2. When following that line, that would also be applicable for things like
> > commons-collections and commons-io. If a connector wants to use them, it
> > should make sure that it bundles those artifacts itself.
> > 3. While flink-python now fails the CI, it shouldn't actually depend on the
> > externalized connectors. I'm not sure what PyFlink does with it, but if
> > belongs to the connector code, that code should also be moved to the
> > individual connector repo. If it's just a generic test, we could consider
> > creating a generic test against released connector versions to determine
> > compatibility.
> >
> > I'm curious about the opinions of others as well.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com> wrote:
> >
> > > I have an issue here that needs to upgrade commons-collections[1] (this
> > is
> > > an example), but PR ci fails because flink-python test cases depend on
> > > flink-sql-connector-kafka, but kafka-sql-connector is a small jar, does
> > not
> > > include this dependency, so flink ci cause exception[2]. Current my
> > > solution is [3]. But even if this PR is done, the upgrade of flink still
> > > requires kafka-connector released.
> > >
> > > This issue leads to deeper problems. Although the connectors have been
> > > externalized, many UTs of flink-python depend on these connectors, and a
> > > basic agreement of externalized connectors is that other dependencies
> > > cannot be introduced explicitly, which means the externalized connectors
> > > use dependencies inherited from flink. In this way, when flink main
> > > upgrades some dependencies, it is easy to fail when executing
> > flink-python
> > > test cases，because flink no longer has this class, and the connector does
> > > not contain it. It's circular problem.
> > >
> > > Unless, the connector self-consistently includes all dependencies, which
> > is
> > > uncontrollable.
> > > (only a few connectors include all jars in shade phase)
> > >
> > > In short, the current flink-python module's dependencies on the connector
> > > leads to an incomplete process of externalization and decoupling, which
> > > will lead to circular dependencies when flink upgrade or change some
> > > dependencies.
> > >
> > > I don't know if I made it clear. I hope to get everyone's opinions on
> > what
> > > better solutions we should adopt for similar problems in the future.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-30274
> > > [2]
> > >
> > >
> > https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
> > >
> > >
> > >
> > https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> > > [3] https://github.com/apache/flink-connector-kafka/pull/38
> > >
> > > Best Regards,
> > > Ran Tao
> > > https://github.com/chucheng92
> > >
> >
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Ran Tao <ch...@gmail.com>.
@Martijn
thanks for clear explanations.
If we follow the line you specified (Connectors shouldn't rely on
dependencies that may or may not be
available in Flink itself)
It seems that we should add a certain dependency if we need(such as
commons-io, commons-collection) in connector pom explicitly.
And bundle it in sql-connector uber jar.
Then there is only one thing left that we need to make flink-python test
not depend on the released flink-connector.
Maybe we should check it out and decouple it like you suggested.
Best Regards,
Ran Tao
https://github.com/chucheng92
Martijn Visser <ma...@apache.org> 于2023年7月3日周一 22:06写道：
> Hi Ran Tao,
>
> Thanks for opening this topic. I think there's a couple of things at hand:
> 1. Connectors shouldn't rely on dependencies that may or may not be
> available in Flink itself, like we've seen with flink-shaded. That avoids a
> tight coupling between Flink and connectors, which is exactly what we try
> to avoid.
> 2. When following that line, that would also be applicable for things like
> commons-collections and commons-io. If a connector wants to use them, it
> should make sure that it bundles those artifacts itself.
> 3. While flink-python now fails the CI, it shouldn't actually depend on the
> externalized connectors. I'm not sure what PyFlink does with it, but if
> belongs to the connector code, that code should also be moved to the
> individual connector repo. If it's just a generic test, we could consider
> creating a generic test against released connector versions to determine
> compatibility.
>
> I'm curious about the opinions of others as well.
>
> Best regards,
>
> Martijn
>
> On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com> wrote:
>
> > I have an issue here that needs to upgrade commons-collections[1] (this
> is
> > an example), but PR ci fails because flink-python test cases depend on
> > flink-sql-connector-kafka, but kafka-sql-connector is a small jar, does
> not
> > include this dependency, so flink ci cause exception[2]. Current my
> > solution is [3]. But even if this PR is done, the upgrade of flink still
> > requires kafka-connector released.
> >
> > This issue leads to deeper problems. Although the connectors have been
> > externalized, many UTs of flink-python depend on these connectors, and a
> > basic agreement of externalized connectors is that other dependencies
> > cannot be introduced explicitly, which means the externalized connectors
> > use dependencies inherited from flink. In this way, when flink main
> > upgrades some dependencies, it is easy to fail when executing
> flink-python
> > test cases，because flink no longer has this class, and the connector does
> > not contain it. It's circular problem.
> >
> > Unless, the connector self-consistently includes all dependencies, which
> is
> > uncontrollable.
> > (only a few connectors include all jars in shade phase)
> >
> > In short, the current flink-python module's dependencies on the connector
> > leads to an incomplete process of externalization and decoupling, which
> > will lead to circular dependencies when flink upgrade or change some
> > dependencies.
> >
> > I don't know if I made it clear. I hope to get everyone's opinions on
> what
> > better solutions we should adopt for similar problems in the future.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-30274
> > [2]
> >
> >
> https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
> >
> >
> >
> https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> > [3] https://github.com/apache/flink-connector-kafka/pull/38
> >
> > Best Regards,
> > Ran Tao
> > https://github.com/chucheng92
> >
>
Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems
Posted by Martijn Visser <ma...@apache.org>.
Hi Ran Tao,
Thanks for opening this topic. I think there's a couple of things at hand:
1. Connectors shouldn't rely on dependencies that may or may not be
available in Flink itself, like we've seen with flink-shaded. That avoids a
tight coupling between Flink and connectors, which is exactly what we try
to avoid.
2. When following that line, that would also be applicable for things like
commons-collections and commons-io. If a connector wants to use them, it
should make sure that it bundles those artifacts itself.
3. While flink-python now fails the CI, it shouldn't actually depend on the
externalized connectors. I'm not sure what PyFlink does with it, but if
belongs to the connector code, that code should also be moved to the
individual connector repo. If it's just a generic test, we could consider
creating a generic test against released connector versions to determine
compatibility.
I'm curious about the opinions of others as well.
Best regards,
Martijn
On Mon, Jul 3, 2023 at 3:37 PM Ran Tao <ch...@gmail.com> wrote:
> I have an issue here that needs to upgrade commons-collections[1] (this is
> an example), but PR ci fails because flink-python test cases depend on
> flink-sql-connector-kafka, but kafka-sql-connector is a small jar, does not
> include this dependency, so flink ci cause exception[2]. Current my
> solution is [3]. But even if this PR is done, the upgrade of flink still
> requires kafka-connector released.
>
> This issue leads to deeper problems. Although the connectors have been
> externalized, many UTs of flink-python depend on these connectors, and a
> basic agreement of externalized connectors is that other dependencies
> cannot be introduced explicitly, which means the externalized connectors
> use dependencies inherited from flink. In this way, when flink main
> upgrades some dependencies, it is easy to fail when executing flink-python
> test cases，because flink no longer has this class, and the connector does
> not contain it. It's circular problem.
>
> Unless, the connector self-consistently includes all dependencies, which is
> uncontrollable.
> (only a few connectors include all jars in shade phase)
>
> In short, the current flink-python module's dependencies on the connector
> leads to an incomplete process of externalization and decoupling, which
> will lead to circular dependencies when flink upgrade or change some
> dependencies.
>
> I don't know if I made it clear. I hope to get everyone's opinions on what
> better solutions we should adopt for similar problems in the future.
>
> [1] https://issues.apache.org/jira/browse/FLINK-30274
> [2]
>
> https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
>
>
> https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> [3] https://github.com/apache/flink-connector-kafka/pull/38
>
> Best Regards,
> Ran Tao
> https://github.com/chucheng92
>