You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sebastián Magrí <se...@gmail.com> on 2021/02/02 20:15:43 UTC
Re: Conflicts between the JDBC and postgresql-cdc SQL connectors
The root of the previous error seemed to be the flink version the connector
was compiled for. I've tried compiling my own postgresql-cdc connector, but
still have some issues with dependencies.
On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí <se...@gmail.com> wrote:
> Applied that parameter and that seems to get me some progress here.
>
> I still get the shade overlapping classes warning, but I get the
> PostgreSQLTableFactory in the merged table.factories.Factory service file.
>
> However, now on runtime the application fails to find the debezium source
> function class coming down to this error:
>
> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
> Cannot load user class:
> com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
> ClassLoader info: URL ClassLoader:
> Class not resolvable through given classloader.
>
> The class is indeed in jar, though.
>
> Any thougths?
>
> On Thu, 28 Jan 2021 at 09:57, Jark Wu <im...@gmail.com> wrote:
>
>> Hi Sebastián,
>>
>> Could you try to add combine.children="append" attribute to the
>> transformers configuration?
>> You can also see the full shade plugin configuration here [1].
>>
>> Best,
>> Jark
>>
>> [1]:
>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources
>>
>> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <se...@gmail.com>
>> wrote:
>>
>>> Hi Jark!
>>>
>>> Please find the full pom file attached.
>>>
>>> Best Regards,
>>>
>>> On Thu, 28 Jan 2021 at 03:21, Jark Wu <im...@gmail.com> wrote:
>>>
>>>> Hi Sebastián,
>>>>
>>>> I think Dawid is right.
>>>>
>>>> Could you share the pom file? I also tried to
>>>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and
>>>> the Factory file contains
>>>>
>>>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory
>>>>
>>>>
>>>> Best,
>>>> Jark
>>>>
>>>>
>>>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <se...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks a lot for looking into it Dawid,
>>>>>
>>>>> In the
>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
>>>>> file I only see
>>>>>
>>>>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>>>>>
>>>>> Even after applying the ServicesResourceTransformer.
>>>>>
>>>>>
>>>>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <dw...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Unfortunately I am not familiar with the packaging of
>>>>>> flink-connector-postgres-cdc. Maybe @Jark could help here?
>>>>>>
>>>>>> However, I think the problem that you cannot find the connector is
>>>>>> caused because of lack of entry in the resulting Manifest file. If there
>>>>>> are overlapping classes maven does not exclude whole dependencies, but
>>>>>> rather picks the overlapping class from one of the two. Could you check if
>>>>>> you see entries for all tables in
>>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>>>>>>
>>>>>> If not, you could try applying the ServicesResourceTransformer[1]
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Dawid
>>>>>>
>>>>>> [1]
>>>>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>>>>>> On 26/01/2021 12:29, Sebastián Magrí wrote:
>>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> I've reported an issue with the postgresql-cdc connector apparently
>>>>>> caused by the maven shade plugin excluding either the JDBC connector or the
>>>>>> cdc connector due to overlapping classes. The issue for reference is here:
>>>>>>
>>>>>> https://github.com/ververica/flink-cdc-connectors/issues/90
>>>>>>
>>>>>> In the meantime, however, I've been trying to figure out if I can set
>>>>>> up an exclusion rule to fix this in my pom.xml file, without success.
>>>>>>
>>>>>> The `org.postgresql:postgresql` dependency is being added manually by
>>>>>> me to have a sink on a postgresql table and injected by the cdc connector
>>>>>> seemingly via its debezium connector dependency.
>>>>>>
>>>>>> Any guidance or hints I could follow would be really appreciated.
>>>>>>
>>>>>> --
>>>>>> Sebastián Ramírez Magrí
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Sebastián Ramírez Magrí
>>>>>
>>>>
>>>
>>> --
>>> Sebastián Ramírez Magrí
>>>
>>
>
> --
> Sebastián Ramírez Magrí
>
--
Sebastián Ramírez Magrí
Re: Conflicts between the JDBC and postgresql-cdc SQL connectors
Posted by Robert Metzger <rm...@apache.org>.
I don't know what your dependency issue is (post it here if you want
help!), but I generally recommend using mvn dependency:tree to debug
version clashes (and then pin or exclude versions)
On Tue, Feb 2, 2021 at 9:23 PM Sebastián Magrí <se...@gmail.com> wrote:
> The root of the previous error seemed to be the flink version the
> connector was compiled for. I've tried compiling my own postgresql-cdc
> connector, but still have some issues with dependencies.
>
> On Thu, 28 Jan 2021 at 11:24, Sebastián Magrí <se...@gmail.com>
> wrote:
>
>> Applied that parameter and that seems to get me some progress here.
>>
>> I still get the shade overlapping classes warning, but I get the
>> PostgreSQLTableFactory in the merged table.factories.Factory service file.
>>
>> However, now on runtime the application fails to find the debezium source
>> function class coming down to this error:
>>
>> Caused by: org.apache.flink.streaming.runtime.tasks.StreamTaskException:
>> Cannot load user class:
>> com.alibaba.ververica.cdc.debezium.DebeziumSourceFunction
>> ClassLoader info: URL ClassLoader:
>> Class not resolvable through given classloader.
>>
>> The class is indeed in jar, though.
>>
>> Any thougths?
>>
>> On Thu, 28 Jan 2021 at 09:57, Jark Wu <im...@gmail.com> wrote:
>>
>>> Hi Sebastián,
>>>
>>> Could you try to add combine.children="append" attribute to the
>>> transformers configuration?
>>> You can also see the full shade plugin configuration here [1].
>>>
>>> Best,
>>> Jark
>>>
>>> [1]:
>>> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/#transform-table-connectorformat-resources
>>>
>>> On Thu, 28 Jan 2021 at 17:28, Sebastián Magrí <se...@gmail.com>
>>> wrote:
>>>
>>>> Hi Jark!
>>>>
>>>> Please find the full pom file attached.
>>>>
>>>> Best Regards,
>>>>
>>>> On Thu, 28 Jan 2021 at 03:21, Jark Wu <im...@gmail.com> wrote:
>>>>
>>>>> Hi Sebastián,
>>>>>
>>>>> I think Dawid is right.
>>>>>
>>>>> Could you share the pom file? I also tried to
>>>>> package flink-connector-postgres-cdc with ServicesResourceTransformer, and
>>>>> the Factory file contains
>>>>>
>>>>> com.alibaba.ververica.cdc.connectors.postgres.table.PostgreSQLTableFactory
>>>>>
>>>>>
>>>>> Best,
>>>>> Jark
>>>>>
>>>>>
>>>>> On Tue, 26 Jan 2021 at 21:17, Sebastián Magrí <se...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks a lot for looking into it Dawid,
>>>>>>
>>>>>> In the
>>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory
>>>>>> file I only see
>>>>>>
>>>>>> org.apache.flink.connector.jdbc.table.JdbcDynamicTableFactory
>>>>>>
>>>>>> Even after applying the ServicesResourceTransformer.
>>>>>>
>>>>>>
>>>>>> On Tue, 26 Jan 2021 at 11:58, Dawid Wysakowicz <
>>>>>> dwysakowicz@apache.org> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Unfortunately I am not familiar with the packaging of
>>>>>>> flink-connector-postgres-cdc. Maybe @Jark could help here?
>>>>>>>
>>>>>>> However, I think the problem that you cannot find the connector is
>>>>>>> caused because of lack of entry in the resulting Manifest file. If there
>>>>>>> are overlapping classes maven does not exclude whole dependencies, but
>>>>>>> rather picks the overlapping class from one of the two. Could you check if
>>>>>>> you see entries for all tables in
>>>>>>> src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory.
>>>>>>>
>>>>>>> If not, you could try applying the ServicesResourceTransformer[1]
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> Dawid
>>>>>>>
>>>>>>> [1]
>>>>>>> https://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html#ServicesResourceTransformer
>>>>>>> On 26/01/2021 12:29, Sebastián Magrí wrote:
>>>>>>>
>>>>>>> Hi!
>>>>>>>
>>>>>>> I've reported an issue with the postgresql-cdc connector apparently
>>>>>>> caused by the maven shade plugin excluding either the JDBC connector or the
>>>>>>> cdc connector due to overlapping classes. The issue for reference is here:
>>>>>>>
>>>>>>> https://github.com/ververica/flink-cdc-connectors/issues/90
>>>>>>>
>>>>>>> In the meantime, however, I've been trying to figure out if I can
>>>>>>> set up an exclusion rule to fix this in my pom.xml file, without success.
>>>>>>>
>>>>>>> The `org.postgresql:postgresql` dependency is being added manually
>>>>>>> by me to have a sink on a postgresql table and injected by the cdc
>>>>>>> connector seemingly via its debezium connector dependency.
>>>>>>>
>>>>>>> Any guidance or hints I could follow would be really appreciated.
>>>>>>>
>>>>>>> --
>>>>>>> Sebastián Ramírez Magrí
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Sebastián Ramírez Magrí
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Sebastián Ramírez Magrí
>>>>
>>>
>>
>> --
>> Sebastián Ramírez Magrí
>>
>
>
> --
> Sebastián Ramírez Magrí
>