You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by João Boto <es...@apache.org> on 2024/05/03 12:25:31 UTC

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

Hi Muhammet,

While I generally agree, given our current usage, I'm struggling to discern any clear advantage. We already have abstract implementations that cover all necessary interfaces and offer essential functionality, complemented by a robust set of reusable tests to streamline implementation.

With this established infrastructure in place, coupled with the added import overhead of introducing another module, I find it difficult to identify any distinct benefits at this point.

Best

On 2024/04/26 02:18:52 Muhammet Orazov wrote:
> Hey João,
> 
> Thanks for FLIP proposal!
> 
> Since proposal is to introduce modules, would it make sense
> to have another module for APIs (flink-jdbc-connector-api)?
> 
> For this I would suggest to move all public interfaces (e.g,
> JdbcRowConverter, JdbcConnectionProvider). And even convert
> some classes into interface with their default implementations,
> for example, JdbcSink, JdbcConnectionOptions.
> 
> This way users would have clear interfaces to build their own
> JDBC based Flink connectors.
> 
> Here I am not suggesting to introduce new interfaces, only
> suggest also to separate the API from the core implementation.
> 
> What do you think?
> 
> Best,
> Muhammet
> 
> 
> On 2024-04-25 08:54, Joao Boto wrote:
> > Hi all,
> > 
> > I'd like to start a discussion on FLIP-449: Reorganization of
> > flink-connector-jdbc [1].
> > As Flink continues to evolve, we've noticed an increasing level of
> > complexity within the JDBC connector.
> > The proposed solution is to address this complexity by separating the 
> > core
> > functionality from individual database components, thereby streamlining 
> > the
> > structure into distinct modules.
> > 
> > Looking forward to your feedback and suggestions, thanks.
> > Best regards,
> > Joao Boto
> > 
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
> 

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

Posted by lo...@ververica.com.INVALID.
Thanks joao for your replies!

I also saw the latest PR that allows properties to be specified.

Thank for adding the pain points as well, that clarifies a lot.
On May 7, 2024 at 09:50 +0200, Muhammet Orazov <mo...@morazow.com.invalid>, wrote:
> Thanks João for pointing it out. I didn't know about the PR, I am going
> to check it.
>
> Best,
> Muhammet
>
>
> On 2024-05-06 14:45, João Boto wrote:
> > Hi Muhammet,
> >
> > Have you had a chance to review the recently merged pull request [1]?
> > We've introduced a new feature allowing users to include ad hoc
> > configurations in the 'JdbcConnectionOptions' class.
> > ```
> > new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
> > .withUrl(FakeDBUtils.TEST_DB_URL)
> > .withProperty("keyA", "valueA")
> > .build();
> > ```
> >
> > This provides flexibility by enabling users to specify additional
> > configuration parameters dynamically.
> >
> > [1] https://github.com/apache/flink-connector-jdbc/pull/115/files
> >
> > Best
> >
> > On 2024/05/06 07:34:06 Muhammet Orazov wrote:
> > > > Morning João,
> > > >
> > > > Recently we had a case where the JDBC drivers authentication was
> > > > different than username&password authentication. For it to work,
> > > > certain
> > > > hacks required, there interface would have been helpful.
> > > >
> > > > But I agree maybe the interface module separation is not required at
> > > > the
> > > > moment.
> > > >
> > > > Thanks for your efforts!
> > > >
> > > > Best,
> > > > Muhammet
> > > >
> > > >
> > > > On 2024-05-03 12:25, João Boto wrote:
> > > > > > Hi Muhammet,
> > > > > >
> > > > > > While I generally agree, given our current usage, I'm struggling to
> > > > > > discern any clear advantage. We already have abstract implementations
> > > > > > that cover all necessary interfaces and offer essential functionality,
> > > > > > complemented by a robust set of reusable tests to streamline
> > > > > > implementation.
> > > > > >
> > > > > > With this established infrastructure in place, coupled with the added
> > > > > > import overhead of introducing another module, I find it difficult to
> > > > > > identify any distinct benefits at this point.
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > On 2024/04/26 02:18:52 Muhammet Orazov wrote:
> > > > > > >> Hey João,
> > > > > > >>
> > > > > > >> Thanks for FLIP proposal!
> > > > > > >>
> > > > > > >> Since proposal is to introduce modules, would it make sense
> > > > > > >> to have another module for APIs (flink-jdbc-connector-api)?
> > > > > > >>
> > > > > > >> For this I would suggest to move all public interfaces (e.g,
> > > > > > >> JdbcRowConverter, JdbcConnectionProvider). And even convert
> > > > > > >> some classes into interface with their default implementations,
> > > > > > >> for example, JdbcSink, JdbcConnectionOptions.
> > > > > > >>
> > > > > > >> This way users would have clear interfaces to build their own
> > > > > > >> JDBC based Flink connectors.
> > > > > > >>
> > > > > > >> Here I am not suggesting to introduce new interfaces, only
> > > > > > >> suggest also to separate the API from the core implementation.
> > > > > > >>
> > > > > > >> What do you think?
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Muhammet
> > > > > > >>
> > > > > > >>
> > > > > > >> On 2024-04-25 08:54, Joao Boto wrote:
> > > > > > > >> > Hi all,
> > > > > > > >> >
> > > > > > > >> > I'd like to start a discussion on FLIP-449: Reorganization of
> > > > > > > >> > flink-connector-jdbc [1].
> > > > > > > >> > As Flink continues to evolve, we've noticed an increasing level of
> > > > > > > >> > complexity within the JDBC connector.
> > > > > > > >> > The proposed solution is to address this complexity by separating the
> > > > > > > >> > core
> > > > > > > >> > functionality from individual database components, thereby streamlining
> > > > > > > >> > the
> > > > > > > >> > structure into distinct modules.
> > > > > > > >> >
> > > > > > > >> > Looking forward to your feedback and suggestions, thanks.
> > > > > > > >> > Best regards,
> > > > > > > >> > Joao Boto
> > > > > > > >> >
> > > > > > > >> > [1]
> > > > > > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
> > > > > > >>
> > > >

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

Posted by Muhammet Orazov <mo...@morazow.com.INVALID>.
Thanks João for pointing it out. I didn't know about the PR, I am going 
to check it.

Best,
Muhammet


On 2024-05-06 14:45, João Boto wrote:
> Hi Muhammet,
> 
> Have you had a chance to review the recently merged pull request [1]?
> We've introduced a new feature allowing users to include ad hoc 
> configurations in the 'JdbcConnectionOptions' class.
> ```
>  new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
>                         .withUrl(FakeDBUtils.TEST_DB_URL)
>                         .withProperty("keyA", "valueA")
>                         .build();
> ```
> 
> This provides flexibility by enabling users to specify additional 
> configuration parameters dynamically.
> 
> [1] https://github.com/apache/flink-connector-jdbc/pull/115/files
> 
> Best
> 
> On 2024/05/06 07:34:06 Muhammet Orazov wrote:
>> Morning João,
>> 
>> Recently we had a case where the JDBC drivers authentication was
>> different than username&password authentication. For it to work, 
>> certain
>> hacks required, there interface would have been helpful.
>> 
>> But I agree maybe the interface module separation is not required at 
>> the
>> moment.
>> 
>> Thanks for your efforts!
>> 
>> Best,
>> Muhammet
>> 
>> 
>> On 2024-05-03 12:25, João Boto wrote:
>> > Hi Muhammet,
>> >
>> > While I generally agree, given our current usage, I'm struggling to
>> > discern any clear advantage. We already have abstract implementations
>> > that cover all necessary interfaces and offer essential functionality,
>> > complemented by a robust set of reusable tests to streamline
>> > implementation.
>> >
>> > With this established infrastructure in place, coupled with the added
>> > import overhead of introducing another module, I find it difficult to
>> > identify any distinct benefits at this point.
>> >
>> > Best
>> >
>> > On 2024/04/26 02:18:52 Muhammet Orazov wrote:
>> >> Hey João,
>> >>
>> >> Thanks for FLIP proposal!
>> >>
>> >> Since proposal is to introduce modules, would it make sense
>> >> to have another module for APIs (flink-jdbc-connector-api)?
>> >>
>> >> For this I would suggest to move all public interfaces (e.g,
>> >> JdbcRowConverter, JdbcConnectionProvider). And even convert
>> >> some classes into interface with their default implementations,
>> >> for example, JdbcSink, JdbcConnectionOptions.
>> >>
>> >> This way users would have clear interfaces to build their own
>> >> JDBC based Flink connectors.
>> >>
>> >> Here I am not suggesting to introduce new interfaces, only
>> >> suggest also to separate the API from the core implementation.
>> >>
>> >> What do you think?
>> >>
>> >> Best,
>> >> Muhammet
>> >>
>> >>
>> >> On 2024-04-25 08:54, Joao Boto wrote:
>> >> > Hi all,
>> >> >
>> >> > I'd like to start a discussion on FLIP-449: Reorganization of
>> >> > flink-connector-jdbc [1].
>> >> > As Flink continues to evolve, we've noticed an increasing level of
>> >> > complexity within the JDBC connector.
>> >> > The proposed solution is to address this complexity by separating the
>> >> > core
>> >> > functionality from individual database components, thereby streamlining
>> >> > the
>> >> > structure into distinct modules.
>> >> >
>> >> > Looking forward to your feedback and suggestions, thanks.
>> >> > Best regards,
>> >> > Joao Boto
>> >> >
>> >> > [1]
>> >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
>> >>
>> 

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

Posted by João Boto <es...@apache.org>.
Hi Muhammet,

Have you had a chance to review the recently merged pull request [1]? 
We've introduced a new feature allowing users to include ad hoc configurations in the 'JdbcConnectionOptions' class.
```
 new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
                        .withUrl(FakeDBUtils.TEST_DB_URL)
                        .withProperty("keyA", "valueA")
                        .build();
```

This provides flexibility by enabling users to specify additional configuration parameters dynamically. 

[1] https://github.com/apache/flink-connector-jdbc/pull/115/files

Best

On 2024/05/06 07:34:06 Muhammet Orazov wrote:
> Morning João,
> 
> Recently we had a case where the JDBC drivers authentication was 
> different than username&password authentication. For it to work, certain 
> hacks required, there interface would have been helpful.
> 
> But I agree maybe the interface module separation is not required at the 
> moment.
> 
> Thanks for your efforts!
> 
> Best,
> Muhammet
> 
> 
> On 2024-05-03 12:25, João Boto wrote:
> > Hi Muhammet,
> > 
> > While I generally agree, given our current usage, I'm struggling to 
> > discern any clear advantage. We already have abstract implementations 
> > that cover all necessary interfaces and offer essential functionality, 
> > complemented by a robust set of reusable tests to streamline 
> > implementation.
> > 
> > With this established infrastructure in place, coupled with the added 
> > import overhead of introducing another module, I find it difficult to 
> > identify any distinct benefits at this point.
> > 
> > Best
> > 
> > On 2024/04/26 02:18:52 Muhammet Orazov wrote:
> >> Hey João,
> >> 
> >> Thanks for FLIP proposal!
> >> 
> >> Since proposal is to introduce modules, would it make sense
> >> to have another module for APIs (flink-jdbc-connector-api)?
> >> 
> >> For this I would suggest to move all public interfaces (e.g,
> >> JdbcRowConverter, JdbcConnectionProvider). And even convert
> >> some classes into interface with their default implementations,
> >> for example, JdbcSink, JdbcConnectionOptions.
> >> 
> >> This way users would have clear interfaces to build their own
> >> JDBC based Flink connectors.
> >> 
> >> Here I am not suggesting to introduce new interfaces, only
> >> suggest also to separate the API from the core implementation.
> >> 
> >> What do you think?
> >> 
> >> Best,
> >> Muhammet
> >> 
> >> 
> >> On 2024-04-25 08:54, Joao Boto wrote:
> >> > Hi all,
> >> >
> >> > I'd like to start a discussion on FLIP-449: Reorganization of
> >> > flink-connector-jdbc [1].
> >> > As Flink continues to evolve, we've noticed an increasing level of
> >> > complexity within the JDBC connector.
> >> > The proposed solution is to address this complexity by separating the
> >> > core
> >> > functionality from individual database components, thereby streamlining
> >> > the
> >> > structure into distinct modules.
> >> >
> >> > Looking forward to your feedback and suggestions, thanks.
> >> > Best regards,
> >> > Joao Boto
> >> >
> >> > [1]
> >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
> >> 
> 

Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc

Posted by Muhammet Orazov <mo...@morazow.com.INVALID>.
Morning João,

Recently we had a case where the JDBC drivers authentication was 
different than username&password authentication. For it to work, certain 
hacks required, there interface would have been helpful.

But I agree maybe the interface module separation is not required at the 
moment.

Thanks for your efforts!

Best,
Muhammet


On 2024-05-03 12:25, João Boto wrote:
> Hi Muhammet,
> 
> While I generally agree, given our current usage, I'm struggling to 
> discern any clear advantage. We already have abstract implementations 
> that cover all necessary interfaces and offer essential functionality, 
> complemented by a robust set of reusable tests to streamline 
> implementation.
> 
> With this established infrastructure in place, coupled with the added 
> import overhead of introducing another module, I find it difficult to 
> identify any distinct benefits at this point.
> 
> Best
> 
> On 2024/04/26 02:18:52 Muhammet Orazov wrote:
>> Hey João,
>> 
>> Thanks for FLIP proposal!
>> 
>> Since proposal is to introduce modules, would it make sense
>> to have another module for APIs (flink-jdbc-connector-api)?
>> 
>> For this I would suggest to move all public interfaces (e.g,
>> JdbcRowConverter, JdbcConnectionProvider). And even convert
>> some classes into interface with their default implementations,
>> for example, JdbcSink, JdbcConnectionOptions.
>> 
>> This way users would have clear interfaces to build their own
>> JDBC based Flink connectors.
>> 
>> Here I am not suggesting to introduce new interfaces, only
>> suggest also to separate the API from the core implementation.
>> 
>> What do you think?
>> 
>> Best,
>> Muhammet
>> 
>> 
>> On 2024-04-25 08:54, Joao Boto wrote:
>> > Hi all,
>> >
>> > I'd like to start a discussion on FLIP-449: Reorganization of
>> > flink-connector-jdbc [1].
>> > As Flink continues to evolve, we've noticed an increasing level of
>> > complexity within the JDBC connector.
>> > The proposed solution is to address this complexity by separating the
>> > core
>> > functionality from individual database components, thereby streamlining
>> > the
>> > structure into distinct modules.
>> >
>> > Looking forward to your feedback and suggestions, thanks.
>> > Best regards,
>> > Joao Boto
>> >
>> > [1]
>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
>>