You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by João Boto <es...@apache.org> on 2024/05/03 12:25:31 UTC
Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc
Hi Muhammet,
While I generally agree, given our current usage, I'm struggling to discern any clear advantage. We already have abstract implementations that cover all necessary interfaces and offer essential functionality, complemented by a robust set of reusable tests to streamline implementation.
With this established infrastructure in place, coupled with the added import overhead of introducing another module, I find it difficult to identify any distinct benefits at this point.
Best
On 2024/04/26 02:18:52 Muhammet Orazov wrote:
> Hey João,
>
> Thanks for FLIP proposal!
>
> Since proposal is to introduce modules, would it make sense
> to have another module for APIs (flink-jdbc-connector-api)?
>
> For this I would suggest to move all public interfaces (e.g,
> JdbcRowConverter, JdbcConnectionProvider). And even convert
> some classes into interface with their default implementations,
> for example, JdbcSink, JdbcConnectionOptions.
>
> This way users would have clear interfaces to build their own
> JDBC based Flink connectors.
>
> Here I am not suggesting to introduce new interfaces, only
> suggest also to separate the API from the core implementation.
>
> What do you think?
>
> Best,
> Muhammet
>
>
> On 2024-04-25 08:54, Joao Boto wrote:
> > Hi all,
> >
> > I'd like to start a discussion on FLIP-449: Reorganization of
> > flink-connector-jdbc [1].
> > As Flink continues to evolve, we've noticed an increasing level of
> > complexity within the JDBC connector.
> > The proposed solution is to address this complexity by separating the
> > core
> > functionality from individual database components, thereby streamlining
> > the
> > structure into distinct modules.
> >
> > Looking forward to your feedback and suggestions, thanks.
> > Best regards,
> > Joao Boto
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
>
Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc
Posted by lo...@ververica.com.INVALID.
Thanks joao for your replies!
I also saw the latest PR that allows properties to be specified.
Thank for adding the pain points as well, that clarifies a lot.
On May 7, 2024 at 09:50 +0200, Muhammet Orazov <mo...@morazow.com.invalid>, wrote:
> Thanks João for pointing it out. I didn't know about the PR, I am going
> to check it.
>
> Best,
> Muhammet
>
>
> On 2024-05-06 14:45, João Boto wrote:
> > Hi Muhammet,
> >
> > Have you had a chance to review the recently merged pull request [1]?
> > We've introduced a new feature allowing users to include ad hoc
> > configurations in the 'JdbcConnectionOptions' class.
> > ```
> > new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
> > .withUrl(FakeDBUtils.TEST_DB_URL)
> > .withProperty("keyA", "valueA")
> > .build();
> > ```
> >
> > This provides flexibility by enabling users to specify additional
> > configuration parameters dynamically.
> >
> > [1] https://github.com/apache/flink-connector-jdbc/pull/115/files
> >
> > Best
> >
> > On 2024/05/06 07:34:06 Muhammet Orazov wrote:
> > > > Morning João,
> > > >
> > > > Recently we had a case where the JDBC drivers authentication was
> > > > different than username&password authentication. For it to work,
> > > > certain
> > > > hacks required, there interface would have been helpful.
> > > >
> > > > But I agree maybe the interface module separation is not required at
> > > > the
> > > > moment.
> > > >
> > > > Thanks for your efforts!
> > > >
> > > > Best,
> > > > Muhammet
> > > >
> > > >
> > > > On 2024-05-03 12:25, João Boto wrote:
> > > > > > Hi Muhammet,
> > > > > >
> > > > > > While I generally agree, given our current usage, I'm struggling to
> > > > > > discern any clear advantage. We already have abstract implementations
> > > > > > that cover all necessary interfaces and offer essential functionality,
> > > > > > complemented by a robust set of reusable tests to streamline
> > > > > > implementation.
> > > > > >
> > > > > > With this established infrastructure in place, coupled with the added
> > > > > > import overhead of introducing another module, I find it difficult to
> > > > > > identify any distinct benefits at this point.
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > On 2024/04/26 02:18:52 Muhammet Orazov wrote:
> > > > > > >> Hey João,
> > > > > > >>
> > > > > > >> Thanks for FLIP proposal!
> > > > > > >>
> > > > > > >> Since proposal is to introduce modules, would it make sense
> > > > > > >> to have another module for APIs (flink-jdbc-connector-api)?
> > > > > > >>
> > > > > > >> For this I would suggest to move all public interfaces (e.g,
> > > > > > >> JdbcRowConverter, JdbcConnectionProvider). And even convert
> > > > > > >> some classes into interface with their default implementations,
> > > > > > >> for example, JdbcSink, JdbcConnectionOptions.
> > > > > > >>
> > > > > > >> This way users would have clear interfaces to build their own
> > > > > > >> JDBC based Flink connectors.
> > > > > > >>
> > > > > > >> Here I am not suggesting to introduce new interfaces, only
> > > > > > >> suggest also to separate the API from the core implementation.
> > > > > > >>
> > > > > > >> What do you think?
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Muhammet
> > > > > > >>
> > > > > > >>
> > > > > > >> On 2024-04-25 08:54, Joao Boto wrote:
> > > > > > > >> > Hi all,
> > > > > > > >> >
> > > > > > > >> > I'd like to start a discussion on FLIP-449: Reorganization of
> > > > > > > >> > flink-connector-jdbc [1].
> > > > > > > >> > As Flink continues to evolve, we've noticed an increasing level of
> > > > > > > >> > complexity within the JDBC connector.
> > > > > > > >> > The proposed solution is to address this complexity by separating the
> > > > > > > >> > core
> > > > > > > >> > functionality from individual database components, thereby streamlining
> > > > > > > >> > the
> > > > > > > >> > structure into distinct modules.
> > > > > > > >> >
> > > > > > > >> > Looking forward to your feedback and suggestions, thanks.
> > > > > > > >> > Best regards,
> > > > > > > >> > Joao Boto
> > > > > > > >> >
> > > > > > > >> > [1]
> > > > > > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
> > > > > > >>
> > > >
Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc
Posted by Muhammet Orazov <mo...@morazow.com.INVALID>.
Thanks João for pointing it out. I didn't know about the PR, I am going
to check it.
Best,
Muhammet
On 2024-05-06 14:45, João Boto wrote:
> Hi Muhammet,
>
> Have you had a chance to review the recently merged pull request [1]?
> We've introduced a new feature allowing users to include ad hoc
> configurations in the 'JdbcConnectionOptions' class.
> ```
> new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
> .withUrl(FakeDBUtils.TEST_DB_URL)
> .withProperty("keyA", "valueA")
> .build();
> ```
>
> This provides flexibility by enabling users to specify additional
> configuration parameters dynamically.
>
> [1] https://github.com/apache/flink-connector-jdbc/pull/115/files
>
> Best
>
> On 2024/05/06 07:34:06 Muhammet Orazov wrote:
>> Morning João,
>>
>> Recently we had a case where the JDBC drivers authentication was
>> different than username&password authentication. For it to work,
>> certain
>> hacks required, there interface would have been helpful.
>>
>> But I agree maybe the interface module separation is not required at
>> the
>> moment.
>>
>> Thanks for your efforts!
>>
>> Best,
>> Muhammet
>>
>>
>> On 2024-05-03 12:25, João Boto wrote:
>> > Hi Muhammet,
>> >
>> > While I generally agree, given our current usage, I'm struggling to
>> > discern any clear advantage. We already have abstract implementations
>> > that cover all necessary interfaces and offer essential functionality,
>> > complemented by a robust set of reusable tests to streamline
>> > implementation.
>> >
>> > With this established infrastructure in place, coupled with the added
>> > import overhead of introducing another module, I find it difficult to
>> > identify any distinct benefits at this point.
>> >
>> > Best
>> >
>> > On 2024/04/26 02:18:52 Muhammet Orazov wrote:
>> >> Hey João,
>> >>
>> >> Thanks for FLIP proposal!
>> >>
>> >> Since proposal is to introduce modules, would it make sense
>> >> to have another module for APIs (flink-jdbc-connector-api)?
>> >>
>> >> For this I would suggest to move all public interfaces (e.g,
>> >> JdbcRowConverter, JdbcConnectionProvider). And even convert
>> >> some classes into interface with their default implementations,
>> >> for example, JdbcSink, JdbcConnectionOptions.
>> >>
>> >> This way users would have clear interfaces to build their own
>> >> JDBC based Flink connectors.
>> >>
>> >> Here I am not suggesting to introduce new interfaces, only
>> >> suggest also to separate the API from the core implementation.
>> >>
>> >> What do you think?
>> >>
>> >> Best,
>> >> Muhammet
>> >>
>> >>
>> >> On 2024-04-25 08:54, Joao Boto wrote:
>> >> > Hi all,
>> >> >
>> >> > I'd like to start a discussion on FLIP-449: Reorganization of
>> >> > flink-connector-jdbc [1].
>> >> > As Flink continues to evolve, we've noticed an increasing level of
>> >> > complexity within the JDBC connector.
>> >> > The proposed solution is to address this complexity by separating the
>> >> > core
>> >> > functionality from individual database components, thereby streamlining
>> >> > the
>> >> > structure into distinct modules.
>> >> >
>> >> > Looking forward to your feedback and suggestions, thanks.
>> >> > Best regards,
>> >> > Joao Boto
>> >> >
>> >> > [1]
>> >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
>> >>
>>
Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc
Posted by João Boto <es...@apache.org>.
Hi Muhammet,
Have you had a chance to review the recently merged pull request [1]?
We've introduced a new feature allowing users to include ad hoc configurations in the 'JdbcConnectionOptions' class.
```
new JdbcConnectionOptions.JdbcConnectionOptionsBuilder()
.withUrl(FakeDBUtils.TEST_DB_URL)
.withProperty("keyA", "valueA")
.build();
```
This provides flexibility by enabling users to specify additional configuration parameters dynamically.
[1] https://github.com/apache/flink-connector-jdbc/pull/115/files
Best
On 2024/05/06 07:34:06 Muhammet Orazov wrote:
> Morning João,
>
> Recently we had a case where the JDBC drivers authentication was
> different than username&password authentication. For it to work, certain
> hacks required, there interface would have been helpful.
>
> But I agree maybe the interface module separation is not required at the
> moment.
>
> Thanks for your efforts!
>
> Best,
> Muhammet
>
>
> On 2024-05-03 12:25, João Boto wrote:
> > Hi Muhammet,
> >
> > While I generally agree, given our current usage, I'm struggling to
> > discern any clear advantage. We already have abstract implementations
> > that cover all necessary interfaces and offer essential functionality,
> > complemented by a robust set of reusable tests to streamline
> > implementation.
> >
> > With this established infrastructure in place, coupled with the added
> > import overhead of introducing another module, I find it difficult to
> > identify any distinct benefits at this point.
> >
> > Best
> >
> > On 2024/04/26 02:18:52 Muhammet Orazov wrote:
> >> Hey João,
> >>
> >> Thanks for FLIP proposal!
> >>
> >> Since proposal is to introduce modules, would it make sense
> >> to have another module for APIs (flink-jdbc-connector-api)?
> >>
> >> For this I would suggest to move all public interfaces (e.g,
> >> JdbcRowConverter, JdbcConnectionProvider). And even convert
> >> some classes into interface with their default implementations,
> >> for example, JdbcSink, JdbcConnectionOptions.
> >>
> >> This way users would have clear interfaces to build their own
> >> JDBC based Flink connectors.
> >>
> >> Here I am not suggesting to introduce new interfaces, only
> >> suggest also to separate the API from the core implementation.
> >>
> >> What do you think?
> >>
> >> Best,
> >> Muhammet
> >>
> >>
> >> On 2024-04-25 08:54, Joao Boto wrote:
> >> > Hi all,
> >> >
> >> > I'd like to start a discussion on FLIP-449: Reorganization of
> >> > flink-connector-jdbc [1].
> >> > As Flink continues to evolve, we've noticed an increasing level of
> >> > complexity within the JDBC connector.
> >> > The proposed solution is to address this complexity by separating the
> >> > core
> >> > functionality from individual database components, thereby streamlining
> >> > the
> >> > structure into distinct modules.
> >> >
> >> > Looking forward to your feedback and suggestions, thanks.
> >> > Best regards,
> >> > Joao Boto
> >> >
> >> > [1]
> >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
> >>
>
Re: [DISCUSSION] FLIP-449: Reorganization of flink-connector-jdbc
Posted by Muhammet Orazov <mo...@morazow.com.INVALID>.
Morning João,
Recently we had a case where the JDBC drivers authentication was
different than username&password authentication. For it to work, certain
hacks required, there interface would have been helpful.
But I agree maybe the interface module separation is not required at the
moment.
Thanks for your efforts!
Best,
Muhammet
On 2024-05-03 12:25, João Boto wrote:
> Hi Muhammet,
>
> While I generally agree, given our current usage, I'm struggling to
> discern any clear advantage. We already have abstract implementations
> that cover all necessary interfaces and offer essential functionality,
> complemented by a robust set of reusable tests to streamline
> implementation.
>
> With this established infrastructure in place, coupled with the added
> import overhead of introducing another module, I find it difficult to
> identify any distinct benefits at this point.
>
> Best
>
> On 2024/04/26 02:18:52 Muhammet Orazov wrote:
>> Hey João,
>>
>> Thanks for FLIP proposal!
>>
>> Since proposal is to introduce modules, would it make sense
>> to have another module for APIs (flink-jdbc-connector-api)?
>>
>> For this I would suggest to move all public interfaces (e.g,
>> JdbcRowConverter, JdbcConnectionProvider). And even convert
>> some classes into interface with their default implementations,
>> for example, JdbcSink, JdbcConnectionOptions.
>>
>> This way users would have clear interfaces to build their own
>> JDBC based Flink connectors.
>>
>> Here I am not suggesting to introduce new interfaces, only
>> suggest also to separate the API from the core implementation.
>>
>> What do you think?
>>
>> Best,
>> Muhammet
>>
>>
>> On 2024-04-25 08:54, Joao Boto wrote:
>> > Hi all,
>> >
>> > I'd like to start a discussion on FLIP-449: Reorganization of
>> > flink-connector-jdbc [1].
>> > As Flink continues to evolve, we've noticed an increasing level of
>> > complexity within the JDBC connector.
>> > The proposed solution is to address this complexity by separating the
>> > core
>> > functionality from individual database components, thereby streamlining
>> > the
>> > structure into distinct modules.
>> >
>> > Looking forward to your feedback and suggestions, thanks.
>> > Best regards,
>> > Joao Boto
>> >
>> > [1]
>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-449%3A+Reorganization+of+flink-connector-jdbc
>>