You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Martijn Visser <ma...@ververica.com> on 2022/02/01 20:07:32 UTC
Re: [DISCUSS] Looking for maintainers for Cassandra connector or decide to remove connector
Hi Marco,
Thanks for volunteering! It would be great if you could help with
contributions, they are much appreciated.
The plan is indeed to move connectors to its own individual repository, for
Cassandra that most likely would be something like
https://github.com/apache/flink-connector-cassandra. That is pending the
availability of some generic stuff (like testing framework). Until that
move is made, contributions for improving the Cassandra connector are still
accepted.
Regarding the priorities, I think you're spot on. With regards to which
Cassandra version should be supported, I'm interested in what you think on
that, also depending on the backwards compatibility of Cassandra itself.
For example with Elasticsearch, the decision was made to follow Elastic's
support policy, meaning we support the latest major version (7.*) and the
latest minor of the major before that (6.8.*).
We can also sync on the current Cassandra related tickets and
clean-up/prioritize those.
Best regards,
Martijn
On Mon, 31 Jan 2022 at 15:23, Marco Zühlke <mz...@apache.org> wrote:
> Hi,
>
> just discovered the voting thread and wanted to raise my hand to in an
> attempt to maintain the Apache Cassandra connector. At work Kafka -> Flink
> -> Cassandra is a very common usage pattern for me.
>
> From other ongoing threads I got that the connectors would be removed from
> main repository anyways and in the future maintained in the
> https://github.com/apache/flink-connectors repo, right ?
>
> From the listed topic I think that the adopting the new ASync API combined
> with probably support for Cassandra 4.0 are the most pressing issue.
>
> Best reagards,
> Marco
>
> On 2022/01/21 08:22:42 Martijn Visser wrote:
> > Hi everyone,
> >
> > We're looking for community members, who would like to maintain Flink's
> > Cassandra connector [1] going forward. The connector currently is only
> > available as a Sink for DataStream users and the original contributors
> are
> > unable to work on further improvements.
> >
> > An overview of some of the things that are missing on the Cassandra
> > connector:
> >
> > * Not using the Unified Sink API [2] or ASync API [3]
> > * Can't be used in the Table API / SQL
> > * Not available as a Source
> > * Not available for Lookups
> > * Not using the latest supported versions for Cassandra
> >
> > If you would like to take on this responsibility or can join this effort
> in
> > a supporting role, please reach out!
> >
> > If we can't find maintainers for this connector, I'll open up a vote to
> > deprecate this connector and remove it.
> >
> > I'm looking forward to your thoughts.
> >
> > Best regards,
> >
> > Martijn Visser
> > https://twitter.com/MartijnVisser82
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/cassandra/
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> > [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> >
>
Re: [DISCUSS] Looking for maintainers for Cassandra connector or decide to remove connector
Posted by Fabian Paul <fp...@apache.org>.
Hi Marco,
Great to hear that you put some thought into the topic. Judging from
the past we already tried once to support multiple external versions
within one connector (ElasticSearch) and it complicates things a lot.
So if it makes your development easier to create a different module
that should be fine. Usually, we try not to break/remove existing
connectors if they are not deprecated yet. In your case, I'd recommend
first developing the unified connector and then deprecating the old
connector.
Regarding the external connector repository, we plan to only
externalize implementations that are based on the unified Source and
Sink interfaces so that we can slowly deprecate the old interfaces.
Best,
Fabian
Re: [DISCUSS] Looking for maintainers for Cassandra connector or decide to remove connector
Posted by Marco Zühlke <mz...@apache.org>.
Hi Martin,
I took a look at how other plugins have handled the migration and I have the feeling most oped for a new connector (new module), instead of adding some more packages to the existing one. Should I take the same route ?
The current connector builds a fat-jar that included version 3.0 of the cassandra driver.
this is a pretty old version. Newer versions (4.x) support Cassandra server version 2.1+ but are not source compatible with the 3.x drivers: https://github.com/datastax/java-driver
So if we update the driver in the connector user can still use it with nearly any running server version, but if they use the driver in their own code they may have to adapt to the new version. As there are anyways changes by the user required, if he starts using a connector with the new sink API, this a perfect moment for such a break.
I spend some time looking at other connectors, the respective FLIPs and plan to start with a new connector next to existing one so user can switch. i try to re-use as much as possible from the old one.
Cheers,
Marco
On 2022/02/01 20:07:32 Martijn Visser wrote:
> Hi Marco,
>
> Thanks for volunteering! It would be great if you could help with
> contributions, they are much appreciated.
>
> The plan is indeed to move connectors to its own individual repository, for
> Cassandra that most likely would be something like
> https://github.com/apache/flink-connector-cassandra. That is pending the
> availability of some generic stuff (like testing framework). Until that
> move is made, contributions for improving the Cassandra connector are still
> accepted.
>
> Regarding the priorities, I think you're spot on. With regards to which
> Cassandra version should be supported, I'm interested in what you think on
> that, also depending on the backwards compatibility of Cassandra itself.
> For example with Elasticsearch, the decision was made to follow Elastic's
> support policy, meaning we support the latest major version (7.*) and the
> latest minor of the major before that (6.8.*).
>
> We can also sync on the current Cassandra related tickets and
> clean-up/prioritize those.
>
> Best regards,
>
> Martijn
>
> On Mon, 31 Jan 2022 at 15:23, Marco Zühlke <mz...@apache.org> wrote:
>
> > Hi,
> >
> > just discovered the voting thread and wanted to raise my hand to in an
> > attempt to maintain the Apache Cassandra connector. At work Kafka -> Flink
> > -> Cassandra is a very common usage pattern for me.
> >
> > From other ongoing threads I got that the connectors would be removed from
> > main repository anyways and in the future maintained in the
> > https://github.com/apache/flink-connectors repo, right ?
> >
> > From the listed topic I think that the adopting the new ASync API combined
> > with probably support for Cassandra 4.0 are the most pressing issue.
> >
> > Best reagards,
> > Marco
> >
> > On 2022/01/21 08:22:42 Martijn Visser wrote:
> > > Hi everyone,
> > >
> > > We're looking for community members, who would like to maintain Flink's
> > > Cassandra connector [1] going forward. The connector currently is only
> > > available as a Sink for DataStream users and the original contributors
> > are
> > > unable to work on further improvements.
> > >
> > > An overview of some of the things that are missing on the Cassandra
> > > connector:
> > >
> > > * Not using the Unified Sink API [2] or ASync API [3]
> > > * Can't be used in the Table API / SQL
> > > * Not available as a Source
> > > * Not available for Lookups
> > > * Not using the latest supported versions for Cassandra
> > >
> > > If you would like to take on this responsibility or can join this effort
> > in
> > > a supporting role, please reach out!
> > >
> > > If we can't find maintainers for this connector, I'll open up a vote to
> > > deprecate this connector and remove it.
> > >
> > > I'm looking forward to your thoughts.
> > >
> > > Best regards,
> > >
> > > Martijn Visser
> > > https://twitter.com/MartijnVisser82
> > >
> > > [1]
> > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/cassandra/
> > > [2]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API
> > > [3]
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
> > >
> >
>