You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Martijn Visser <ma...@apache.org> on 2022/11/17 08:29:23 UTC
Re: [DISCUSS] Updating Flink HBase Connectors
Hi Ferenc,
I think you're good to go, since no comments were there. Do let us know if
you need any help :)
Thanks,
Martijn
On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky <fe...@pm.me.invalid>
wrote:
> Hi,
>
> just pinging this thread in case someone missed it and has any opinion
> about the discussed actions.
>
> Best,
> F
>
>
>
>
> ------- Original Message -------
> On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky
> <fe...@pm.me.INVALID> wrote:
>
>
> >
> >
> > Hi Martijn,
> >
> > Thank you for your comment. About HBase 2.x, correct, that is my thought
> process, but it has to be tested and verified.
> >
> > +1 from my side about merging these updates with the connector
> externalization.
> >
> > Best,
> > F
> >
> >
> > ------- Original Message -------
> > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser
> martijnvisser@apache.org wrote:
> >
> >
> >
> > > Hi Ferenc,
> > >
> > > Thanks for opening the discussion on this topic!
> > >
> > > +1 for dropping HBase 1.x.
> > >
> > > Regarding HBase 2.x, if I understand correctly it should be possible to
> > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't it
> make
> > > more sense to always support the latest available version, so
> basically 2.5
> > > at the moment? We could always include a test to check that
> implementation
> > > against an older HBase version.
> > >
> > > I also have a follow-up question already: if there's an agreement on
> this
> > > topic, does it make sense to directly build a new HBase connector in
> its
> > > own external connector repo, since I believe the current connector
> uses the
> > > old source/sink interfaces. We could then directly drop the ones in the
> > > Flink repo and replace it with new implementations?
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky
> <ferenc.csaky@pm.me.invalid
> > >
> > > > Hi everyone,
> > > >
> > > > Now that the connector externalization effort ig going on, I think
> it is
> > > > definitely work to revisit the currently supported HBase versions
> for the
> > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2
> connector
> > > > versions, although both of those versions are kind of outdated.
> > > >
> > > > From the HBase point of view the following can be considered [1]:
> > > >
> > > > - HBase 1.x is dead, so on the way forward it should be safe to drop
> it.
> > > > - HBase 2.2 is EoL, but still used actively, we are also supporting
> it ins
> > > > some of our still active releases as Cloudera.
> > > > - HBase 2.4 is the main thing now and probably will be supported for
> a
> > > > while (by us, definitely).
> > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it is
> > > > possible it won't live long.
> > > > - HBase 3 is in alpha, but shooting for that probably would be early
> in
> > > > the near future.
> > > >
> > > > In addition, if we are only using the standard HBase 2.x client
> APIs, then
> > > > it should be possible to be compile it with any Hbase 2.x version.
> Also,
> > > > any HBase 2.x cluster should be backwards compatible with all
> earlier HBase
> > > > 2.x client libraries. I did not check this part thorougly but I
> think this
> > > > should be true, so ideally it would be enough to have an HBase 2.4
> > > > connector. [2]
> > > >
> > > > Looking forward to your opinions about this topic.
> > > >
> > > > Best,
> > > > F
> > > >
> > > > [1] https://hbase.apache.org/downloads.html
> > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10
> (Client
> > > > API compatibility)
> > >
> > > --
> > > Martijn
> > > https://twitter.com/MartijnVisser82
> > > https://github.com/MartijnVisser
>
Re: [DISCUSS] Updating Flink HBase Connectors
Posted by Martijn Visser <ma...@apache.org>.
Hi Ferenc,
The repository is there https://github.com/apache/flink-connector-hbase
You know where to reach out to if you have any questions.
Best regards,
Martijn
On Mon, Nov 21, 2022 at 12:42 PM Ferenc Csaky <fe...@pm.me.invalid>
wrote:
> Hi!
>
> Sure, thank you both for the input. If the repo is ready, I'll start the
> work.
>
> Best,
> F
>
>
> ------- Original Message -------
> On Thursday, November 17th, 2022 at 09:39, Gabor Somogyi <
> gabor.g.somogyi@gmail.com> wrote:
>
>
> >
> >
> > Hi All,
> >
> > +1 to go. Since we are refurbishing the HBase area maybe we can move the
> > token provider into HBase base project.
> > This would fit into the high level effort to extract everything into
> > external connectors. If you do this and facing any issues just ping me :)
> >
> > BR,
> > G
> >
> >
> > On Thu, Nov 17, 2022 at 9:29 AM Martijn Visser martijnvisser@apache.org
> >
> > wrote:
> >
> > > Hi Ferenc,
> > >
> > > I think you're good to go, since no comments were there. Do let us
> know if
> > > you need any help :)
> > >
> > > Thanks,
> > >
> > > Martijn
> > >
> > > On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky ferenc.csaky@pm.me.invalid
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > just pinging this thread in case someone missed it and has any
> opinion
> > > > about the discussed actions.
> > > >
> > > > Best,
> > > > F
> > > >
> > > > ------- Original Message -------
> > > > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky
> > > > ferenc.csaky@pm.me.INVALID wrote:
> > > >
> > > > > Hi Martijn,
> > > > >
> > > > > Thank you for your comment. About HBase 2.x, correct, that is my
> > > > > thought
> > > > > process, but it has to be tested and verified.
> > > > >
> > > > > +1 from my side about merging these updates with the connector
> > > > > externalization.
> > > > >
> > > > > Best,
> > > > > F
> > > > >
> > > > > ------- Original Message -------
> > > > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser
> > > > > martijnvisser@apache.org wrote:
> > > > >
> > > > > > Hi Ferenc,
> > > > > >
> > > > > > Thanks for opening the discussion on this topic!
> > > > > >
> > > > > > +1 for dropping HBase 1.x.
> > > > > >
> > > > > > Regarding HBase 2.x, if I understand correctly it should be
> possible
> > > > > > to
> > > > > > connect to any 2.x cluster if you're using the 2.x client.
> Wouldn't
> > > > > > it
> > > > > > make
> > > > > > more sense to always support the latest available version, so
> > > > > > basically 2.5
> > > > > > at the moment? We could always include a test to check that
> > > > > > implementation
> > > > > > against an older HBase version.
> > > > > >
> > > > > > I also have a follow-up question already: if there's an
> agreement on
> > > > > > this
> > > > > > topic, does it make sense to directly build a new HBase
> connector in
> > > > > > its
> > > > > > own external connector repo, since I believe the current
> connector
> > > > > > uses the
> > > > > > old source/sink interfaces. We could then directly drop the ones
> in
> > > > > > the
> > > > > > Flink repo and replace it with new implementations?
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Martijn
> > > > > >
> > > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky
> > > > > > <ferenc.csaky@pm.me.invalid
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Now that the connector externalization effort ig going on, I
> think
> > > > > > > it is
> > > > > > > definitely work to revisit the currently supported HBase
> versions
> > > > > > > for the
> > > > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2
> > > > > > > connector
> > > > > > > versions, although both of those versions are kind of outdated.
> > > > > > >
> > > > > > > From the HBase point of view the following can be considered
> [1]:
> > > > > > >
> > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to
> > > > > > > drop
> > > > > > > it.
> > > > > > > - HBase 2.2 is EoL, but still used actively, we are also
> supporting
> > > > > > > it ins
> > > > > > > some of our still active releases as Cloudera.
> > > > > > > - HBase 2.4 is the main thing now and probably will be
> supported
> > > > > > > for
> > > > > > > a
> > > > > > > while (by us, definitely).
> > > > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so
> it
> > > > > > > is
> > > > > > > possible it won't live long.
> > > > > > > - HBase 3 is in alpha, but shooting for that probably would be
> > > > > > > early
> > > > > > > in
> > > > > > > the near future.
> > > > > > >
> > > > > > > In addition, if we are only using the standard HBase 2.x client
> > > > > > > APIs, then
> > > > > > > it should be possible to be compile it with any Hbase 2.x
> version.
> > > > > > > Also,
> > > > > > > any HBase 2.x cluster should be backwards compatible with all
> > > > > > > earlier HBase
> > > > > > > 2.x client libraries. I did not check this part thorougly but I
> > > > > > > think this
> > > > > > > should be true, so ideally it would be enough to have an HBase
> 2.4
> > > > > > > connector. [2]
> > > > > > >
> > > > > > > Looking forward to your opinions about this topic.
> > > > > > >
> > > > > > > Best,
> > > > > > > F
> > > > > > >
> > > > > > > [1] https://hbase.apache.org/downloads.html
> > > > > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10
> > > > > > > (Client
> > > > > > > API compatibility)
> > > > > >
> > > > > > --
> > > > > > Martijn
> > > > > > https://twitter.com/MartijnVisser82
> > > > > > https://github.com/MartijnVisser
>
Re: [DISCUSS] Updating Flink HBase Connectors
Posted by Ferenc Csaky <fe...@pm.me.INVALID>.
Hi!
Sure, thank you both for the input. If the repo is ready, I'll start the work.
Best,
F
------- Original Message -------
On Thursday, November 17th, 2022 at 09:39, Gabor Somogyi <ga...@gmail.com> wrote:
>
>
> Hi All,
>
> +1 to go. Since we are refurbishing the HBase area maybe we can move the
> token provider into HBase base project.
> This would fit into the high level effort to extract everything into
> external connectors. If you do this and facing any issues just ping me :)
>
> BR,
> G
>
>
> On Thu, Nov 17, 2022 at 9:29 AM Martijn Visser martijnvisser@apache.org
>
> wrote:
>
> > Hi Ferenc,
> >
> > I think you're good to go, since no comments were there. Do let us know if
> > you need any help :)
> >
> > Thanks,
> >
> > Martijn
> >
> > On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky ferenc.csaky@pm.me.invalid
> > wrote:
> >
> > > Hi,
> > >
> > > just pinging this thread in case someone missed it and has any opinion
> > > about the discussed actions.
> > >
> > > Best,
> > > F
> > >
> > > ------- Original Message -------
> > > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky
> > > ferenc.csaky@pm.me.INVALID wrote:
> > >
> > > > Hi Martijn,
> > > >
> > > > Thank you for your comment. About HBase 2.x, correct, that is my
> > > > thought
> > > > process, but it has to be tested and verified.
> > > >
> > > > +1 from my side about merging these updates with the connector
> > > > externalization.
> > > >
> > > > Best,
> > > > F
> > > >
> > > > ------- Original Message -------
> > > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser
> > > > martijnvisser@apache.org wrote:
> > > >
> > > > > Hi Ferenc,
> > > > >
> > > > > Thanks for opening the discussion on this topic!
> > > > >
> > > > > +1 for dropping HBase 1.x.
> > > > >
> > > > > Regarding HBase 2.x, if I understand correctly it should be possible
> > > > > to
> > > > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't
> > > > > it
> > > > > make
> > > > > more sense to always support the latest available version, so
> > > > > basically 2.5
> > > > > at the moment? We could always include a test to check that
> > > > > implementation
> > > > > against an older HBase version.
> > > > >
> > > > > I also have a follow-up question already: if there's an agreement on
> > > > > this
> > > > > topic, does it make sense to directly build a new HBase connector in
> > > > > its
> > > > > own external connector repo, since I believe the current connector
> > > > > uses the
> > > > > old source/sink interfaces. We could then directly drop the ones in
> > > > > the
> > > > > Flink repo and replace it with new implementations?
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Martijn
> > > > >
> > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky
> > > > > <ferenc.csaky@pm.me.invalid
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > Now that the connector externalization effort ig going on, I think
> > > > > > it is
> > > > > > definitely work to revisit the currently supported HBase versions
> > > > > > for the
> > > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2
> > > > > > connector
> > > > > > versions, although both of those versions are kind of outdated.
> > > > > >
> > > > > > From the HBase point of view the following can be considered [1]:
> > > > > >
> > > > > > - HBase 1.x is dead, so on the way forward it should be safe to
> > > > > > drop
> > > > > > it.
> > > > > > - HBase 2.2 is EoL, but still used actively, we are also supporting
> > > > > > it ins
> > > > > > some of our still active releases as Cloudera.
> > > > > > - HBase 2.4 is the main thing now and probably will be supported
> > > > > > for
> > > > > > a
> > > > > > while (by us, definitely).
> > > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it
> > > > > > is
> > > > > > possible it won't live long.
> > > > > > - HBase 3 is in alpha, but shooting for that probably would be
> > > > > > early
> > > > > > in
> > > > > > the near future.
> > > > > >
> > > > > > In addition, if we are only using the standard HBase 2.x client
> > > > > > APIs, then
> > > > > > it should be possible to be compile it with any Hbase 2.x version.
> > > > > > Also,
> > > > > > any HBase 2.x cluster should be backwards compatible with all
> > > > > > earlier HBase
> > > > > > 2.x client libraries. I did not check this part thorougly but I
> > > > > > think this
> > > > > > should be true, so ideally it would be enough to have an HBase 2.4
> > > > > > connector. [2]
> > > > > >
> > > > > > Looking forward to your opinions about this topic.
> > > > > >
> > > > > > Best,
> > > > > > F
> > > > > >
> > > > > > [1] https://hbase.apache.org/downloads.html
> > > > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10
> > > > > > (Client
> > > > > > API compatibility)
> > > > >
> > > > > --
> > > > > Martijn
> > > > > https://twitter.com/MartijnVisser82
> > > > > https://github.com/MartijnVisser
Re: [DISCUSS] Updating Flink HBase Connectors
Posted by Gabor Somogyi <ga...@gmail.com>.
Hi All,
+1 to go. Since we are refurbishing the HBase area maybe we can move the
token provider into HBase base project.
This would fit into the high level effort to extract everything into
external connectors. If you do this and facing any issues just ping me :)
BR,
G
On Thu, Nov 17, 2022 at 9:29 AM Martijn Visser <ma...@apache.org>
wrote:
> Hi Ferenc,
>
> I think you're good to go, since no comments were there. Do let us know if
> you need any help :)
>
> Thanks,
>
> Martijn
>
> On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky <fe...@pm.me.invalid>
> wrote:
>
> > Hi,
> >
> > just pinging this thread in case someone missed it and has any opinion
> > about the discussed actions.
> >
> > Best,
> > F
> >
> >
> >
> >
> > ------- Original Message -------
> > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky
> > <fe...@pm.me.INVALID> wrote:
> >
> >
> > >
> > >
> > > Hi Martijn,
> > >
> > > Thank you for your comment. About HBase 2.x, correct, that is my
> thought
> > process, but it has to be tested and verified.
> > >
> > > +1 from my side about merging these updates with the connector
> > externalization.
> > >
> > > Best,
> > > F
> > >
> > >
> > > ------- Original Message -------
> > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser
> > martijnvisser@apache.org wrote:
> > >
> > >
> > >
> > > > Hi Ferenc,
> > > >
> > > > Thanks for opening the discussion on this topic!
> > > >
> > > > +1 for dropping HBase 1.x.
> > > >
> > > > Regarding HBase 2.x, if I understand correctly it should be possible
> to
> > > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't
> it
> > make
> > > > more sense to always support the latest available version, so
> > basically 2.5
> > > > at the moment? We could always include a test to check that
> > implementation
> > > > against an older HBase version.
> > > >
> > > > I also have a follow-up question already: if there's an agreement on
> > this
> > > > topic, does it make sense to directly build a new HBase connector in
> > its
> > > > own external connector repo, since I believe the current connector
> > uses the
> > > > old source/sink interfaces. We could then directly drop the ones in
> the
> > > > Flink repo and replace it with new implementations?
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky
> > <ferenc.csaky@pm.me.invalid
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > Now that the connector externalization effort ig going on, I think
> > it is
> > > > > definitely work to revisit the currently supported HBase versions
> > for the
> > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2
> > connector
> > > > > versions, although both of those versions are kind of outdated.
> > > > >
> > > > > From the HBase point of view the following can be considered [1]:
> > > > >
> > > > > - HBase 1.x is dead, so on the way forward it should be safe to
> drop
> > it.
> > > > > - HBase 2.2 is EoL, but still used actively, we are also supporting
> > it ins
> > > > > some of our still active releases as Cloudera.
> > > > > - HBase 2.4 is the main thing now and probably will be supported
> for
> > a
> > > > > while (by us, definitely).
> > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it
> is
> > > > > possible it won't live long.
> > > > > - HBase 3 is in alpha, but shooting for that probably would be
> early
> > in
> > > > > the near future.
> > > > >
> > > > > In addition, if we are only using the standard HBase 2.x client
> > APIs, then
> > > > > it should be possible to be compile it with any Hbase 2.x version.
> > Also,
> > > > > any HBase 2.x cluster should be backwards compatible with all
> > earlier HBase
> > > > > 2.x client libraries. I did not check this part thorougly but I
> > think this
> > > > > should be true, so ideally it would be enough to have an HBase 2.4
> > > > > connector. [2]
> > > > >
> > > > > Looking forward to your opinions about this topic.
> > > > >
> > > > > Best,
> > > > > F
> > > > >
> > > > > [1] https://hbase.apache.org/downloads.html
> > > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10
> > (Client
> > > > > API compatibility)
> > > >
> > > > --
> > > > Martijn
> > > > https://twitter.com/MartijnVisser82
> > > > https://github.com/MartijnVisser
> >
>