You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gyula Fóra <gy...@gmail.com> on 2020/06/18 13:25:26 UTC
[DISCUSS] Upgrade HBase connector to 2.2.x
Hi All!
I would like to revive an old ticket
<https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
upgrading the HBase version of the connector.
The current HBase version is 1.4.3 which is over 2 years old at this point
and incompatible with the newer HBase versions used at many companies.
We propose to upgrade the connector to the latest version and drop support
for the old version starting from the 1.12 Flink release. This would help
us maintain and improve the HBase connector over time.
If the community agrees we are happy to contribute this upgrade as we have
already developed and tested the updated version.
Cheers,
Gyula
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Jark Wu <im...@gmail.com>.
cc'ed YuLi and ZhengHu who are HBase PMC members and may know how many
users are using HBase 1.4.x and 2.2.x
Best,
Jark
On Fri, 19 Jun 2020 at 14:20, jackylau <li...@gmail.com> wrote:
> + 1 to support HBase 2.x and the hbase 2.x client dependencies are simple
> and clear. Tbe hbase project shades them all
>
> Jark Wu-3 wrote
> +1 to support HBase 2.x But not sure about dropping support for 1.4.x I
> cc'ed to user@ and user-zh@ to hear more feedback from users. Best, Jark
> On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <[hidden email]
> <http:///user/SendEmail.jtp?type=email&email=gyula.fora%40>> wrote: > Hi
> All! > > I would like to revive an old ticket > <
> https://issues.apache.org/jira/browse/FLINK-9849> and discussion around >
> upgrading the HBase version of the connector. > > The current HBase version
> is 1.4.3 which is over 2 years old at this point > and incompatible with
> the newer HBase versions used at many companies. > > We propose to upgrade
> the connector to the latest version and drop support > for the old version
> starting from the 1.12 Flink release. This would help > us maintain and
> improve the HBase connector over time. > > If the community agrees we are
> happy to contribute this upgrade as we have > already developed and tested
> the updated version. > > Cheers, > Gyula >
>
>
> ------------------------------
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Jark Wu <im...@gmail.com>.
cc'ed YuLi and ZhengHu who are HBase PMC members and may know how many
users are using HBase 1.4.x and 2.2.x
Best,
Jark
On Fri, 19 Jun 2020 at 14:20, jackylau <li...@gmail.com> wrote:
> + 1 to support HBase 2.x and the hbase 2.x client dependencies are simple
> and clear. Tbe hbase project shades them all
>
> Jark Wu-3 wrote
> +1 to support HBase 2.x But not sure about dropping support for 1.4.x I
> cc'ed to user@ and user-zh@ to hear more feedback from users. Best, Jark
> On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <[hidden email]
> <http:///user/SendEmail.jtp?type=email&email=gyula.fora%40>> wrote: > Hi
> All! > > I would like to revive an old ticket > <
> https://issues.apache.org/jira/browse/FLINK-9849> and discussion around >
> upgrading the HBase version of the connector. > > The current HBase version
> is 1.4.3 which is over 2 years old at this point > and incompatible with
> the newer HBase versions used at many companies. > > We propose to upgrade
> the connector to the latest version and drop support > for the old version
> starting from the 1.12 Flink release. This would help > us maintain and
> improve the HBase connector over time. > > If the community agrees we are
> happy to contribute this upgrade as we have > already developed and tested
> the updated version. > > Cheers, > Gyula >
>
>
> ------------------------------
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at
> Nabble.com.
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Leonard Xu <xb...@gmail.com>.
Hi, all
I’m +1 to support HBase 2.x and also keep the HBase1.x in flink project.
IIRC, HBase1.x is still widely used in production in the early discussion of this thread according to HBase PMC memeber Yu Li and OpenInx’s investigation.
Moving HBase1.x connector to Bahir means the HBase1.x connector will lose the timely support from Flink community in my opinion, I’m slightly -1 for this.
Best
Leonard Xu
> 在 2020年8月11日，02:36，Márton Balassi <ba...@gmail.com> 写道：
>
> Hi All,
>
> I am also fairly torn on this one, however unless we are vigilant in keeping the flink repository relatively lean the number of modules will just keep increasing and pose an increasingly greater maintainability challenge.
> Less frequently used connectors are a strong candidate to be maintained in bahir-flink and/or via flink-packages.org <http://flink-packages.org/> (I do not support creating a third option in apache/flink-connectors). If the testing infrastructure of bahir-flink is a concern, then we should invest into improving that, so that it can serve as a reasonable alternative.
>
> I prefer the option of HBase 2.x in Flink and 1.x in Bahir, with a community commitment of improving the Bahir testing infra. If taking this step immediately is deemed too risky I can accept having the two version side-by-side in Flink for the time being, but without refactoring them to use a common base module (like flink-kafka-connector-base) as we expect to move 1.x to Bahir when the infra is satisfactory.
>
> My position is not against HBase by any means, it is for a more maintainable Flink repository. I have assigned [1] to Miklos, he aims at opening a PR in the coming days - which we might modify based on the outcome of this discussion.
>
> [1] https://issues.apache.org/jira/browse/FLINK-18795 <https://issues.apache.org/jira/browse/FLINK-18795>
> On Mon, Aug 10, 2020 at 4:16 PM Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
> @Jark: Thanks for bringing up these concerns.
> All the problems you've mentioned are "solvable":
> - uber jar: Bahir could provide a hbase1 uber jar (we could theoretically also add a dependency from flink to bahir and provide the uber jar from Flink)
> - e2e tests: we know that the connector is stable, as long as we are not adding major changes (or we are moving the respective e2e tests to bahir).
>
> On the other hand, I agree with you that supporting multiple versions of a connector is pretty common (see Kafka or elasticsearch), so why can't we allow it for Hbase now?
>
> I'm really torn on this and would like to hear more opinions on this.
>
>
> On Fri, Aug 7, 2020 at 11:24 PM Felipe Lolas <flolas@icloud.com <ma...@icloud.com>> wrote:
> Hi all!
>
> Im new here; I have been using the flink connector for hbase 1.2, but recently opt to upgrading to hbase 2.1(basically because was bundled in CDH6)
>
> it would be nice to add support for hbase 2.x!
> I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and keeping that in mind last week I sent a PR with a solution supporting 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules sending the PR).
>
> i would be happy to help if needed!
>
>
>
> Felipe.
>
>> El 07-08-2020, a la(s) 10:53, Jark Wu <imjark@gmail.com <ma...@gmail.com>> escribió:
>>
>>
>> I'm +1 to add HBase 2.x
>>
>> However, I have some concerns about moving HBase 1.x to Bahir:
>> 1) As discussed above, there are still lots of people using HBase 1.x.
>> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E tests.
>> 3) We also paid lots of effort to provide an uber connector jar for HBase (not yet released), it is helpful to improve the out-of-box experience.
>>
>> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It doesn't add too much work to maintain a new version.
>> Keeping the old version can also help us to develop the new one. I would suggest to keep HBase 1.x in the repository for at least one more release.
>> Another idea is that maybe it's a good time to have a "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to it.
>> It would also be a good place to accept the contribution of pulsar connector and other connectors.
>>
>> Best,
>> Jark
>>
>>
>> On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
>> Hi,
>>
>> Thank you for picking this up so quickly. I have no objections regarding
>> all the proposed items.
>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>> need somebody to merge it.
>>
>>
>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <balassi.marton@gmail.com <ma...@gmail.com>>
>> wrote:
>>
>> > Hi Robert and Gyula,
>> >
>> > Thanks for reviving this thread. We have the implementation (currently for
>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
>> > recently written a readme for said version, he would be interested in
>> > contributing the upgraded connector back. The latest HBase version is
>> > 2.3.0, if we are touching the codebase anyway I would propose to have that.
>> >
>> > If everyone is comfortable with it I would assign [1] to Miklos with
>> > double checking the all functionality that Felipe has proposed is included.
>> > [1] https://issues.apache.org/jira/browse/FLINK-18795 <https://issues.apache.org/jira/browse/FLINK-18795>
>> > [2] https://hbase.apache.org/downloads.html <https://hbase.apache.org/downloads.html>
>> >
>> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gyula.fora@gmail.com <ma...@gmail.com>> wrote:
>> >
>> >> Hi Robert,
>> >>
>> >> I completely agree with you on the Bahir based approach.
>> >>
>> >> I am happy to help with the contribution on the bahir side, with thorough
>> >> review and testing.
>> >>
>> >> Cheers,
>> >> Gyula
>> >>
>> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
>> >>
>> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
>> >>>
>> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rmetzger@apache.org <ma...@apache.org>>
>> >>> wrote:
>> >>>
>> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>> >>>> for adding HBase 2 support.
>> >>>>
>> >>>> it seems that there is demand for a HBase 2 connector, and consensus to
>> >>>> do it.
>> >>>>
>> >>>> The remaining question in this thread seems to be the "how". I would
>> >>>> propose to go the other way around as Gyula suggested: We move the legacy
>> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>> >>>> also run Hbase end to end tests. This will help us to stabilize the new
>> >>>> connector and ensure a good quality.
>> >>>> It also, the perception of what goes into Flink, and what into Bahir is
>> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>> >>>> legacy, experimental or unstable connectors into Bahir.
>> >>>>
>> >>>>
>> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>> >>>> review and contribution to Bahir)
>> >>>>
>> >>>>
>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795 <https://issues.apache.org/jira/browse/FLINK-18795>
>> >>>> [2] https://github.com/apache/flink/pull/13047 <https://github.com/apache/flink/pull/13047>
>> >>>>
>> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gyula.fora@gmail.com <ma...@gmail.com>>
>> >>>> wrote:
>> >>>>
>> >>>>> If we were to go the bahir route, I don't see the point in migrating
>> >>>>> the 1.4.x version there since that's already available in Flink. To me that
>> >>>>> is almost the same as dropping explicit support for 1.4 and telling users
>> >>>>> to use older connector versions if they wish to keep using it.
>> >>>>>
>> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>> >>>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>> >>>>>
>> >>>>> What do you think?
>> >>>>>
>> >>>>> Gyula
>> >>>>>
>> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <arvid@ververica.com <ma...@ververica.com>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>> >>>>>> out to Bahir and list them in flink-packages to avoid adding even more
>> >>>>>> modules to Flink core?
>> >>>>>>
>> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <openinx@gmail.com <ma...@gmail.com>> wrote:
>> >>>>>>
>> >>>>>>> Hi
>> >>>>>>>
>> >>>>>>> According to my observation in the hbase community, there are still
>> >>>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>> >>>>>>> or 1.5.x). so I'd like to suggest that
>> >>>>>>> supporting both hbase1.x & hbase2.x connector.
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <brighty916@gmail.com <ma...@gmail.com>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>> >>>>>>>> doing for Kafka.
>> >>>>>>>>
>> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
>> >>>>>>>>
>> >>>>>>>>> One supplement:
>> >>>>>>>>>
>> >>>>>>>>> I noticed that there are discussions in HBase ML this March about
>> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>> >>>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>> >>>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>> >>>>>>>>> production.
>> >>>>>>>>>
>> >>>>>>>>> Best Regards,
>> >>>>>>>>> Yu
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E <http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>> >>>>>>>>>> indeed an old version.
>> >>>>>>>>>>
>> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>> >>>>>>>>>> production. We could also see that the HBase community is still maintaining
>> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>> >>>>>>>>>>
>> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>> >>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>> >>>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>> >>>>>>>>>> be able to work with 1.x HBase clusters.
>> >>>>>>>>>>
>> >>>>>>>>>> I totally understand the additional efforts of maintaining two
>> >>>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>> >>>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>> >>>>>>>>>>
>> >>>>>>>>>> Best Regards,
>> >>>>>>>>>> Yu
>> >>>>>>>>>>
>> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html <http://hbase.apache.org/downloads.html>
>> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning <https://hbase.apache.org/book.html#hbase.versioning>
>> >>>>>>>>>> [3] https://semver.org/ <https://semver.org/>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xbjtdcq@gmail.com <ma...@gmail.com>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>> >>>>>>>>>>> deprecates finished(maybe one version is enough).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>> >>>>>>>>>>> dependencies is a little more complex.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <liuyongvs@gmail.com <ma...@gmail.com>> 写道：
>> >>>>>>>>>>>
>> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Leonard Xu
>> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687 <https://github.com/apache/flink/pull/12687>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Best Regards
>> >>>>>>>> Michael Li
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Arvid Heise | Senior Java Developer
>> >>>>>>
>> >>>>>> <https://www.ververica.com/ <https://www.ververica.com/>>
>> >>>>>>
>> >>>>>> Follow us @VervericaData
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Join Flink Forward <https://flink-forward.org/ <https://flink-forward.org/>> - The Apache Flink
>> >>>>>> Conference
>> >>>>>>
>> >>>>>> Stream Processing | Event Driven | Real Time
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> >>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Ververica GmbH
>> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>> >>>>>> Ji (Toni) Cheng
>> >>>>>>
>> >>>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Leonard Xu <xb...@gmail.com>.
Hi, all
I’m +1 to support HBase 2.x and also keep the HBase1.x in flink project.
IIRC, HBase1.x is still widely used in production in the early discussion of this thread according to HBase PMC memeber Yu Li and OpenInx’s investigation.
Moving HBase1.x connector to Bahir means the HBase1.x connector will lose the timely support from Flink community in my opinion, I’m slightly -1 for this.
Best
Leonard Xu
> 在 2020年8月11日，02:36，Márton Balassi <ba...@gmail.com> 写道：
>
> Hi All,
>
> I am also fairly torn on this one, however unless we are vigilant in keeping the flink repository relatively lean the number of modules will just keep increasing and pose an increasingly greater maintainability challenge.
> Less frequently used connectors are a strong candidate to be maintained in bahir-flink and/or via flink-packages.org <http://flink-packages.org/> (I do not support creating a third option in apache/flink-connectors). If the testing infrastructure of bahir-flink is a concern, then we should invest into improving that, so that it can serve as a reasonable alternative.
>
> I prefer the option of HBase 2.x in Flink and 1.x in Bahir, with a community commitment of improving the Bahir testing infra. If taking this step immediately is deemed too risky I can accept having the two version side-by-side in Flink for the time being, but without refactoring them to use a common base module (like flink-kafka-connector-base) as we expect to move 1.x to Bahir when the infra is satisfactory.
>
> My position is not against HBase by any means, it is for a more maintainable Flink repository. I have assigned [1] to Miklos, he aims at opening a PR in the coming days - which we might modify based on the outcome of this discussion.
>
> [1] https://issues.apache.org/jira/browse/FLINK-18795 <https://issues.apache.org/jira/browse/FLINK-18795>
> On Mon, Aug 10, 2020 at 4:16 PM Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
> @Jark: Thanks for bringing up these concerns.
> All the problems you've mentioned are "solvable":
> - uber jar: Bahir could provide a hbase1 uber jar (we could theoretically also add a dependency from flink to bahir and provide the uber jar from Flink)
> - e2e tests: we know that the connector is stable, as long as we are not adding major changes (or we are moving the respective e2e tests to bahir).
>
> On the other hand, I agree with you that supporting multiple versions of a connector is pretty common (see Kafka or elasticsearch), so why can't we allow it for Hbase now?
>
> I'm really torn on this and would like to hear more opinions on this.
>
>
> On Fri, Aug 7, 2020 at 11:24 PM Felipe Lolas <flolas@icloud.com <ma...@icloud.com>> wrote:
> Hi all!
>
> Im new here; I have been using the flink connector for hbase 1.2, but recently opt to upgrading to hbase 2.1(basically because was bundled in CDH6)
>
> it would be nice to add support for hbase 2.x!
> I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and keeping that in mind last week I sent a PR with a solution supporting 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules sending the PR).
>
> i would be happy to help if needed!
>
>
>
> Felipe.
>
>> El 07-08-2020, a la(s) 10:53, Jark Wu <imjark@gmail.com <ma...@gmail.com>> escribió:
>>
>>
>> I'm +1 to add HBase 2.x
>>
>> However, I have some concerns about moving HBase 1.x to Bahir:
>> 1) As discussed above, there are still lots of people using HBase 1.x.
>> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E tests.
>> 3) We also paid lots of effort to provide an uber connector jar for HBase (not yet released), it is helpful to improve the out-of-box experience.
>>
>> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It doesn't add too much work to maintain a new version.
>> Keeping the old version can also help us to develop the new one. I would suggest to keep HBase 1.x in the repository for at least one more release.
>> Another idea is that maybe it's a good time to have a "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to it.
>> It would also be a good place to accept the contribution of pulsar connector and other connectors.
>>
>> Best,
>> Jark
>>
>>
>> On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
>> Hi,
>>
>> Thank you for picking this up so quickly. I have no objections regarding
>> all the proposed items.
>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>> need somebody to merge it.
>>
>>
>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <balassi.marton@gmail.com <ma...@gmail.com>>
>> wrote:
>>
>> > Hi Robert and Gyula,
>> >
>> > Thanks for reviving this thread. We have the implementation (currently for
>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
>> > recently written a readme for said version, he would be interested in
>> > contributing the upgraded connector back. The latest HBase version is
>> > 2.3.0, if we are touching the codebase anyway I would propose to have that.
>> >
>> > If everyone is comfortable with it I would assign [1] to Miklos with
>> > double checking the all functionality that Felipe has proposed is included.
>> > [1] https://issues.apache.org/jira/browse/FLINK-18795 <https://issues.apache.org/jira/browse/FLINK-18795>
>> > [2] https://hbase.apache.org/downloads.html <https://hbase.apache.org/downloads.html>
>> >
>> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gyula.fora@gmail.com <ma...@gmail.com>> wrote:
>> >
>> >> Hi Robert,
>> >>
>> >> I completely agree with you on the Bahir based approach.
>> >>
>> >> I am happy to help with the contribution on the bahir side, with thorough
>> >> review and testing.
>> >>
>> >> Cheers,
>> >> Gyula
>> >>
>> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rmetzger@apache.org <ma...@apache.org>> wrote:
>> >>
>> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
>> >>>
>> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rmetzger@apache.org <ma...@apache.org>>
>> >>> wrote:
>> >>>
>> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>> >>>> for adding HBase 2 support.
>> >>>>
>> >>>> it seems that there is demand for a HBase 2 connector, and consensus to
>> >>>> do it.
>> >>>>
>> >>>> The remaining question in this thread seems to be the "how". I would
>> >>>> propose to go the other way around as Gyula suggested: We move the legacy
>> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>> >>>> also run Hbase end to end tests. This will help us to stabilize the new
>> >>>> connector and ensure a good quality.
>> >>>> It also, the perception of what goes into Flink, and what into Bahir is
>> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>> >>>> legacy, experimental or unstable connectors into Bahir.
>> >>>>
>> >>>>
>> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>> >>>> review and contribution to Bahir)
>> >>>>
>> >>>>
>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795 <https://issues.apache.org/jira/browse/FLINK-18795>
>> >>>> [2] https://github.com/apache/flink/pull/13047 <https://github.com/apache/flink/pull/13047>
>> >>>>
>> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gyula.fora@gmail.com <ma...@gmail.com>>
>> >>>> wrote:
>> >>>>
>> >>>>> If we were to go the bahir route, I don't see the point in migrating
>> >>>>> the 1.4.x version there since that's already available in Flink. To me that
>> >>>>> is almost the same as dropping explicit support for 1.4 and telling users
>> >>>>> to use older connector versions if they wish to keep using it.
>> >>>>>
>> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>> >>>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>> >>>>>
>> >>>>> What do you think?
>> >>>>>
>> >>>>> Gyula
>> >>>>>
>> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <arvid@ververica.com <ma...@ververica.com>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>> >>>>>> out to Bahir and list them in flink-packages to avoid adding even more
>> >>>>>> modules to Flink core?
>> >>>>>>
>> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <openinx@gmail.com <ma...@gmail.com>> wrote:
>> >>>>>>
>> >>>>>>> Hi
>> >>>>>>>
>> >>>>>>> According to my observation in the hbase community, there are still
>> >>>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>> >>>>>>> or 1.5.x). so I'd like to suggest that
>> >>>>>>> supporting both hbase1.x & hbase2.x connector.
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <brighty916@gmail.com <ma...@gmail.com>>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>> >>>>>>>> doing for Kafka.
>> >>>>>>>>
>> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
>> >>>>>>>>
>> >>>>>>>>> One supplement:
>> >>>>>>>>>
>> >>>>>>>>> I noticed that there are discussions in HBase ML this March about
>> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>> >>>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>> >>>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>> >>>>>>>>> production.
>> >>>>>>>>>
>> >>>>>>>>> Best Regards,
>> >>>>>>>>> Yu
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E <http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E>
>> >>>>>>>>>
>> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <carp84@gmail.com <ma...@gmail.com>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>> >>>>>>>>>> indeed an old version.
>> >>>>>>>>>>
>> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>> >>>>>>>>>> production. We could also see that the HBase community is still maintaining
>> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>> >>>>>>>>>>
>> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>> >>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>> >>>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>> >>>>>>>>>> be able to work with 1.x HBase clusters.
>> >>>>>>>>>>
>> >>>>>>>>>> I totally understand the additional efforts of maintaining two
>> >>>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>> >>>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>> >>>>>>>>>>
>> >>>>>>>>>> Best Regards,
>> >>>>>>>>>> Yu
>> >>>>>>>>>>
>> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html <http://hbase.apache.org/downloads.html>
>> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning <https://hbase.apache.org/book.html#hbase.versioning>
>> >>>>>>>>>> [3] https://semver.org/ <https://semver.org/>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xbjtdcq@gmail.com <ma...@gmail.com>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>> >>>>>>>>>>> deprecates finished(maybe one version is enough).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>> >>>>>>>>>>> dependencies is a little more complex.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <liuyongvs@gmail.com <ma...@gmail.com>> 写道：
>> >>>>>>>>>>>
>> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Leonard Xu
>> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687 <https://github.com/apache/flink/pull/12687>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Best Regards
>> >>>>>>>> Michael Li
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Arvid Heise | Senior Java Developer
>> >>>>>>
>> >>>>>> <https://www.ververica.com/ <https://www.ververica.com/>>
>> >>>>>>
>> >>>>>> Follow us @VervericaData
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Join Flink Forward <https://flink-forward.org/ <https://flink-forward.org/>> - The Apache Flink
>> >>>>>> Conference
>> >>>>>>
>> >>>>>> Stream Processing | Event Driven | Real Time
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> >>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Ververica GmbH
>> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>> >>>>>> Ji (Toni) Cheng
>> >>>>>>
>> >>>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Márton Balassi <ba...@gmail.com>.
Hi All,
I am also fairly torn on this one, however unless we are vigilant in
keeping the flink repository relatively lean the number of modules will
just keep increasing and pose an increasingly greater maintainability
challenge.
Less frequently used connectors are a strong candidate to be maintained in
bahir-flink and/or via flink-packages.org (I do not support creating a
third option in apache/flink-connectors). If the testing infrastructure of
bahir-flink is a concern, then we should invest into improving that, so
that it can serve as a reasonable alternative.
I prefer the option of HBase 2.x in Flink and 1.x in Bahir, with a
community commitment of improving the Bahir testing infra. If taking this
step immediately is deemed too risky I can accept having the two version
side-by-side in Flink for the time being, but without refactoring them to
use a common base module (like flink-kafka-connector-base) as we expect to
move 1.x to Bahir when the infra is satisfactory.
My position is not against HBase by any means, it is for a more
maintainable Flink repository. I have assigned [1] to Miklos, he aims at
opening a PR in the coming days - which we might modify based on the
outcome of this discussion.
[1] https://issues.apache.org/jira/browse/FLINK-18795
On Mon, Aug 10, 2020 at 4:16 PM Robert Metzger <rm...@apache.org> wrote:
> @Jark: Thanks for bringing up these concerns.
> All the problems you've mentioned are "solvable":
> - uber jar: Bahir could provide a hbase1 uber jar (we could theoretically
> also add a dependency from flink to bahir and provide the uber jar from
> Flink)
> - e2e tests: we know that the connector is stable, as long as we are not
> adding major changes (or we are moving the respective e2e tests to bahir).
>
> On the other hand, I agree with you that supporting multiple versions of a
> connector is pretty common (see Kafka or elasticsearch), so why can't we
> allow it for Hbase now?
>
> I'm really torn on this and would like to hear more opinions on this.
>
>
> On Fri, Aug 7, 2020 at 11:24 PM Felipe Lolas <fl...@icloud.com> wrote:
>
>> Hi all!
>>
>> Im new here; I have been using the flink connector for hbase 1.2, but
>> recently opt to upgrading to hbase 2.1(basically because was bundled in
>> CDH6)
>>
>> it would be nice to add support for hbase 2.x!
>> I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and
>> keeping that in mind last week I sent a PR with a solution supporting
>> 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules
>> sending the PR).
>>
>> i would be happy to help if needed!
>>
>>
>>
>> Felipe.
>>
>> El 07-08-2020, a la(s) 10:53, Jark Wu <im...@gmail.com> escribió:
>>
>>
>> I'm +1 to add HBase 2.x
>>
>> However, I have some concerns about moving HBase 1.x to Bahir:
>> 1) As discussed above, there are still lots of people using HBase 1.x.
>> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E
>> tests.
>> 3) We also paid lots of effort to provide an uber connector jar for HBase
>> (not yet released), it is helpful to improve the out-of-box experience.
>>
>> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It
>> doesn't add too much work to maintain a new version.
>> Keeping the old version can also help us to develop the new one. I would
>> suggest to keep HBase 1.x in the repository for at least one more release.
>> Another idea is that maybe it's a good time to have a
>> "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to
>> it.
>> It would also be a good place to accept the contribution of pulsar
>> connector and other connectors.
>>
>> Best,
>> Jark
>>
>>
>> On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rm...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> Thank you for picking this up so quickly. I have no objections regarding
>>> all the proposed items.
>>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>>> need somebody to merge it.
>>>
>>>
>>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <balassi.marton@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Robert and Gyula,
>>> >
>>> > Thanks for reviving this thread. We have the implementation (currently
>>> for
>>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd)
>>> has
>>> > recently written a readme for said version, he would be interested in
>>> > contributing the upgraded connector back. The latest HBase version is
>>> > 2.3.0, if we are touching the codebase anyway I would propose to have
>>> that.
>>> >
>>> > If everyone is comfortable with it I would assign [1] to Miklos with
>>> > double checking the all functionality that Felipe has proposed is
>>> included.
>>> > [1] https://issues.apache.org/jira/browse/FLINK-18795
>>> > [2] https://hbase.apache.org/downloads.html
>>> >
>>> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com>
>>> wrote:
>>> >
>>> >> Hi Robert,
>>> >>
>>> >> I completely agree with you on the Bahir based approach.
>>> >>
>>> >> I am happy to help with the contribution on the bahir side, with
>>> thorough
>>> >> review and testing.
>>> >>
>>> >> Cheers,
>>> >> Gyula
>>> >>
>>> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org>
>>> wrote:
>>> >>
>>> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
>>> >>>
>>> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>>> >>> wrote:
>>> >>>
>>> >>>> I would like to revive this discussion. There's a new JIRA[1] +
>>> PR[2]
>>> >>>> for adding HBase 2 support.
>>> >>>>
>>> >>>> it seems that there is demand for a HBase 2 connector, and
>>> consensus to
>>> >>>> do it.
>>> >>>>
>>> >>>> The remaining question in this thread seems to be the "how". I would
>>> >>>> propose to go the other way around as Gyula suggested: We move the
>>> legacy
>>> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>>> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where
>>> we
>>> >>>> also run Hbase end to end tests. This will help us to stabilize the
>>> new
>>> >>>> connector and ensure a good quality.
>>> >>>> It also, the perception of what goes into Flink, and what into
>>> Bahir is
>>> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>>> >>>> legacy, experimental or unstable connectors into Bahir.
>>> >>>>
>>> >>>>
>>> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>>> >>>> review and contribution to Bahir)
>>> >>>>
>>> >>>>
>>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>>> >>>> [2] https://github.com/apache/flink/pull/13047
>>> >>>>
>>> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> If we were to go the bahir route, I don't see the point in
>>> migrating
>>> >>>>> the 1.4.x version there since that's already available in Flink.
>>> To me that
>>> >>>>> is almost the same as dropping explicit support for 1.4 and
>>> telling users
>>> >>>>> to use older connector versions if they wish to keep using it.
>>> >>>>>
>>> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>> >>>>> that, we can do that inside Flink and only push the 2.4.x version
>>> to bahir.
>>> >>>>>
>>> >>>>> What do you think?
>>> >>>>>
>>> >>>>> Gyula
>>> >>>>>
>>> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull
>>> them
>>> >>>>>> out to Bahir and list them in flink-packages to avoid adding even
>>> more
>>> >>>>>> modules to Flink core?
>>> >>>>>>
>>> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>>> Hi
>>> >>>>>>>
>>> >>>>>>> According to my observation in the hbase community, there are
>>> still
>>> >>>>>>> lots of hbase users running their production cluster with
>>> version 1.x (1.4x
>>> >>>>>>> or 1.5.x). so I'd like to suggest that
>>> >>>>>>> supporting both hbase1.x & hbase2.x connector.
>>> >>>>>>>
>>> >>>>>>> Thanks.
>>> >>>>>>>
>>> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we
>>> are
>>> >>>>>>>> doing for Kafka.
>>> >>>>>>>>
>>> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> One supplement:
>>> >>>>>>>>>
>>> >>>>>>>>> I noticed that there are discussions in HBase ML this March
>>> about
>>> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will
>>> follow up in
>>> >>>>>>>>> HBase community about why we didn't take real action. However,
>>> this doesn't
>>> >>>>>>>>> change my previous statement / stand due to the number of 1.x
>>> usages in
>>> >>>>>>>>> production.
>>> >>>>>>>>>
>>> >>>>>>>>> Best Regards,
>>> >>>>>>>>> Yu
>>> >>>>>>>>>
>>> >>>>>>>>> [1]
>>> >>>>>>>>>
>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>> >>>>>>>>>
>>> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3
>>> is
>>> >>>>>>>>>> indeed an old version.
>>> >>>>>>>>>>
>>> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>> >>>>>>>>>> production. We could also see that the HBase community is
>>> still maintaining
>>> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13)
>>> [1]
>>> >>>>>>>>>>
>>> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2]
>>> [3]
>>> >>>>>>>>>> thus don't promise any kind of compatibility
>>> (source/binary/wire, etc.)
>>> >>>>>>>>>> between major versions. So if we only maintain 2.x connector,
>>> it would not
>>> >>>>>>>>>> be able to work with 1.x HBase clusters.
>>> >>>>>>>>>>
>>> >>>>>>>>>> I totally understand the additional efforts of maintaining two
>>> >>>>>>>>>> modules, but since we're also reserving multiple versions for
>>> kafka
>>> >>>>>>>>>> connector, meantime considering the current HBase
>>> in-production status, I'd
>>> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>> >>>>>>>>>>
>>> >>>>>>>>>> Best Regards,
>>> >>>>>>>>>> Yu
>>> >>>>>>>>>>
>>> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>> >>>>>>>>>> [3] https://semver.org/
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3
>>> until we
>>> >>>>>>>>>>> deprecates finished(maybe one version is enough).
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old,
>>> and
>>> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for
>>> pure SQL user, the
>>> >>>>>>>>>>> dependencies is a little more complex.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client
>>> dependencies
>>> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Best,
>>> >>>>>>>>>>> Leonard Xu
>>> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> --
>>> >>>>>>>> Best Regards
>>> >>>>>>>> Michael Li
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>>
>>> >>>>>> Arvid Heise | Senior Java Developer
>>> >>>>>>
>>> >>>>>> <https://www.ververica.com/>
>>> >>>>>>
>>> >>>>>> Follow us @VervericaData
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>>
>>> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache
>>> Flink
>>> >>>>>> Conference
>>> >>>>>>
>>> >>>>>> Stream Processing | Event Driven | Real Time
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>>
>>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>> >>>>>> <
>>> https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g
>>> >
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Ververica GmbH
>>> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung
>>> Jason,
>>> >>>>>> Ji (Toni) Cheng
>>> >>>>>>
>>> >>>>>
>>>
>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Márton Balassi <ba...@gmail.com>.
Hi All,
I am also fairly torn on this one, however unless we are vigilant in
keeping the flink repository relatively lean the number of modules will
just keep increasing and pose an increasingly greater maintainability
challenge.
Less frequently used connectors are a strong candidate to be maintained in
bahir-flink and/or via flink-packages.org (I do not support creating a
third option in apache/flink-connectors). If the testing infrastructure of
bahir-flink is a concern, then we should invest into improving that, so
that it can serve as a reasonable alternative.
I prefer the option of HBase 2.x in Flink and 1.x in Bahir, with a
community commitment of improving the Bahir testing infra. If taking this
step immediately is deemed too risky I can accept having the two version
side-by-side in Flink for the time being, but without refactoring them to
use a common base module (like flink-kafka-connector-base) as we expect to
move 1.x to Bahir when the infra is satisfactory.
My position is not against HBase by any means, it is for a more
maintainable Flink repository. I have assigned [1] to Miklos, he aims at
opening a PR in the coming days - which we might modify based on the
outcome of this discussion.
[1] https://issues.apache.org/jira/browse/FLINK-18795
On Mon, Aug 10, 2020 at 4:16 PM Robert Metzger <rm...@apache.org> wrote:
> @Jark: Thanks for bringing up these concerns.
> All the problems you've mentioned are "solvable":
> - uber jar: Bahir could provide a hbase1 uber jar (we could theoretically
> also add a dependency from flink to bahir and provide the uber jar from
> Flink)
> - e2e tests: we know that the connector is stable, as long as we are not
> adding major changes (or we are moving the respective e2e tests to bahir).
>
> On the other hand, I agree with you that supporting multiple versions of a
> connector is pretty common (see Kafka or elasticsearch), so why can't we
> allow it for Hbase now?
>
> I'm really torn on this and would like to hear more opinions on this.
>
>
> On Fri, Aug 7, 2020 at 11:24 PM Felipe Lolas <fl...@icloud.com> wrote:
>
>> Hi all!
>>
>> Im new here; I have been using the flink connector for hbase 1.2, but
>> recently opt to upgrading to hbase 2.1(basically because was bundled in
>> CDH6)
>>
>> it would be nice to add support for hbase 2.x!
>> I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and
>> keeping that in mind last week I sent a PR with a solution supporting
>> 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules
>> sending the PR).
>>
>> i would be happy to help if needed!
>>
>>
>>
>> Felipe.
>>
>> El 07-08-2020, a la(s) 10:53, Jark Wu <im...@gmail.com> escribió:
>>
>>
>> I'm +1 to add HBase 2.x
>>
>> However, I have some concerns about moving HBase 1.x to Bahir:
>> 1) As discussed above, there are still lots of people using HBase 1.x.
>> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E
>> tests.
>> 3) We also paid lots of effort to provide an uber connector jar for HBase
>> (not yet released), it is helpful to improve the out-of-box experience.
>>
>> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It
>> doesn't add too much work to maintain a new version.
>> Keeping the old version can also help us to develop the new one. I would
>> suggest to keep HBase 1.x in the repository for at least one more release.
>> Another idea is that maybe it's a good time to have a
>> "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to
>> it.
>> It would also be a good place to accept the contribution of pulsar
>> connector and other connectors.
>>
>> Best,
>> Jark
>>
>>
>> On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rm...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> Thank you for picking this up so quickly. I have no objections regarding
>>> all the proposed items.
>>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>>> need somebody to merge it.
>>>
>>>
>>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <balassi.marton@gmail.com
>>> >
>>> wrote:
>>>
>>> > Hi Robert and Gyula,
>>> >
>>> > Thanks for reviving this thread. We have the implementation (currently
>>> for
>>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd)
>>> has
>>> > recently written a readme for said version, he would be interested in
>>> > contributing the upgraded connector back. The latest HBase version is
>>> > 2.3.0, if we are touching the codebase anyway I would propose to have
>>> that.
>>> >
>>> > If everyone is comfortable with it I would assign [1] to Miklos with
>>> > double checking the all functionality that Felipe has proposed is
>>> included.
>>> > [1] https://issues.apache.org/jira/browse/FLINK-18795
>>> > [2] https://hbase.apache.org/downloads.html
>>> >
>>> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com>
>>> wrote:
>>> >
>>> >> Hi Robert,
>>> >>
>>> >> I completely agree with you on the Bahir based approach.
>>> >>
>>> >> I am happy to help with the contribution on the bahir side, with
>>> thorough
>>> >> review and testing.
>>> >>
>>> >> Cheers,
>>> >> Gyula
>>> >>
>>> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org>
>>> wrote:
>>> >>
>>> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
>>> >>>
>>> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>>> >>> wrote:
>>> >>>
>>> >>>> I would like to revive this discussion. There's a new JIRA[1] +
>>> PR[2]
>>> >>>> for adding HBase 2 support.
>>> >>>>
>>> >>>> it seems that there is demand for a HBase 2 connector, and
>>> consensus to
>>> >>>> do it.
>>> >>>>
>>> >>>> The remaining question in this thread seems to be the "how". I would
>>> >>>> propose to go the other way around as Gyula suggested: We move the
>>> legacy
>>> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>>> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where
>>> we
>>> >>>> also run Hbase end to end tests. This will help us to stabilize the
>>> new
>>> >>>> connector and ensure a good quality.
>>> >>>> It also, the perception of what goes into Flink, and what into
>>> Bahir is
>>> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>>> >>>> legacy, experimental or unstable connectors into Bahir.
>>> >>>>
>>> >>>>
>>> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>>> >>>> review and contribution to Bahir)
>>> >>>>
>>> >>>>
>>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>>> >>>> [2] https://github.com/apache/flink/pull/13047
>>> >>>>
>>> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> If we were to go the bahir route, I don't see the point in
>>> migrating
>>> >>>>> the 1.4.x version there since that's already available in Flink.
>>> To me that
>>> >>>>> is almost the same as dropping explicit support for 1.4 and
>>> telling users
>>> >>>>> to use older connector versions if they wish to keep using it.
>>> >>>>>
>>> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>> >>>>> that, we can do that inside Flink and only push the 2.4.x version
>>> to bahir.
>>> >>>>>
>>> >>>>> What do you think?
>>> >>>>>
>>> >>>>> Gyula
>>> >>>>>
>>> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull
>>> them
>>> >>>>>> out to Bahir and list them in flink-packages to avoid adding even
>>> more
>>> >>>>>> modules to Flink core?
>>> >>>>>>
>>> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>>> Hi
>>> >>>>>>>
>>> >>>>>>> According to my observation in the hbase community, there are
>>> still
>>> >>>>>>> lots of hbase users running their production cluster with
>>> version 1.x (1.4x
>>> >>>>>>> or 1.5.x). so I'd like to suggest that
>>> >>>>>>> supporting both hbase1.x & hbase2.x connector.
>>> >>>>>>>
>>> >>>>>>> Thanks.
>>> >>>>>>>
>>> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we
>>> are
>>> >>>>>>>> doing for Kafka.
>>> >>>>>>>>
>>> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>> >>>>>>>>
>>> >>>>>>>>> One supplement:
>>> >>>>>>>>>
>>> >>>>>>>>> I noticed that there are discussions in HBase ML this March
>>> about
>>> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will
>>> follow up in
>>> >>>>>>>>> HBase community about why we didn't take real action. However,
>>> this doesn't
>>> >>>>>>>>> change my previous statement / stand due to the number of 1.x
>>> usages in
>>> >>>>>>>>> production.
>>> >>>>>>>>>
>>> >>>>>>>>> Best Regards,
>>> >>>>>>>>> Yu
>>> >>>>>>>>>
>>> >>>>>>>>> [1]
>>> >>>>>>>>>
>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>> >>>>>>>>>
>>> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3
>>> is
>>> >>>>>>>>>> indeed an old version.
>>> >>>>>>>>>>
>>> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>> >>>>>>>>>> production. We could also see that the HBase community is
>>> still maintaining
>>> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13)
>>> [1]
>>> >>>>>>>>>>
>>> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2]
>>> [3]
>>> >>>>>>>>>> thus don't promise any kind of compatibility
>>> (source/binary/wire, etc.)
>>> >>>>>>>>>> between major versions. So if we only maintain 2.x connector,
>>> it would not
>>> >>>>>>>>>> be able to work with 1.x HBase clusters.
>>> >>>>>>>>>>
>>> >>>>>>>>>> I totally understand the additional efforts of maintaining two
>>> >>>>>>>>>> modules, but since we're also reserving multiple versions for
>>> kafka
>>> >>>>>>>>>> connector, meantime considering the current HBase
>>> in-production status, I'd
>>> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>> >>>>>>>>>>
>>> >>>>>>>>>> Best Regards,
>>> >>>>>>>>>> Yu
>>> >>>>>>>>>>
>>> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>> >>>>>>>>>> [3] https://semver.org/
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3
>>> until we
>>> >>>>>>>>>>> deprecates finished(maybe one version is enough).
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old,
>>> and
>>> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for
>>> pure SQL user, the
>>> >>>>>>>>>>> dependencies is a little more complex.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client
>>> dependencies
>>> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Best,
>>> >>>>>>>>>>> Leonard Xu
>>> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> --
>>> >>>>>>>> Best Regards
>>> >>>>>>>> Michael Li
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>>
>>> >>>>>> Arvid Heise | Senior Java Developer
>>> >>>>>>
>>> >>>>>> <https://www.ververica.com/>
>>> >>>>>>
>>> >>>>>> Follow us @VervericaData
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>>
>>> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache
>>> Flink
>>> >>>>>> Conference
>>> >>>>>>
>>> >>>>>> Stream Processing | Event Driven | Real Time
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>>
>>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>> >>>>>> <
>>> https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g
>>> >
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Ververica GmbH
>>> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung
>>> Jason,
>>> >>>>>> Ji (Toni) Cheng
>>> >>>>>>
>>> >>>>>
>>>
>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Robert Metzger <rm...@apache.org>.
@Jark: Thanks for bringing up these concerns.
All the problems you've mentioned are "solvable":
- uber jar: Bahir could provide a hbase1 uber jar (we could theoretically
also add a dependency from flink to bahir and provide the uber jar from
Flink)
- e2e tests: we know that the connector is stable, as long as we are not
adding major changes (or we are moving the respective e2e tests to bahir).
On the other hand, I agree with you that supporting multiple versions of a
connector is pretty common (see Kafka or elasticsearch), so why can't we
allow it for Hbase now?
I'm really torn on this and would like to hear more opinions on this.
On Fri, Aug 7, 2020 at 11:24 PM Felipe Lolas <fl...@icloud.com> wrote:
> Hi all!
>
> Im new here; I have been using the flink connector for hbase 1.2, but
> recently opt to upgrading to hbase 2.1(basically because was bundled in
> CDH6)
>
> it would be nice to add support for hbase 2.x!
> I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and
> keeping that in mind last week I sent a PR with a solution supporting
> 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules
> sending the PR).
>
> i would be happy to help if needed!
>
>
>
> Felipe.
>
> El 07-08-2020, a la(s) 10:53, Jark Wu <im...@gmail.com> escribió:
>
>
> I'm +1 to add HBase 2.x
>
> However, I have some concerns about moving HBase 1.x to Bahir:
> 1) As discussed above, there are still lots of people using HBase 1.x.
> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E
> tests.
> 3) We also paid lots of effort to provide an uber connector jar for HBase
> (not yet released), it is helpful to improve the out-of-box experience.
>
> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It
> doesn't add too much work to maintain a new version.
> Keeping the old version can also help us to develop the new one. I would
> suggest to keep HBase 1.x in the repository for at least one more release.
> Another idea is that maybe it's a good time to have a
> "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to
> it.
> It would also be a good place to accept the contribution of pulsar
> connector and other connectors.
>
> Best,
> Jark
>
>
> On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rm...@apache.org> wrote:
>
>> Hi,
>>
>> Thank you for picking this up so quickly. I have no objections regarding
>> all the proposed items.
>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>> need somebody to merge it.
>>
>>
>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>> > Hi Robert and Gyula,
>> >
>> > Thanks for reviving this thread. We have the implementation (currently
>> for
>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
>> > recently written a readme for said version, he would be interested in
>> > contributing the upgraded connector back. The latest HBase version is
>> > 2.3.0, if we are touching the codebase anyway I would propose to have
>> that.
>> >
>> > If everyone is comfortable with it I would assign [1] to Miklos with
>> > double checking the all functionality that Felipe has proposed is
>> included.
>> > [1] https://issues.apache.org/jira/browse/FLINK-18795
>> > [2] https://hbase.apache.org/downloads.html
>> >
>> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com>
>> wrote:
>> >
>> >> Hi Robert,
>> >>
>> >> I completely agree with you on the Bahir based approach.
>> >>
>> >> I am happy to help with the contribution on the bahir side, with
>> thorough
>> >> review and testing.
>> >>
>> >> Cheers,
>> >> Gyula
>> >>
>> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org>
>> wrote:
>> >>
>> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
>> >>>
>> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>> >>> wrote:
>> >>>
>> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>> >>>> for adding HBase 2 support.
>> >>>>
>> >>>> it seems that there is demand for a HBase 2 connector, and consensus
>> to
>> >>>> do it.
>> >>>>
>> >>>> The remaining question in this thread seems to be the "how". I would
>> >>>> propose to go the other way around as Gyula suggested: We move the
>> legacy
>> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where
>> we
>> >>>> also run Hbase end to end tests. This will help us to stabilize the
>> new
>> >>>> connector and ensure a good quality.
>> >>>> It also, the perception of what goes into Flink, and what into Bahir
>> is
>> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>> >>>> legacy, experimental or unstable connectors into Bahir.
>> >>>>
>> >>>>
>> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>> >>>> review and contribution to Bahir)
>> >>>>
>> >>>>
>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>> >>>> [2] https://github.com/apache/flink/pull/13047
>> >>>>
>> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> If we were to go the bahir route, I don't see the point in migrating
>> >>>>> the 1.4.x version there since that's already available in Flink. To
>> me that
>> >>>>> is almost the same as dropping explicit support for 1.4 and telling
>> users
>> >>>>> to use older connector versions if they wish to keep using it.
>> >>>>>
>> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>> >>>>> that, we can do that inside Flink and only push the 2.4.x version
>> to bahir.
>> >>>>>
>> >>>>> What do you think?
>> >>>>>
>> >>>>> Gyula
>> >>>>>
>> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull
>> them
>> >>>>>> out to Bahir and list them in flink-packages to avoid adding even
>> more
>> >>>>>> modules to Flink core?
>> >>>>>>
>> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> Hi
>> >>>>>>>
>> >>>>>>> According to my observation in the hbase community, there are
>> still
>> >>>>>>> lots of hbase users running their production cluster with version
>> 1.x (1.4x
>> >>>>>>> or 1.5.x). so I'd like to suggest that
>> >>>>>>> supporting both hbase1.x & hbase2.x connector.
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we
>> are
>> >>>>>>>> doing for Kafka.
>> >>>>>>>>
>> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> One supplement:
>> >>>>>>>>>
>> >>>>>>>>> I noticed that there are discussions in HBase ML this March
>> about
>> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will
>> follow up in
>> >>>>>>>>> HBase community about why we didn't take real action. However,
>> this doesn't
>> >>>>>>>>> change my previous statement / stand due to the number of 1.x
>> usages in
>> >>>>>>>>> production.
>> >>>>>>>>>
>> >>>>>>>>> Best Regards,
>> >>>>>>>>> Yu
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>>>
>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>> >>>>>>>>>
>> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3
>> is
>> >>>>>>>>>> indeed an old version.
>> >>>>>>>>>>
>> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>> >>>>>>>>>> production. We could also see that the HBase community is
>> still maintaining
>> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>> >>>>>>>>>>
>> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2]
>> [3]
>> >>>>>>>>>> thus don't promise any kind of compatibility
>> (source/binary/wire, etc.)
>> >>>>>>>>>> between major versions. So if we only maintain 2.x connector,
>> it would not
>> >>>>>>>>>> be able to work with 1.x HBase clusters.
>> >>>>>>>>>>
>> >>>>>>>>>> I totally understand the additional efforts of maintaining two
>> >>>>>>>>>> modules, but since we're also reserving multiple versions for
>> kafka
>> >>>>>>>>>> connector, meantime considering the current HBase
>> in-production status, I'd
>> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>> >>>>>>>>>>
>> >>>>>>>>>> Best Regards,
>> >>>>>>>>>> Yu
>> >>>>>>>>>>
>> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>> >>>>>>>>>> [3] https://semver.org/
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until
>> we
>> >>>>>>>>>>> deprecates finished(maybe one version is enough).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure
>> SQL user, the
>> >>>>>>>>>>> dependencies is a little more complex.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>> >>>>>>>>>>>
>> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Leonard Xu
>> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Best Regards
>> >>>>>>>> Michael Li
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Arvid Heise | Senior Java Developer
>> >>>>>>
>> >>>>>> <https://www.ververica.com/>
>> >>>>>>
>> >>>>>> Follow us @VervericaData
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> >>>>>> Conference
>> >>>>>>
>> >>>>>> Stream Processing | Event Driven | Real Time
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> >>>>>> <
>> https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g
>> >
>> >>>>>>
>> >>>>>> --
>> >>>>>> Ververica GmbH
>> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung
>> Jason,
>> >>>>>> Ji (Toni) Cheng
>> >>>>>>
>> >>>>>
>>
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Robert Metzger <rm...@apache.org>.
@Jark: Thanks for bringing up these concerns.
All the problems you've mentioned are "solvable":
- uber jar: Bahir could provide a hbase1 uber jar (we could theoretically
also add a dependency from flink to bahir and provide the uber jar from
Flink)
- e2e tests: we know that the connector is stable, as long as we are not
adding major changes (or we are moving the respective e2e tests to bahir).
On the other hand, I agree with you that supporting multiple versions of a
connector is pretty common (see Kafka or elasticsearch), so why can't we
allow it for Hbase now?
I'm really torn on this and would like to hear more opinions on this.
On Fri, Aug 7, 2020 at 11:24 PM Felipe Lolas <fl...@icloud.com> wrote:
> Hi all!
>
> Im new here; I have been using the flink connector for hbase 1.2, but
> recently opt to upgrading to hbase 2.1(basically because was bundled in
> CDH6)
>
> it would be nice to add support for hbase 2.x!
> I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and
> keeping that in mind last week I sent a PR with a solution supporting
> 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules
> sending the PR).
>
> i would be happy to help if needed!
>
>
>
> Felipe.
>
> El 07-08-2020, a la(s) 10:53, Jark Wu <im...@gmail.com> escribió:
>
>
> I'm +1 to add HBase 2.x
>
> However, I have some concerns about moving HBase 1.x to Bahir:
> 1) As discussed above, there are still lots of people using HBase 1.x.
> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E
> tests.
> 3) We also paid lots of effort to provide an uber connector jar for HBase
> (not yet released), it is helpful to improve the out-of-box experience.
>
> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It
> doesn't add too much work to maintain a new version.
> Keeping the old version can also help us to develop the new one. I would
> suggest to keep HBase 1.x in the repository for at least one more release.
> Another idea is that maybe it's a good time to have a
> "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to
> it.
> It would also be a good place to accept the contribution of pulsar
> connector and other connectors.
>
> Best,
> Jark
>
>
> On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rm...@apache.org> wrote:
>
>> Hi,
>>
>> Thank you for picking this up so quickly. I have no objections regarding
>> all the proposed items.
>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>> need somebody to merge it.
>>
>>
>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>> > Hi Robert and Gyula,
>> >
>> > Thanks for reviving this thread. We have the implementation (currently
>> for
>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
>> > recently written a readme for said version, he would be interested in
>> > contributing the upgraded connector back. The latest HBase version is
>> > 2.3.0, if we are touching the codebase anyway I would propose to have
>> that.
>> >
>> > If everyone is comfortable with it I would assign [1] to Miklos with
>> > double checking the all functionality that Felipe has proposed is
>> included.
>> > [1] https://issues.apache.org/jira/browse/FLINK-18795
>> > [2] https://hbase.apache.org/downloads.html
>> >
>> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com>
>> wrote:
>> >
>> >> Hi Robert,
>> >>
>> >> I completely agree with you on the Bahir based approach.
>> >>
>> >> I am happy to help with the contribution on the bahir side, with
>> thorough
>> >> review and testing.
>> >>
>> >> Cheers,
>> >> Gyula
>> >>
>> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org>
>> wrote:
>> >>
>> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
>> >>>
>> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>> >>> wrote:
>> >>>
>> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>> >>>> for adding HBase 2 support.
>> >>>>
>> >>>> it seems that there is demand for a HBase 2 connector, and consensus
>> to
>> >>>> do it.
>> >>>>
>> >>>> The remaining question in this thread seems to be the "how". I would
>> >>>> propose to go the other way around as Gyula suggested: We move the
>> legacy
>> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where
>> we
>> >>>> also run Hbase end to end tests. This will help us to stabilize the
>> new
>> >>>> connector and ensure a good quality.
>> >>>> It also, the perception of what goes into Flink, and what into Bahir
>> is
>> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>> >>>> legacy, experimental or unstable connectors into Bahir.
>> >>>>
>> >>>>
>> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>> >>>> review and contribution to Bahir)
>> >>>>
>> >>>>
>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>> >>>> [2] https://github.com/apache/flink/pull/13047
>> >>>>
>> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> If we were to go the bahir route, I don't see the point in migrating
>> >>>>> the 1.4.x version there since that's already available in Flink. To
>> me that
>> >>>>> is almost the same as dropping explicit support for 1.4 and telling
>> users
>> >>>>> to use older connector versions if they wish to keep using it.
>> >>>>>
>> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>> >>>>> that, we can do that inside Flink and only push the 2.4.x version
>> to bahir.
>> >>>>>
>> >>>>> What do you think?
>> >>>>>
>> >>>>> Gyula
>> >>>>>
>> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull
>> them
>> >>>>>> out to Bahir and list them in flink-packages to avoid adding even
>> more
>> >>>>>> modules to Flink core?
>> >>>>>>
>> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> Hi
>> >>>>>>>
>> >>>>>>> According to my observation in the hbase community, there are
>> still
>> >>>>>>> lots of hbase users running their production cluster with version
>> 1.x (1.4x
>> >>>>>>> or 1.5.x). so I'd like to suggest that
>> >>>>>>> supporting both hbase1.x & hbase2.x connector.
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we
>> are
>> >>>>>>>> doing for Kafka.
>> >>>>>>>>
>> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> One supplement:
>> >>>>>>>>>
>> >>>>>>>>> I noticed that there are discussions in HBase ML this March
>> about
>> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will
>> follow up in
>> >>>>>>>>> HBase community about why we didn't take real action. However,
>> this doesn't
>> >>>>>>>>> change my previous statement / stand due to the number of 1.x
>> usages in
>> >>>>>>>>> production.
>> >>>>>>>>>
>> >>>>>>>>> Best Regards,
>> >>>>>>>>> Yu
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>>>
>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>> >>>>>>>>>
>> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3
>> is
>> >>>>>>>>>> indeed an old version.
>> >>>>>>>>>>
>> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>> >>>>>>>>>> production. We could also see that the HBase community is
>> still maintaining
>> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>> >>>>>>>>>>
>> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2]
>> [3]
>> >>>>>>>>>> thus don't promise any kind of compatibility
>> (source/binary/wire, etc.)
>> >>>>>>>>>> between major versions. So if we only maintain 2.x connector,
>> it would not
>> >>>>>>>>>> be able to work with 1.x HBase clusters.
>> >>>>>>>>>>
>> >>>>>>>>>> I totally understand the additional efforts of maintaining two
>> >>>>>>>>>> modules, but since we're also reserving multiple versions for
>> kafka
>> >>>>>>>>>> connector, meantime considering the current HBase
>> in-production status, I'd
>> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>> >>>>>>>>>>
>> >>>>>>>>>> Best Regards,
>> >>>>>>>>>> Yu
>> >>>>>>>>>>
>> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>> >>>>>>>>>> [3] https://semver.org/
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until
>> we
>> >>>>>>>>>>> deprecates finished(maybe one version is enough).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure
>> SQL user, the
>> >>>>>>>>>>> dependencies is a little more complex.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>> >>>>>>>>>>>
>> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Leonard Xu
>> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Best Regards
>> >>>>>>>> Michael Li
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Arvid Heise | Senior Java Developer
>> >>>>>>
>> >>>>>> <https://www.ververica.com/>
>> >>>>>>
>> >>>>>> Follow us @VervericaData
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> >>>>>> Conference
>> >>>>>>
>> >>>>>> Stream Processing | Event Driven | Real Time
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> >>>>>> <
>> https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g
>> >
>> >>>>>>
>> >>>>>> --
>> >>>>>> Ververica GmbH
>> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung
>> Jason,
>> >>>>>> Ji (Toni) Cheng
>> >>>>>>
>> >>>>>
>>
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Felipe Lolas <fl...@icloud.com>.
Hi all!
Im new here; I have been using the flink connector for hbase 1.2, but recently opt to upgrading to hbase 2.1(basically because was bundled in CDH6)
it would be nice to add support for hbase 2.x!
I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and keeping that in mind last week I sent a PR with a solution supporting 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules sending the PR).
i would be happy to help if needed!
Felipe.
> El 07-08-2020, a la(s) 10:53, Jark Wu <im...@gmail.com> escribió:
>
> I'm +1 to add HBase 2.x
>
> However, I have some concerns about moving HBase 1.x to Bahir:
> 1) As discussed above, there are still lots of people using HBase 1.x.
> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E tests.
> 3) We also paid lots of effort to provide an uber connector jar for HBase (not yet released), it is helpful to improve the out-of-box experience.
>
> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It doesn't add too much work to maintain a new version.
> Keeping the old version can also help us to develop the new one. I would suggest to keep HBase 1.x in the repository for at least one more release.
> Another idea is that maybe it's a good time to have a "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to it.
> It would also be a good place to accept the contribution of pulsar connector and other connectors.
>
> Best,
> Jark
>
>
> On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rm...@apache.org> wrote:
>> Hi,
>>
>> Thank you for picking this up so quickly. I have no objections regarding
>> all the proposed items.
>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>> need somebody to merge it.
>>
>>
>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <ba...@gmail.com>
>> wrote:
>>
>> > Hi Robert and Gyula,
>> >
>> > Thanks for reviving this thread. We have the implementation (currently for
>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
>> > recently written a readme for said version, he would be interested in
>> > contributing the upgraded connector back. The latest HBase version is
>> > 2.3.0, if we are touching the codebase anyway I would propose to have that.
>> >
>> > If everyone is comfortable with it I would assign [1] to Miklos with
>> > double checking the all functionality that Felipe has proposed is included.
>> > [1] https://issues.apache.org/jira/browse/FLINK-18795
>> > [2] https://hbase.apache.org/downloads.html
>> >
>> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com> wrote:
>> >
>> >> Hi Robert,
>> >>
>> >> I completely agree with you on the Bahir based approach.
>> >>
>> >> I am happy to help with the contribution on the bahir side, with thorough
>> >> review and testing.
>> >>
>> >> Cheers,
>> >> Gyula
>> >>
>> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org> wrote:
>> >>
>> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
>> >>>
>> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>> >>> wrote:
>> >>>
>> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>> >>>> for adding HBase 2 support.
>> >>>>
>> >>>> it seems that there is demand for a HBase 2 connector, and consensus to
>> >>>> do it.
>> >>>>
>> >>>> The remaining question in this thread seems to be the "how". I would
>> >>>> propose to go the other way around as Gyula suggested: We move the legacy
>> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>> >>>> also run Hbase end to end tests. This will help us to stabilize the new
>> >>>> connector and ensure a good quality.
>> >>>> It also, the perception of what goes into Flink, and what into Bahir is
>> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>> >>>> legacy, experimental or unstable connectors into Bahir.
>> >>>>
>> >>>>
>> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>> >>>> review and contribution to Bahir)
>> >>>>
>> >>>>
>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>> >>>> [2] https://github.com/apache/flink/pull/13047
>> >>>>
>> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> If we were to go the bahir route, I don't see the point in migrating
>> >>>>> the 1.4.x version there since that's already available in Flink. To me that
>> >>>>> is almost the same as dropping explicit support for 1.4 and telling users
>> >>>>> to use older connector versions if they wish to keep using it.
>> >>>>>
>> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>> >>>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>> >>>>>
>> >>>>> What do you think?
>> >>>>>
>> >>>>> Gyula
>> >>>>>
>> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>> >>>>>> out to Bahir and list them in flink-packages to avoid adding even more
>> >>>>>> modules to Flink core?
>> >>>>>>
>> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> Hi
>> >>>>>>>
>> >>>>>>> According to my observation in the hbase community, there are still
>> >>>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>> >>>>>>> or 1.5.x). so I'd like to suggest that
>> >>>>>>> supporting both hbase1.x & hbase2.x connector.
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>> >>>>>>>> doing for Kafka.
>> >>>>>>>>
>> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>>> One supplement:
>> >>>>>>>>>
>> >>>>>>>>> I noticed that there are discussions in HBase ML this March about
>> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>> >>>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>> >>>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>> >>>>>>>>> production.
>> >>>>>>>>>
>> >>>>>>>>> Best Regards,
>> >>>>>>>>> Yu
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>> >>>>>>>>>
>> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>> >>>>>>>>>> indeed an old version.
>> >>>>>>>>>>
>> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>> >>>>>>>>>> production. We could also see that the HBase community is still maintaining
>> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>> >>>>>>>>>>
>> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>> >>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>> >>>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>> >>>>>>>>>> be able to work with 1.x HBase clusters.
>> >>>>>>>>>>
>> >>>>>>>>>> I totally understand the additional efforts of maintaining two
>> >>>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>> >>>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>> >>>>>>>>>>
>> >>>>>>>>>> Best Regards,
>> >>>>>>>>>> Yu
>> >>>>>>>>>>
>> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>> >>>>>>>>>> [3] https://semver.org/
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>> >>>>>>>>>>> deprecates finished(maybe one version is enough).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>> >>>>>>>>>>> dependencies is a little more complex.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>> >>>>>>>>>>>
>> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> Best,
>> >>>>>>>>>>> Leonard Xu
>> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> Best Regards
>> >>>>>>>> Michael Li
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Arvid Heise | Senior Java Developer
>> >>>>>>
>> >>>>>> <https://www.ververica.com/>
>> >>>>>>
>> >>>>>> Follow us @VervericaData
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> >>>>>> Conference
>> >>>>>>
>> >>>>>> Stream Processing | Event Driven | Real Time
>> >>>>>>
>> >>>>>> --
>> >>>>>>
>> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> >>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Ververica GmbH
>> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>> >>>>>> Ji (Toni) Cheng
>> >>>>>>
>> >>>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Jark Wu <im...@gmail.com>.
I'm +1 to add HBase 2.x
However, I have some concerns about moving HBase 1.x to Bahir:
1) As discussed above, there are still lots of people using HBase 1.x.
2) Bahir doesn't have the infrastructure to run the existing HBase E2E
tests.
3) We also paid lots of effort to provide an uber connector jar for HBase
(not yet released), it is helpful to improve the out-of-box experience.
My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It
doesn't add too much work to maintain a new version.
Keeping the old version can also help us to develop the new one. I would
suggest to keep HBase 1.x in the repository for at least one more release.
Another idea is that maybe it's a good time to have a
"apache/flink-connectors" repository, and move both HBase 1.x and 2.x to
it.
It would also be a good place to accept the contribution of pulsar
connector and other connectors.
Best,
Jark
On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> Thank you for picking this up so quickly. I have no objections regarding
> all the proposed items.
> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
> need somebody to merge it.
>
>
> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hi Robert and Gyula,
> >
> > Thanks for reviving this thread. We have the implementation (currently
> for
> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
> > recently written a readme for said version, he would be interested in
> > contributing the upgraded connector back. The latest HBase version is
> > 2.3.0, if we are touching the codebase anyway I would propose to have
> that.
> >
> > If everyone is comfortable with it I would assign [1] to Miklos with
> > double checking the all functionality that Felipe has proposed is
> included.
> > [1] https://issues.apache.org/jira/browse/FLINK-18795
> > [2] https://hbase.apache.org/downloads.html
> >
> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com> wrote:
> >
> >> Hi Robert,
> >>
> >> I completely agree with you on the Bahir based approach.
> >>
> >> I am happy to help with the contribution on the bahir side, with
> thorough
> >> review and testing.
> >>
> >> Cheers,
> >> Gyula
> >>
> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org>
> wrote:
> >>
> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
> >>>
> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
> >>> wrote:
> >>>
> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
> >>>> for adding HBase 2 support.
> >>>>
> >>>> it seems that there is demand for a HBase 2 connector, and consensus
> to
> >>>> do it.
> >>>>
> >>>> The remaining question in this thread seems to be the "how". I would
> >>>> propose to go the other way around as Gyula suggested: We move the
> legacy
> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
> >>>> also run Hbase end to end tests. This will help us to stabilize the
> new
> >>>> connector and ensure a good quality.
> >>>> It also, the perception of what goes into Flink, and what into Bahir
> is
> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
> >>>> legacy, experimental or unstable connectors into Bahir.
> >>>>
> >>>>
> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
> >>>> review and contribution to Bahir)
> >>>>
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
> >>>> [2] https://github.com/apache/flink/pull/13047
> >>>>
> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> If we were to go the bahir route, I don't see the point in migrating
> >>>>> the 1.4.x version there since that's already available in Flink. To
> me that
> >>>>> is almost the same as dropping explicit support for 1.4 and telling
> users
> >>>>> to use older connector versions if they wish to keep using it.
> >>>>>
> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
> >>>>> that, we can do that inside Flink and only push the 2.4.x version to
> bahir.
> >>>>>
> >>>>> What do you think?
> >>>>>
> >>>>> Gyula
> >>>>>
> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
> >>>>> wrote:
> >>>>>
> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull
> them
> >>>>>> out to Bahir and list them in flink-packages to avoid adding even
> more
> >>>>>> modules to Flink core?
> >>>>>>
> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi
> >>>>>>>
> >>>>>>> According to my observation in the hbase community, there are still
> >>>>>>> lots of hbase users running their production cluster with version
> 1.x (1.4x
> >>>>>>> or 1.5.x). so I'd like to suggest that
> >>>>>>> supporting both hbase1.x & hbase2.x connector.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
> >>>>>>>> doing for Kafka.
> >>>>>>>>
> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> One supplement:
> >>>>>>>>>
> >>>>>>>>> I noticed that there are discussions in HBase ML this March about
> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow
> up in
> >>>>>>>>> HBase community about why we didn't take real action. However,
> this doesn't
> >>>>>>>>> change my previous statement / stand due to the number of 1.x
> usages in
> >>>>>>>>> production.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Yu
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>>
> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
> >>>>>>>>>
> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
> >>>>>>>>>> indeed an old version.
> >>>>>>>>>>
> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
> >>>>>>>>>> production. We could also see that the HBase community is still
> maintaining
> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
> >>>>>>>>>>
> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2]
> [3]
> >>>>>>>>>> thus don't promise any kind of compatibility
> (source/binary/wire, etc.)
> >>>>>>>>>> between major versions. So if we only maintain 2.x connector,
> it would not
> >>>>>>>>>> be able to work with 1.x HBase clusters.
> >>>>>>>>>>
> >>>>>>>>>> I totally understand the additional efforts of maintaining two
> >>>>>>>>>> modules, but since we're also reserving multiple versions for
> kafka
> >>>>>>>>>> connector, meantime considering the current HBase in-production
> status, I'd
> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Yu
> >>>>>>>>>>
> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html
> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
> >>>>>>>>>> [3] https://semver.org/
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until
> we
> >>>>>>>>>>> deprecates finished(maybe one version is enough).
> >>>>>>>>>>>
> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure
> SQL user, the
> >>>>>>>>>>> dependencies is a little more complex.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
> >>>>>>>>>>>
> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Leonard Xu
> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Best Regards
> >>>>>>>> Michael Li
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Arvid Heise | Senior Java Developer
> >>>>>>
> >>>>>> <https://www.ververica.com/>
> >>>>>>
> >>>>>> Follow us @VervericaData
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> >>>>>> Conference
> >>>>>>
> >>>>>> Stream Processing | Event Driven | Real Time
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> >>>>>> <
> https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g
> >
> >>>>>>
> >>>>>> --
> >>>>>> Ververica GmbH
> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
> >>>>>> Ji (Toni) Cheng
> >>>>>>
> >>>>>
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Jark Wu <im...@gmail.com>.
I'm +1 to add HBase 2.x
However, I have some concerns about moving HBase 1.x to Bahir:
1) As discussed above, there are still lots of people using HBase 1.x.
2) Bahir doesn't have the infrastructure to run the existing HBase E2E
tests.
3) We also paid lots of effort to provide an uber connector jar for HBase
(not yet released), it is helpful to improve the out-of-box experience.
My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It
doesn't add too much work to maintain a new version.
Keeping the old version can also help us to develop the new one. I would
suggest to keep HBase 1.x in the repository for at least one more release.
Another idea is that maybe it's a good time to have a
"apache/flink-connectors" repository, and move both HBase 1.x and 2.x to
it.
It would also be a good place to accept the contribution of pulsar
connector and other connectors.
Best,
Jark
On Fri, 7 Aug 2020 at 17:54, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> Thank you for picking this up so quickly. I have no objections regarding
> all the proposed items.
> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
> need somebody to merge it.
>
>
> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <ba...@gmail.com>
> wrote:
>
> > Hi Robert and Gyula,
> >
> > Thanks for reviving this thread. We have the implementation (currently
> for
> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
> > recently written a readme for said version, he would be interested in
> > contributing the upgraded connector back. The latest HBase version is
> > 2.3.0, if we are touching the codebase anyway I would propose to have
> that.
> >
> > If everyone is comfortable with it I would assign [1] to Miklos with
> > double checking the all functionality that Felipe has proposed is
> included.
> > [1] https://issues.apache.org/jira/browse/FLINK-18795
> > [2] https://hbase.apache.org/downloads.html
> >
> > On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com> wrote:
> >
> >> Hi Robert,
> >>
> >> I completely agree with you on the Bahir based approach.
> >>
> >> I am happy to help with the contribution on the bahir side, with
> thorough
> >> review and testing.
> >>
> >> Cheers,
> >> Gyula
> >>
> >> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org>
> wrote:
> >>
> >>> It seems that this thead is not on dev@ anymore. Adding it back ...
> >>>
> >>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
> >>> wrote:
> >>>
> >>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
> >>>> for adding HBase 2 support.
> >>>>
> >>>> it seems that there is demand for a HBase 2 connector, and consensus
> to
> >>>> do it.
> >>>>
> >>>> The remaining question in this thread seems to be the "how". I would
> >>>> propose to go the other way around as Gyula suggested: We move the
> legacy
> >>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
> >>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
> >>>> also run Hbase end to end tests. This will help us to stabilize the
> new
> >>>> connector and ensure a good quality.
> >>>> It also, the perception of what goes into Flink, and what into Bahir
> is
> >>>> a bit clearer if we put the stable, up to date stuff into Flink, and
> >>>> legacy, experimental or unstable connectors into Bahir.
> >>>>
> >>>>
> >>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
> >>>> review and contribution to Bahir)
> >>>>
> >>>>
> >>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
> >>>> [2] https://github.com/apache/flink/pull/13047
> >>>>
> >>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> If we were to go the bahir route, I don't see the point in migrating
> >>>>> the 1.4.x version there since that's already available in Flink. To
> me that
> >>>>> is almost the same as dropping explicit support for 1.4 and telling
> users
> >>>>> to use older connector versions if they wish to keep using it.
> >>>>>
> >>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
> >>>>> that, we can do that inside Flink and only push the 2.4.x version to
> bahir.
> >>>>>
> >>>>> What do you think?
> >>>>>
> >>>>> Gyula
> >>>>>
> >>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
> >>>>> wrote:
> >>>>>
> >>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull
> them
> >>>>>> out to Bahir and list them in flink-packages to avoid adding even
> more
> >>>>>> modules to Flink core?
> >>>>>>
> >>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Hi
> >>>>>>>
> >>>>>>> According to my observation in the hbase community, there are still
> >>>>>>> lots of hbase users running their production cluster with version
> 1.x (1.4x
> >>>>>>> or 1.5.x). so I'd like to suggest that
> >>>>>>> supporting both hbase1.x & hbase2.x connector.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
> >>>>>>>> doing for Kafka.
> >>>>>>>>
> >>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> One supplement:
> >>>>>>>>>
> >>>>>>>>> I noticed that there are discussions in HBase ML this March about
> >>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow
> up in
> >>>>>>>>> HBase community about why we didn't take real action. However,
> this doesn't
> >>>>>>>>> change my previous statement / stand due to the number of 1.x
> usages in
> >>>>>>>>> production.
> >>>>>>>>>
> >>>>>>>>> Best Regards,
> >>>>>>>>> Yu
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>>>
> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
> >>>>>>>>>
> >>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
> >>>>>>>>>> indeed an old version.
> >>>>>>>>>>
> >>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
> >>>>>>>>>> production. We could also see that the HBase community is still
> maintaining
> >>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
> >>>>>>>>>>
> >>>>>>>>>> Please also notice that HBase follows semantic versioning [2]
> [3]
> >>>>>>>>>> thus don't promise any kind of compatibility
> (source/binary/wire, etc.)
> >>>>>>>>>> between major versions. So if we only maintain 2.x connector,
> it would not
> >>>>>>>>>> be able to work with 1.x HBase clusters.
> >>>>>>>>>>
> >>>>>>>>>> I totally understand the additional efforts of maintaining two
> >>>>>>>>>> modules, but since we're also reserving multiple versions for
> kafka
> >>>>>>>>>> connector, meantime considering the current HBase in-production
> status, I'd
> >>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
> >>>>>>>>>>
> >>>>>>>>>> Best Regards,
> >>>>>>>>>> Yu
> >>>>>>>>>>
> >>>>>>>>>> [1] http://hbase.apache.org/downloads.html
> >>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
> >>>>>>>>>> [3] https://semver.org/
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until
> we
> >>>>>>>>>>> deprecates finished(maybe one version is enough).
> >>>>>>>>>>>
> >>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
> >>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure
> SQL user, the
> >>>>>>>>>>> dependencies is a little more complex.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
> >>>>>>>>>>>
> >>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
> >>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Best,
> >>>>>>>>>>> Leonard Xu
> >>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Best Regards
> >>>>>>>> Michael Li
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Arvid Heise | Senior Java Developer
> >>>>>>
> >>>>>> <https://www.ververica.com/>
> >>>>>>
> >>>>>> Follow us @VervericaData
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> >>>>>> Conference
> >>>>>>
> >>>>>> Stream Processing | Event Driven | Real Time
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> >>>>>> <
> https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g
> >
> >>>>>>
> >>>>>> --
> >>>>>> Ververica GmbH
> >>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> >>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
> >>>>>> Ji (Toni) Cheng
> >>>>>>
> >>>>>
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Robert Metzger <rm...@apache.org>.
Hi,
Thank you for picking this up so quickly. I have no objections regarding
all the proposed items.
@Gyula: Once the bahir contribution is properly reviewed, ping me if you
need somebody to merge it.
On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <ba...@gmail.com>
wrote:
> Hi Robert and Gyula,
>
> Thanks for reviving this thread. We have the implementation (currently for
> 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
> recently written a readme for said version, he would be interested in
> contributing the upgraded connector back. The latest HBase version is
> 2.3.0, if we are touching the codebase anyway I would propose to have that.
>
> If everyone is comfortable with it I would assign [1] to Miklos with
> double checking the all functionality that Felipe has proposed is included.
> [1] https://issues.apache.org/jira/browse/FLINK-18795
> [2] https://hbase.apache.org/downloads.html
>
> On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com> wrote:
>
>> Hi Robert,
>>
>> I completely agree with you on the Bahir based approach.
>>
>> I am happy to help with the contribution on the bahir side, with thorough
>> review and testing.
>>
>> Cheers,
>> Gyula
>>
>> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org> wrote:
>>
>>> It seems that this thead is not on dev@ anymore. Adding it back ...
>>>
>>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>>> wrote:
>>>
>>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>>>> for adding HBase 2 support.
>>>>
>>>> it seems that there is demand for a HBase 2 connector, and consensus to
>>>> do it.
>>>>
>>>> The remaining question in this thread seems to be the "how". I would
>>>> propose to go the other way around as Gyula suggested: We move the legacy
>>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>>>> also run Hbase end to end tests. This will help us to stabilize the new
>>>> connector and ensure a good quality.
>>>> It also, the perception of what goes into Flink, and what into Bahir is
>>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>>>> legacy, experimental or unstable connectors into Bahir.
>>>>
>>>>
>>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>>>> review and contribution to Bahir)
>>>>
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>>>> [2] https://github.com/apache/flink/pull/13047
>>>>
>>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
>>>> wrote:
>>>>
>>>>> If we were to go the bahir route, I don't see the point in migrating
>>>>> the 1.4.x version there since that's already available in Flink. To me that
>>>>> is almost the same as dropping explicit support for 1.4 and telling users
>>>>> to use older connector versions if they wish to keep using it.
>>>>>
>>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Gyula
>>>>>
>>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>>>>> wrote:
>>>>>
>>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>>>>> out to Bahir and list them in flink-packages to avoid adding even more
>>>>>> modules to Flink core?
>>>>>>
>>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> According to my observation in the hbase community, there are still
>>>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>>>>> or 1.5.x). so I'd like to suggest that
>>>>>>> supporting both hbase1.x & hbase2.x connector.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>>>>> doing for Kafka.
>>>>>>>>
>>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> One supplement:
>>>>>>>>>
>>>>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>>>>> production.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Yu
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>>>>
>>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>>>>> indeed an old version.
>>>>>>>>>>
>>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>>>>
>>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>>>>
>>>>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Yu
>>>>>>>>>>
>>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>>>>> [3] https://semver.org/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>>>>
>>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>>>>> dependencies is a little more complex.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>>>>
>>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Leonard Xu
>>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Michael Li
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Arvid Heise | Senior Java Developer
>>>>>>
>>>>>> <https://www.ververica.com/>
>>>>>>
>>>>>> Follow us @VervericaData
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>>>> Conference
>>>>>>
>>>>>> Stream Processing | Event Driven | Real Time
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>
>>>>>>
>>>>>> --
>>>>>> Ververica GmbH
>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>>>>>> Ji (Toni) Cheng
>>>>>>
>>>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Robert Metzger <rm...@apache.org>.
Hi,
Thank you for picking this up so quickly. I have no objections regarding
all the proposed items.
@Gyula: Once the bahir contribution is properly reviewed, ping me if you
need somebody to merge it.
On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi <ba...@gmail.com>
wrote:
> Hi Robert and Gyula,
>
> Thanks for reviving this thread. We have the implementation (currently for
> 2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
> recently written a readme for said version, he would be interested in
> contributing the upgraded connector back. The latest HBase version is
> 2.3.0, if we are touching the codebase anyway I would propose to have that.
>
> If everyone is comfortable with it I would assign [1] to Miklos with
> double checking the all functionality that Felipe has proposed is included.
> [1] https://issues.apache.org/jira/browse/FLINK-18795
> [2] https://hbase.apache.org/downloads.html
>
> On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com> wrote:
>
>> Hi Robert,
>>
>> I completely agree with you on the Bahir based approach.
>>
>> I am happy to help with the contribution on the bahir side, with thorough
>> review and testing.
>>
>> Cheers,
>> Gyula
>>
>> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org> wrote:
>>
>>> It seems that this thead is not on dev@ anymore. Adding it back ...
>>>
>>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>>> wrote:
>>>
>>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>>>> for adding HBase 2 support.
>>>>
>>>> it seems that there is demand for a HBase 2 connector, and consensus to
>>>> do it.
>>>>
>>>> The remaining question in this thread seems to be the "how". I would
>>>> propose to go the other way around as Gyula suggested: We move the legacy
>>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>>>> also run Hbase end to end tests. This will help us to stabilize the new
>>>> connector and ensure a good quality.
>>>> It also, the perception of what goes into Flink, and what into Bahir is
>>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>>>> legacy, experimental or unstable connectors into Bahir.
>>>>
>>>>
>>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>>>> review and contribution to Bahir)
>>>>
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>>>> [2] https://github.com/apache/flink/pull/13047
>>>>
>>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com>
>>>> wrote:
>>>>
>>>>> If we were to go the bahir route, I don't see the point in migrating
>>>>> the 1.4.x version there since that's already available in Flink. To me that
>>>>> is almost the same as dropping explicit support for 1.4 and telling users
>>>>> to use older connector versions if they wish to keep using it.
>>>>>
>>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>>>>>
>>>>> What do you think?
>>>>>
>>>>> Gyula
>>>>>
>>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>>>>> wrote:
>>>>>
>>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>>>>> out to Bahir and list them in flink-packages to avoid adding even more
>>>>>> modules to Flink core?
>>>>>>
>>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> According to my observation in the hbase community, there are still
>>>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>>>>> or 1.5.x). so I'd like to suggest that
>>>>>>> supporting both hbase1.x & hbase2.x connector.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>>>>> doing for Kafka.
>>>>>>>>
>>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> One supplement:
>>>>>>>>>
>>>>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>>>>> production.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Yu
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>>>>
>>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>>>>> indeed an old version.
>>>>>>>>>>
>>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>>>>
>>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>>>>
>>>>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Yu
>>>>>>>>>>
>>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>>>>> [3] https://semver.org/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>>>>
>>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>>>>> dependencies is a little more complex.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>>>>
>>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best,
>>>>>>>>>>> Leonard Xu
>>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Best Regards
>>>>>>>> Michael Li
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Arvid Heise | Senior Java Developer
>>>>>>
>>>>>> <https://www.ververica.com/>
>>>>>>
>>>>>> Follow us @VervericaData
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>>>> Conference
>>>>>>
>>>>>> Stream Processing | Event Driven | Real Time
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>
>>>>>>
>>>>>> --
>>>>>> Ververica GmbH
>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>>>>>> Ji (Toni) Cheng
>>>>>>
>>>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Márton Balassi <ba...@gmail.com>.
Hi Robert and Gyula,
Thanks for reviving this thread. We have the implementation (currently for
2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
recently written a readme for said version, he would be interested in
contributing the upgraded connector back. The latest HBase version is
2.3.0, if we are touching the codebase anyway I would propose to have that.
If everyone is comfortable with it I would assign [1] to Miklos with double
checking the all functionality that Felipe has proposed is included.
[1] https://issues.apache.org/jira/browse/FLINK-18795
[2] https://hbase.apache.org/downloads.html
On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com> wrote:
> Hi Robert,
>
> I completely agree with you on the Bahir based approach.
>
> I am happy to help with the contribution on the bahir side, with thorough
> review and testing.
>
> Cheers,
> Gyula
>
> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org> wrote:
>
>> It seems that this thead is not on dev@ anymore. Adding it back ...
>>
>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>> wrote:
>>
>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>>> for adding HBase 2 support.
>>>
>>> it seems that there is demand for a HBase 2 connector, and consensus to
>>> do it.
>>>
>>> The remaining question in this thread seems to be the "how". I would
>>> propose to go the other way around as Gyula suggested: We move the legacy
>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>>> also run Hbase end to end tests. This will help us to stabilize the new
>>> connector and ensure a good quality.
>>> It also, the perception of what goes into Flink, and what into Bahir is
>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>>> legacy, experimental or unstable connectors into Bahir.
>>>
>>>
>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>>> review and contribution to Bahir)
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>>> [2] https://github.com/apache/flink/pull/13047
>>>
>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com> wrote:
>>>
>>>> If we were to go the bahir route, I don't see the point in migrating
>>>> the 1.4.x version there since that's already available in Flink. To me that
>>>> is almost the same as dropping explicit support for 1.4 and telling users
>>>> to use older connector versions if they wish to keep using it.
>>>>
>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>>>>
>>>> What do you think?
>>>>
>>>> Gyula
>>>>
>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>>>> wrote:
>>>>
>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>>>> out to Bahir and list them in flink-packages to avoid adding even more
>>>>> modules to Flink core?
>>>>>
>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> According to my observation in the hbase community, there are still
>>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>>>> or 1.5.x). so I'd like to suggest that
>>>>>> supporting both hbase1.x & hbase2.x connector.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>>>>>
>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>>>> doing for Kafka.
>>>>>>>
>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>>>
>>>>>>>> One supplement:
>>>>>>>>
>>>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>>>> production.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Yu
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>>>
>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>>>> indeed an old version.
>>>>>>>>>
>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>>>
>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>>>
>>>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Yu
>>>>>>>>>
>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>>>> [3] https://semver.org/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>>>
>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>>>> dependencies is a little more complex.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>>>
>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Leonard Xu
>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> Michael Li
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Arvid Heise | Senior Java Developer
>>>>>
>>>>> <https://www.ververica.com/>
>>>>>
>>>>> Follow us @VervericaData
>>>>>
>>>>> --
>>>>>
>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>>> Conference
>>>>>
>>>>> Stream Processing | Event Driven | Real Time
>>>>>
>>>>> --
>>>>>
>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>
>>>>>
>>>>> --
>>>>> Ververica GmbH
>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>>>>> Ji (Toni) Cheng
>>>>>
>>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Márton Balassi <ba...@gmail.com>.
Hi Robert and Gyula,
Thanks for reviving this thread. We have the implementation (currently for
2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
recently written a readme for said version, he would be interested in
contributing the upgraded connector back. The latest HBase version is
2.3.0, if we are touching the codebase anyway I would propose to have that.
If everyone is comfortable with it I would assign [1] to Miklos with double
checking the all functionality that Felipe has proposed is included.
[1] https://issues.apache.org/jira/browse/FLINK-18795
[2] https://hbase.apache.org/downloads.html
On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra <gy...@gmail.com> wrote:
> Hi Robert,
>
> I completely agree with you on the Bahir based approach.
>
> I am happy to help with the contribution on the bahir side, with thorough
> review and testing.
>
> Cheers,
> Gyula
>
> On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org> wrote:
>
>> It seems that this thead is not on dev@ anymore. Adding it back ...
>>
>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org>
>> wrote:
>>
>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>>> for adding HBase 2 support.
>>>
>>> it seems that there is demand for a HBase 2 connector, and consensus to
>>> do it.
>>>
>>> The remaining question in this thread seems to be the "how". I would
>>> propose to go the other way around as Gyula suggested: We move the legacy
>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>>> also run Hbase end to end tests. This will help us to stabilize the new
>>> connector and ensure a good quality.
>>> It also, the perception of what goes into Flink, and what into Bahir is
>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>>> legacy, experimental or unstable connectors into Bahir.
>>>
>>>
>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>>> review and contribution to Bahir)
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>>> [2] https://github.com/apache/flink/pull/13047
>>>
>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com> wrote:
>>>
>>>> If we were to go the bahir route, I don't see the point in migrating
>>>> the 1.4.x version there since that's already available in Flink. To me that
>>>> is almost the same as dropping explicit support for 1.4 and telling users
>>>> to use older connector versions if they wish to keep using it.
>>>>
>>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>>>>
>>>> What do you think?
>>>>
>>>> Gyula
>>>>
>>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com>
>>>> wrote:
>>>>
>>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>>>> out to Bahir and list them in flink-packages to avoid adding even more
>>>>> modules to Flink core?
>>>>>
>>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> According to my observation in the hbase community, there are still
>>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>>>> or 1.5.x). so I'd like to suggest that
>>>>>> supporting both hbase1.x & hbase2.x connector.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>>>>>
>>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>>>> doing for Kafka.
>>>>>>>
>>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>>>
>>>>>>>> One supplement:
>>>>>>>>
>>>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>>>> production.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Yu
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>>>
>>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>>>> indeed an old version.
>>>>>>>>>
>>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>>>
>>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>>>
>>>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Yu
>>>>>>>>>
>>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>>>> [3] https://semver.org/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>>>
>>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and
>>>>>>>>>> I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>>>> dependencies is a little more complex.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>>>
>>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies
>>>>>>>>>> are simple and clear. Tbe hbase project shades them all
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Leonard Xu
>>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> Michael Li
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Arvid Heise | Senior Java Developer
>>>>>
>>>>> <https://www.ververica.com/>
>>>>>
>>>>> Follow us @VervericaData
>>>>>
>>>>> --
>>>>>
>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>>> Conference
>>>>>
>>>>> Stream Processing | Event Driven | Real Time
>>>>>
>>>>> --
>>>>>
>>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>
>>>>>
>>>>> --
>>>>> Ververica GmbH
>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason,
>>>>> Ji (Toni) Cheng
>>>>>
>>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Robert,
I completely agree with you on the Bahir based approach.
I am happy to help with the contribution on the bahir side, with thorough
review and testing.
Cheers,
Gyula
On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org> wrote:
> It seems that this thead is not on dev@ anymore. Adding it back ...
>
> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org> wrote:
>
>> I would like to revive this discussion. There's a new JIRA[1] + PR[2] for
>> adding HBase 2 support.
>>
>> it seems that there is demand for a HBase 2 connector, and consensus to
>> do it.
>>
>> The remaining question in this thread seems to be the "how". I would
>> propose to go the other way around as Gyula suggested: We move the legacy
>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>> also run Hbase end to end tests. This will help us to stabilize the new
>> connector and ensure a good quality.
>> It also, the perception of what goes into Flink, and what into Bahir is a
>> bit clearer if we put the stable, up to date stuff into Flink, and legacy,
>> experimental or unstable connectors into Bahir.
>>
>>
>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>> review and contribution to Bahir)
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>> [2] https://github.com/apache/flink/pull/13047
>>
>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com> wrote:
>>
>>> If we were to go the bahir route, I don't see the point in migrating the
>>> 1.4.x version there since that's already available in Flink. To me that is
>>> almost the same as dropping explicit support for 1.4 and telling users to
>>> use older connector versions if they wish to keep using it.
>>>
>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>>>
>>> What do you think?
>>>
>>> Gyula
>>>
>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com> wrote:
>>>
>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>>> out to Bahir and list them in flink-packages to avoid adding even more
>>>> modules to Flink core?
>>>>
>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> According to my observation in the hbase community, there are still
>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>>> or 1.5.x). so I'd like to suggest that
>>>>> supporting both hbase1.x & hbase2.x connector.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>>>>
>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>>> doing for Kafka.
>>>>>>
>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>>
>>>>>>> One supplement:
>>>>>>>
>>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>>> production.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Yu
>>>>>>>
>>>>>>> [1]
>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>>
>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>>> indeed an old version.
>>>>>>>>
>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>>
>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>>
>>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Yu
>>>>>>>>
>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>>> [3] https://semver.org/
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>>
>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>>>>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>>> dependencies is a little more complex.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>>
>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>>>>>>> simple and clear. Tbe hbase project shades them all
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Leonard Xu
>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>> Michael Li
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Arvid Heise | Senior Java Developer
>>>>
>>>> <https://www.ververica.com/>
>>>>
>>>> Follow us @VervericaData
>>>>
>>>> --
>>>>
>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>> Conference
>>>>
>>>> Stream Processing | Event Driven | Real Time
>>>>
>>>> --
>>>>
>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>
>>>>
>>>> --
>>>> Ververica GmbH
>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>>> (Toni) Cheng
>>>>
>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Robert,
I completely agree with you on the Bahir based approach.
I am happy to help with the contribution on the bahir side, with thorough
review and testing.
Cheers,
Gyula
On Fri, 7 Aug 2020 at 09:30, Robert Metzger <rm...@apache.org> wrote:
> It seems that this thead is not on dev@ anymore. Adding it back ...
>
> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org> wrote:
>
>> I would like to revive this discussion. There's a new JIRA[1] + PR[2] for
>> adding HBase 2 support.
>>
>> it seems that there is demand for a HBase 2 connector, and consensus to
>> do it.
>>
>> The remaining question in this thread seems to be the "how". I would
>> propose to go the other way around as Gyula suggested: We move the legacy
>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>> also run Hbase end to end tests. This will help us to stabilize the new
>> connector and ensure a good quality.
>> It also, the perception of what goes into Flink, and what into Bahir is a
>> bit clearer if we put the stable, up to date stuff into Flink, and legacy,
>> experimental or unstable connectors into Bahir.
>>
>>
>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>> review and contribution to Bahir)
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>> [2] https://github.com/apache/flink/pull/13047
>>
>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com> wrote:
>>
>>> If we were to go the bahir route, I don't see the point in migrating the
>>> 1.4.x version there since that's already available in Flink. To me that is
>>> almost the same as dropping explicit support for 1.4 and telling users to
>>> use older connector versions if they wish to keep using it.
>>>
>>> If we want to keep 1.4 around for legacy users and slowly deprecate
>>> that, we can do that inside Flink and only push the 2.4.x version to bahir.
>>>
>>> What do you think?
>>>
>>> Gyula
>>>
>>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com> wrote:
>>>
>>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>>> out to Bahir and list them in flink-packages to avoid adding even more
>>>> modules to Flink core?
>>>>
>>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> According to my observation in the hbase community, there are still
>>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>>> or 1.5.x). so I'd like to suggest that
>>>>> supporting both hbase1.x & hbase2.x connector.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>>>>
>>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>>> doing for Kafka.
>>>>>>
>>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>>
>>>>>>> One supplement:
>>>>>>>
>>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>>> production.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Yu
>>>>>>>
>>>>>>> [1]
>>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>>
>>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>>> indeed an old version.
>>>>>>>>
>>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>>
>>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>>
>>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Yu
>>>>>>>>
>>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>>> [3] https://semver.org/
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>>
>>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>>>>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>>> dependencies is a little more complex.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>>
>>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>>>>>>> simple and clear. Tbe hbase project shades them all
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Leonard Xu
>>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best Regards
>>>>>> Michael Li
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Arvid Heise | Senior Java Developer
>>>>
>>>> <https://www.ververica.com/>
>>>>
>>>> Follow us @VervericaData
>>>>
>>>> --
>>>>
>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>>> Conference
>>>>
>>>> Stream Processing | Event Driven | Real Time
>>>>
>>>> --
>>>>
>>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>> <https://www.google.com/maps/search/Invalidenstrasse+115,+10115+Berlin,+Germany?entry=gmail&source=g>
>>>>
>>>> --
>>>> Ververica GmbH
>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>>> (Toni) Cheng
>>>>
>>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Robert Metzger <rm...@apache.org>.
It seems that this thead is not on dev@ anymore. Adding it back ...
On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org> wrote:
> I would like to revive this discussion. There's a new JIRA[1] + PR[2] for
> adding HBase 2 support.
>
> it seems that there is demand for a HBase 2 connector, and consensus to do
> it.
>
> The remaining question in this thread seems to be the "how". I would
> propose to go the other way around as Gyula suggested: We move the legacy
> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
> Why? In the Flink repo, we have a pretty solid testing infra, where we
> also run Hbase end to end tests. This will help us to stabilize the new
> connector and ensure a good quality.
> It also, the perception of what goes into Flink, and what into Bahir is a
> bit clearer if we put the stable, up to date stuff into Flink, and legacy,
> experimental or unstable connectors into Bahir.
>
>
> Who can take care of this effort? (Decide which Hbase 2 PR to take, review
> and contribution to Bahir)
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-18795
> [2] https://github.com/apache/flink/pull/13047
>
> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com> wrote:
>
>> If we were to go the bahir route, I don't see the point in migrating the
>> 1.4.x version there since that's already available in Flink. To me that is
>> almost the same as dropping explicit support for 1.4 and telling users to
>> use older connector versions if they wish to keep using it.
>>
>> If we want to keep 1.4 around for legacy users and slowly deprecate that,
>> we can do that inside Flink and only push the 2.4.x version to bahir.
>>
>> What do you think?
>>
>> Gyula
>>
>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com> wrote:
>>
>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>> out to Bahir and list them in flink-packages to avoid adding even more
>>> modules to Flink core?
>>>
>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> According to my observation in the hbase community, there are still
>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>> or 1.5.x). so I'd like to suggest that
>>>> supporting both hbase1.x & hbase2.x connector.
>>>>
>>>> Thanks.
>>>>
>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>>>
>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>> doing for Kafka.
>>>>>
>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>
>>>>>> One supplement:
>>>>>>
>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>> production.
>>>>>>
>>>>>> Best Regards,
>>>>>> Yu
>>>>>>
>>>>>> [1]
>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>
>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>
>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>> indeed an old version.
>>>>>>>
>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>
>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>
>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Yu
>>>>>>>
>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>> [3] https://semver.org/
>>>>>>>
>>>>>>>
>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>
>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>>>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>> dependencies is a little more complex.
>>>>>>>>
>>>>>>>>
>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>
>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>>>>>> simple and clear. Tbe hbase project shades them all
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Leonard Xu
>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>> Michael Li
>>>>>
>>>>
>>>
>>> --
>>>
>>> Arvid Heise | Senior Java Developer
>>>
>>> <https://www.ververica.com/>
>>>
>>> Follow us @VervericaData
>>>
>>> --
>>>
>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>> Conference
>>>
>>> Stream Processing | Event Driven | Real Time
>>>
>>> --
>>>
>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>
>>> --
>>> Ververica GmbH
>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>> (Toni) Cheng
>>>
>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Robert Metzger <rm...@apache.org>.
It seems that this thead is not on dev@ anymore. Adding it back ...
On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger <rm...@apache.org> wrote:
> I would like to revive this discussion. There's a new JIRA[1] + PR[2] for
> adding HBase 2 support.
>
> it seems that there is demand for a HBase 2 connector, and consensus to do
> it.
>
> The remaining question in this thread seems to be the "how". I would
> propose to go the other way around as Gyula suggested: We move the legacy
> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
> Why? In the Flink repo, we have a pretty solid testing infra, where we
> also run Hbase end to end tests. This will help us to stabilize the new
> connector and ensure a good quality.
> It also, the perception of what goes into Flink, and what into Bahir is a
> bit clearer if we put the stable, up to date stuff into Flink, and legacy,
> experimental or unstable connectors into Bahir.
>
>
> Who can take care of this effort? (Decide which Hbase 2 PR to take, review
> and contribution to Bahir)
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-18795
> [2] https://github.com/apache/flink/pull/13047
>
> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com> wrote:
>
>> If we were to go the bahir route, I don't see the point in migrating the
>> 1.4.x version there since that's already available in Flink. To me that is
>> almost the same as dropping explicit support for 1.4 and telling users to
>> use older connector versions if they wish to keep using it.
>>
>> If we want to keep 1.4 around for legacy users and slowly deprecate that,
>> we can do that inside Flink and only push the 2.4.x version to bahir.
>>
>> What do you think?
>>
>> Gyula
>>
>> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com> wrote:
>>
>>> If we support both HBase 1 and 2, maybe it's a good time to pull them
>>> out to Bahir and list them in flink-packages to avoid adding even more
>>> modules to Flink core?
>>>
>>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>>
>>>> Hi
>>>>
>>>> According to my observation in the hbase community, there are still
>>>> lots of hbase users running their production cluster with version 1.x (1.4x
>>>> or 1.5.x). so I'd like to suggest that
>>>> supporting both hbase1.x & hbase2.x connector.
>>>>
>>>> Thanks.
>>>>
>>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>>>
>>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>>> doing for Kafka.
>>>>>
>>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>>
>>>>>> One supplement:
>>>>>>
>>>>>> I noticed that there are discussions in HBase ML this March about
>>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>>> production.
>>>>>>
>>>>>> Best Regards,
>>>>>> Yu
>>>>>>
>>>>>> [1]
>>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>>
>>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>>
>>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>>> indeed an old version.
>>>>>>>
>>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>>
>>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>>> be able to work with 1.x HBase clusters.
>>>>>>>
>>>>>>> I totally understand the additional efforts of maintaining two
>>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Yu
>>>>>>>
>>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>>> [3] https://semver.org/
>>>>>>>
>>>>>>>
>>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>>>>>
>>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>>
>>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>>>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>>> dependencies is a little more complex.
>>>>>>>>
>>>>>>>>
>>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>>
>>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>>>>>> simple and clear. Tbe hbase project shades them all
>>>>>>>>
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Leonard Xu
>>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> --
>>>>> Best Regards
>>>>> Michael Li
>>>>>
>>>>
>>>
>>> --
>>>
>>> Arvid Heise | Senior Java Developer
>>>
>>> <https://www.ververica.com/>
>>>
>>> Follow us @VervericaData
>>>
>>> --
>>>
>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>> Conference
>>>
>>> Stream Processing | Event Driven | Real Time
>>>
>>> --
>>>
>>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>
>>> --
>>> Ververica GmbH
>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>>> (Toni) Cheng
>>>
>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Robert Metzger <rm...@apache.org>.
I would like to revive this discussion. There's a new JIRA[1] + PR[2] for
adding HBase 2 support.
it seems that there is demand for a HBase 2 connector, and consensus to do
it.
The remaining question in this thread seems to be the "how". I would
propose to go the other way around as Gyula suggested: We move the legacy
connector (1.4x) to bahir and add the new (2.x.x) to Flink.
Why? In the Flink repo, we have a pretty solid testing infra, where we also
run Hbase end to end tests. This will help us to stabilize the new
connector and ensure a good quality.
It also, the perception of what goes into Flink, and what into Bahir is a
bit clearer if we put the stable, up to date stuff into Flink, and legacy,
experimental or unstable connectors into Bahir.
Who can take care of this effort? (Decide which Hbase 2 PR to take, review
and contribution to Bahir)
[1] https://issues.apache.org/jira/browse/FLINK-18795
[2] https://github.com/apache/flink/pull/13047
On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra <gy...@gmail.com> wrote:
> If we were to go the bahir route, I don't see the point in migrating the
> 1.4.x version there since that's already available in Flink. To me that is
> almost the same as dropping explicit support for 1.4 and telling users to
> use older connector versions if they wish to keep using it.
>
> If we want to keep 1.4 around for legacy users and slowly deprecate that,
> we can do that inside Flink and only push the 2.4.x version to bahir.
>
> What do you think?
>
> Gyula
>
> On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com> wrote:
>
>> If we support both HBase 1 and 2, maybe it's a good time to pull them out
>> to Bahir and list them in flink-packages to avoid adding even more modules
>> to Flink core?
>>
>> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> According to my observation in the hbase community, there are still lots
>>> of hbase users running their production cluster with version 1.x (1.4x or
>>> 1.5.x). so I'd like to suggest that
>>> supporting both hbase1.x & hbase2.x connector.
>>>
>>> Thanks.
>>>
>>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>>
>>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are
>>>> doing for Kafka.
>>>>
>>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>>
>>>>> One supplement:
>>>>>
>>>>> I noticed that there are discussions in HBase ML this March about
>>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>>> HBase community about why we didn't take real action. However, this doesn't
>>>>> change my previous statement / stand due to the number of 1.x usages in
>>>>> production.
>>>>>
>>>>> Best Regards,
>>>>> Yu
>>>>>
>>>>> [1]
>>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>>
>>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>>
>>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>>> indeed an old version.
>>>>>>
>>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>>> production. We could also see that the HBase community is still maintaining
>>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>>
>>>>>> Please also notice that HBase follows semantic versioning [2] [3]
>>>>>> thus don't promise any kind of compatibility (source/binary/wire, etc.)
>>>>>> between major versions. So if we only maintain 2.x connector, it would not
>>>>>> be able to work with 1.x HBase clusters.
>>>>>>
>>>>>> I totally understand the additional efforts of maintaining two
>>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>>
>>>>>> Best Regards,
>>>>>> Yu
>>>>>>
>>>>>> [1] http://hbase.apache.org/downloads.html
>>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>>> [3] https://semver.org/
>>>>>>
>>>>>>
>>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>>>>
>>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>>> deprecates finished(maybe one version is enough).
>>>>>>>
>>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>>> dependencies is a little more complex.
>>>>>>>
>>>>>>>
>>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>>
>>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>>>>> simple and clear. Tbe hbase project shades them all
>>>>>>>
>>>>>>>
>>>>>>> Best,
>>>>>>> Leonard Xu
>>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>>
>>>>>>>
>>>>
>>>> --
>>>> Best Regards
>>>> Michael Li
>>>>
>>>
>>
>> --
>>
>> Arvid Heise | Senior Java Developer
>>
>> <https://www.ververica.com/>
>>
>> Follow us @VervericaData
>>
>> --
>>
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>>
>> Stream Processing | Event Driven | Real Time
>>
>> --
>>
>> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>
>> --
>> Ververica GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
>> (Toni) Cheng
>>
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Gyula Fóra <gy...@gmail.com>.
If we were to go the bahir route, I don't see the point in migrating the
1.4.x version there since that's already available in Flink. To me that is
almost the same as dropping explicit support for 1.4 and telling users to
use older connector versions if they wish to keep using it.
If we want to keep 1.4 around for legacy users and slowly deprecate that,
we can do that inside Flink and only push the 2.4.x version to bahir.
What do you think?
Gyula
On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise <ar...@ververica.com> wrote:
> If we support both HBase 1 and 2, maybe it's a good time to pull them out
> to Bahir and list them in flink-packages to avoid adding even more modules
> to Flink core?
>
> On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
>
>> Hi
>>
>> According to my observation in the hbase community, there are still lots
>> of hbase users running their production cluster with version 1.x (1.4x or
>> 1.5.x). so I'd like to suggest that
>> supporting both hbase1.x & hbase2.x connector.
>>
>> Thanks.
>>
>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>>
>>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are doing
>>> for Kafka.
>>>
>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>>
>>>> One supplement:
>>>>
>>>> I noticed that there are discussions in HBase ML this March about
>>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>>> HBase community about why we didn't take real action. However, this doesn't
>>>> change my previous statement / stand due to the number of 1.x usages in
>>>> production.
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>> [1]
>>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>>
>>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>>
>>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is
>>>>> indeed an old version.
>>>>>
>>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in
>>>>> production. We could also see that the HBase community is still maintaining
>>>>> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>>>>>
>>>>> Please also notice that HBase follows semantic versioning [2] [3] thus
>>>>> don't promise any kind of compatibility (source/binary/wire, etc.) between
>>>>> major versions. So if we only maintain 2.x connector, it would not be able
>>>>> to work with 1.x HBase clusters.
>>>>>
>>>>> I totally understand the additional efforts of maintaining two
>>>>> modules, but since we're also reserving multiple versions for kafka
>>>>> connector, meantime considering the current HBase in-production status, I'd
>>>>> still suggest to get both 1.4.13 and 2.2.5 supported.
>>>>>
>>>>> Best Regards,
>>>>> Yu
>>>>>
>>>>> [1] http://hbase.apache.org/downloads.html
>>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>>> [3] https://semver.org/
>>>>>
>>>>>
>>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>>>
>>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>>> deprecates finished(maybe one version is enough).
>>>>>>
>>>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>>> dependencies is a little more complex.
>>>>>>
>>>>>>
>>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>>
>>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>>>> simple and clear. Tbe hbase project shades them all
>>>>>>
>>>>>>
>>>>>> Best,
>>>>>> Leonard Xu
>>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>>
>>>>>>
>>>
>>> --
>>> Best Regards
>>> Michael Li
>>>
>>
>
> --
>
> Arvid Heise | Senior Java Developer
>
> <https://www.ververica.com/>
>
> Follow us @VervericaData
>
> --
>
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
>
> Stream Processing | Event Driven | Real Time
>
> --
>
> Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>
> --
> Ververica GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
> (Toni) Cheng
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Arvid Heise <ar...@ververica.com>.
If we support both HBase 1 and 2, maybe it's a good time to pull them out
to Bahir and list them in flink-packages to avoid adding even more modules
to Flink core?
On Mon, Jun 22, 2020 at 4:05 AM OpenInx <op...@gmail.com> wrote:
> Hi
>
> According to my observation in the hbase community, there are still lots
> of hbase users running their production cluster with version 1.x (1.4x or
> 1.5.x). so I'd like to suggest that
> supporting both hbase1.x & hbase2.x connector.
>
> Thanks.
>
> On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
>
>> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are doing
>> for Kafka.
>>
>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>>
>>> One supplement:
>>>
>>> I noticed that there are discussions in HBase ML this March about
>>> removing stable-1 pointer and got consensus [1], and will follow up in
>>> HBase community about why we didn't take real action. However, this doesn't
>>> change my previous statement / stand due to the number of 1.x usages in
>>> production.
>>>
>>> Best Regards,
>>> Yu
>>>
>>> [1]
>>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>>
>>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>>
>>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is indeed
>>>> an old version.
>>>>
>>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in production.
>>>> We could also see that the HBase community is still maintaining 1.x release
>>>> lines (with "stable-1 release" point to 1.4.13) [1]
>>>>
>>>> Please also notice that HBase follows semantic versioning [2] [3] thus
>>>> don't promise any kind of compatibility (source/binary/wire, etc.) between
>>>> major versions. So if we only maintain 2.x connector, it would not be able
>>>> to work with 1.x HBase clusters.
>>>>
>>>> I totally understand the additional efforts of maintaining two modules,
>>>> but since we're also reserving multiple versions for kafka connector,
>>>> meantime considering the current HBase in-production status, I'd still
>>>> suggest to get both 1.4.13 and 2.2.5 supported.
>>>>
>>>> Best Regards,
>>>> Yu
>>>>
>>>> [1] http://hbase.apache.org/downloads.html
>>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>>> [3] https://semver.org/
>>>>
>>>>
>>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>>
>>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>>> deprecates finished(maybe one version is enough).
>>>>>
>>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>>> dependencies is a little more complex.
>>>>>
>>>>>
>>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>>
>>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>>> simple and clear. Tbe hbase project shades them all
>>>>>
>>>>>
>>>>> Best,
>>>>> Leonard Xu
>>>>> [1] https://github.com/apache/flink/pull/12687
>>>>>
>>>>>
>>
>> --
>> Best Regards
>> Michael Li
>>
>
--
Arvid Heise | Senior Java Developer
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji
(Toni) Cheng
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by OpenInx <op...@gmail.com>.
Hi
According to my observation in the hbase community, there are still lots of
hbase users running their production cluster with version 1.x (1.4x or
1.5.x). so I'd like to suggest that
supporting both hbase1.x & hbase2.x connector.
Thanks.
On Sat, Jun 20, 2020 at 2:41 PM Ming Li <br...@gmail.com> wrote:
> +1 to support both HBase 2.x and Hbase 1.4.x, just as what we are doing
> for Kafka.
>
> On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
>
>> One supplement:
>>
>> I noticed that there are discussions in HBase ML this March about
>> removing stable-1 pointer and got consensus [1], and will follow up in
>> HBase community about why we didn't take real action. However, this doesn't
>> change my previous statement / stand due to the number of 1.x usages in
>> production.
>>
>> Best Regards,
>> Yu
>>
>> [1]
>> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>>
>> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>>
>>> +1 on upgrading the HBase version of the connector, and 1.4.3 is indeed
>>> an old version.
>>>
>>> OTOH, AFAIK there're still quite some 1.x HBase clusters in production.
>>> We could also see that the HBase community is still maintaining 1.x release
>>> lines (with "stable-1 release" point to 1.4.13) [1]
>>>
>>> Please also notice that HBase follows semantic versioning [2] [3] thus
>>> don't promise any kind of compatibility (source/binary/wire, etc.) between
>>> major versions. So if we only maintain 2.x connector, it would not be able
>>> to work with 1.x HBase clusters.
>>>
>>> I totally understand the additional efforts of maintaining two modules,
>>> but since we're also reserving multiple versions for kafka connector,
>>> meantime considering the current HBase in-production status, I'd still
>>> suggest to get both 1.4.13 and 2.2.5 supported.
>>>
>>> Best Regards,
>>> Yu
>>>
>>> [1] http://hbase.apache.org/downloads.html
>>> [2] https://hbase.apache.org/book.html#hbase.versioning
>>> [3] https://semver.org/
>>>
>>>
>>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>>
>>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>>> deprecates finished(maybe one version is enough).
>>>>
>>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>>> dependencies is a little more complex.
>>>>
>>>>
>>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>>
>>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>>> simple and clear. Tbe hbase project shades them all
>>>>
>>>>
>>>> Best,
>>>> Leonard Xu
>>>> [1] https://github.com/apache/flink/pull/12687
>>>>
>>>>
>
> --
> Best Regards
> Michael Li
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Ming Li <br...@gmail.com>.
+1 to support both HBase 2.x and Hbase 1.4.x, just as what we are doing
for Kafka.
On Fri, Jun 19, 2020 at 4:02 PM Yu Li <ca...@gmail.com> wrote:
> One supplement:
>
> I noticed that there are discussions in HBase ML this March about removing
> stable-1 pointer and got consensus [1], and will follow up in HBase
> community about why we didn't take real action. However, this doesn't
> change my previous statement / stand due to the number of 1.x usages in
> production.
>
> Best Regards,
> Yu
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
>
> On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
>
>> +1 on upgrading the HBase version of the connector, and 1.4.3 is indeed
>> an old version.
>>
>> OTOH, AFAIK there're still quite some 1.x HBase clusters in production.
>> We could also see that the HBase community is still maintaining 1.x release
>> lines (with "stable-1 release" point to 1.4.13) [1]
>>
>> Please also notice that HBase follows semantic versioning [2] [3] thus
>> don't promise any kind of compatibility (source/binary/wire, etc.) between
>> major versions. So if we only maintain 2.x connector, it would not be able
>> to work with 1.x HBase clusters.
>>
>> I totally understand the additional efforts of maintaining two modules,
>> but since we're also reserving multiple versions for kafka connector,
>> meantime considering the current HBase in-production status, I'd still
>> suggest to get both 1.4.13 and 2.2.5 supported.
>>
>> Best Regards,
>> Yu
>>
>> [1] http://hbase.apache.org/downloads.html
>> [2] https://hbase.apache.org/book.html#hbase.versioning
>> [3] https://semver.org/
>>
>>
>> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>>
>>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>>> deprecates finished(maybe one version is enough).
>>>
>>> Currently we only support HBase 1.4.3 which is pretty old, and I’m
>>> making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>>> dependencies is a little more complex.
>>>
>>>
>>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>>
>>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are
>>> simple and clear. Tbe hbase project shades them all
>>>
>>>
>>> Best,
>>> Leonard Xu
>>> [1] https://github.com/apache/flink/pull/12687
>>>
>>>
--
Best Regards
Michael Li
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Yu Li <ca...@gmail.com>.
One supplement:
I noticed that there are discussions in HBase ML this March about removing
stable-1 pointer and got consensus [1], and will follow up in HBase
community about why we didn't take real action. However, this doesn't
change my previous statement / stand due to the number of 1.x usages in
production.
Best Regards,
Yu
[1]
http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3C30180be2-bd93-d414-a158-16c9c8d010f3@apache.org%3E
On Fri, 19 Jun 2020 at 15:54, Yu Li <ca...@gmail.com> wrote:
> +1 on upgrading the HBase version of the connector, and 1.4.3 is indeed an
> old version.
>
> OTOH, AFAIK there're still quite some 1.x HBase clusters in production. We
> could also see that the HBase community is still maintaining 1.x release
> lines (with "stable-1 release" point to 1.4.13) [1]
>
> Please also notice that HBase follows semantic versioning [2] [3] thus
> don't promise any kind of compatibility (source/binary/wire, etc.) between
> major versions. So if we only maintain 2.x connector, it would not be able
> to work with 1.x HBase clusters.
>
> I totally understand the additional efforts of maintaining two modules,
> but since we're also reserving multiple versions for kafka connector,
> meantime considering the current HBase in-production status, I'd still
> suggest to get both 1.4.13 and 2.2.5 supported.
>
> Best Regards,
> Yu
>
> [1] http://hbase.apache.org/downloads.html
> [2] https://hbase.apache.org/book.html#hbase.versioning
> [3] https://semver.org/
>
>
> On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
>
>> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
>> deprecates finished(maybe one version is enough).
>>
>> Currently we only support HBase 1.4.3 which is pretty old, and I’m making
>> a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
>> dependencies is a little more complex.
>>
>>
>> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>>
>> + 1 to support HBase 2.x and the hbase 2.x client dependencies are simple
>> and clear. Tbe hbase project shades them all
>>
>>
>> Best,
>> Leonard Xu
>> [1] https://github.com/apache/flink/pull/12687
>>
>>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Yu Li <ca...@gmail.com>.
+1 on upgrading the HBase version of the connector, and 1.4.3 is indeed an
old version.
OTOH, AFAIK there're still quite some 1.x HBase clusters in production. We
could also see that the HBase community is still maintaining 1.x release
lines (with "stable-1 release" point to 1.4.13) [1]
Please also notice that HBase follows semantic versioning [2] [3] thus
don't promise any kind of compatibility (source/binary/wire, etc.) between
major versions. So if we only maintain 2.x connector, it would not be able
to work with 1.x HBase clusters.
I totally understand the additional efforts of maintaining two modules, but
since we're also reserving multiple versions for kafka connector, meantime
considering the current HBase in-production status, I'd still suggest to
get both 1.4.13 and 2.2.5 supported.
Best Regards,
Yu
[1] http://hbase.apache.org/downloads.html
[2] https://hbase.apache.org/book.html#hbase.versioning
[3] https://semver.org/
On Fri, 19 Jun 2020 at 14:58, Leonard Xu <xb...@gmail.com> wrote:
> +1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we
> deprecates finished(maybe one version is enough).
>
> Currently we only support HBase 1.4.3 which is pretty old, and I’m making
> a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the
> dependencies is a little more complex.
>
>
> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>
> + 1 to support HBase 2.x and the hbase 2.x client dependencies are simple
> and clear. Tbe hbase project shades them all
>
>
> Best,
> Leonard Xu
> [1] https://github.com/apache/flink/pull/12687
>
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Leonard Xu <xb...@gmail.com>.
+1 to support HBase 2.2.x, and +1 to retain HBase 1.4.3 until we deprecates finished(maybe one version is enough).
Currently we only support HBase 1.4.3 which is pretty old, and I’m making a flink-sql-connector-hbase[1] shaded jar for pure SQL user, the dependencies is a little more complex.
> 在 2020年6月19日，14:20，jackylau <li...@gmail.com> 写道：
>
> + 1 to support HBase 2.x and the hbase 2.x client dependencies are simple and clear. Tbe hbase project shades them all
>
Best,
Leonard Xu
[1] https://github.com/apache/flink/pull/12687 <https://github.com/apache/flink/pull/12687>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by jackylau <li...@gmail.com>.
+ 1 to support HBase 2.xand the hbase 2.x client dependencies are simple and
clear. Tbe hbase project shades them all
Jark Wu-3 wrote
> +1 to support HBase 2.xBut not sure about dropping support for 1.4.xI
> cc'ed to user@ and user-zh@ to hear more feedback from users.Best,JarkOn
> Thu, 18 Jun 2020 at 21:25, Gyula Fóra <
> gyula.fora@
> > wrote:> Hi All!>> I would like to revive an old ticket>
> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion
> around> upgrading the HBase version of the connector.>> The current HBase
> version is 1.4.3 which is over 2 years old at this point> and incompatible
> with the newer HBase versions used at many companies.>> We propose to
> upgrade the connector to the latest version and drop support> for the old
> version starting from the 1.12 Flink release. This would help> us maintain
> and improve the HBase connector over time.>> If the community agrees we
> are happy to contribute this upgrade as we have> already developed and
> tested the updated version.>> Cheers,> Gyula>
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Gyula Fóra <gy...@gmail.com>.
Hi!
Having both 1.4.x and 2.x supported means we need different modules or some
shim logic as they are not compatible with each other. I would love to
avoid this if possible because it is a lot of extra effort from a
maintainability perspective.
It would be great to see how many users still use HBase 1.4.3 and how many
would use a 2.2.x connector instead.
Gyula
On Fri, Jun 19, 2020 at 7:44 AM chaojianok <ch...@163.com> wrote:
> +1 to support HBase 2.x
> And I think the 1.4.x version can be retained for the time being, so that
> users who are currently using the 1.4.x version can have more time to
> evaluate whether their projects need to be upgraded and the cost of
> upgrading.
>
>
>
>
>
>
> At 2020-06-19 12:35:36, "Jark Wu" <im...@gmail.com> wrote:
>
> +1 to support HBase 2.x
> But not sure about dropping support for 1.4.x
>
> I cc'ed to user@ and user-zh@ to hear more feedback from users.
>
> Best,
> Jark
>
> On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
>
>> Hi All!
>>
>> I would like to revive an old ticket
>> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
>> upgrading the HBase version of the connector.
>>
>> The current HBase version is 1.4.3 which is over 2 years old at this point
>> and incompatible with the newer HBase versions used at many companies.
>>
>> We propose to upgrade the connector to the latest version and drop support
>> for the old version starting from the 1.12 Flink release. This would help
>> us maintain and improve the HBase connector over time.
>>
>> If the community agrees we are happy to contribute this upgrade as we have
>> already developed and tested the updated version.
>>
>> Cheers,
>> Gyula
>>
>
>
>
>
Re: Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Gyula Fóra <gy...@gmail.com>.
Hi!
Having both 1.4.x and 2.x supported means we need different modules or some
shim logic as they are not compatible with each other. I would love to
avoid this if possible because it is a lot of extra effort from a
maintainability perspective.
It would be great to see how many users still use HBase 1.4.3 and how many
would use a 2.2.x connector instead.
Gyula
On Fri, Jun 19, 2020 at 7:44 AM chaojianok <ch...@163.com> wrote:
> +1 to support HBase 2.x
> And I think the 1.4.x version can be retained for the time being, so that
> users who are currently using the 1.4.x version can have more time to
> evaluate whether their projects need to be upgraded and the cost of
> upgrading.
>
>
>
>
>
>
> At 2020-06-19 12:35:36, "Jark Wu" <im...@gmail.com> wrote:
>
> +1 to support HBase 2.x
> But not sure about dropping support for 1.4.x
>
> I cc'ed to user@ and user-zh@ to hear more feedback from users.
>
> Best,
> Jark
>
> On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
>
>> Hi All!
>>
>> I would like to revive an old ticket
>> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
>> upgrading the HBase version of the connector.
>>
>> The current HBase version is 1.4.3 which is over 2 years old at this point
>> and incompatible with the newer HBase versions used at many companies.
>>
>> We propose to upgrade the connector to the latest version and drop support
>> for the old version starting from the 1.12 Flink release. This would help
>> us maintain and improve the HBase connector over time.
>>
>> If the community agrees we are happy to contribute this upgrade as we have
>> already developed and tested the updated version.
>>
>> Cheers,
>> Gyula
>>
>
>
>
>
Re: Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Gyula Fóra <gy...@gmail.com>.
Hi!
Having both 1.4.x and 2.x supported means we need different modules or some
shim logic as they are not compatible with each other. I would love to
avoid this if possible because it is a lot of extra effort from a
maintainability perspective.
It would be great to see how many users still use HBase 1.4.3 and how many
would use a 2.2.x connector instead.
Gyula
On Fri, Jun 19, 2020 at 7:44 AM chaojianok <ch...@163.com> wrote:
> +1 to support HBase 2.x
> And I think the 1.4.x version can be retained for the time being, so that
> users who are currently using the 1.4.x version can have more time to
> evaluate whether their projects need to be upgraded and the cost of
> upgrading.
>
>
>
>
>
>
> At 2020-06-19 12:35:36, "Jark Wu" <im...@gmail.com> wrote:
>
> +1 to support HBase 2.x
> But not sure about dropping support for 1.4.x
>
> I cc'ed to user@ and user-zh@ to hear more feedback from users.
>
> Best,
> Jark
>
> On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
>
>> Hi All!
>>
>> I would like to revive an old ticket
>> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
>> upgrading the HBase version of the connector.
>>
>> The current HBase version is 1.4.3 which is over 2 years old at this point
>> and incompatible with the newer HBase versions used at many companies.
>>
>> We propose to upgrade the connector to the latest version and drop support
>> for the old version starting from the 1.12 Flink release. This would help
>> us maintain and improve the HBase connector over time.
>>
>> If the community agrees we are happy to contribute this upgrade as we have
>> already developed and tested the updated version.
>>
>> Cheers,
>> Gyula
>>
>
>
>
>
Re:Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by chaojianok <ch...@163.com>.
+1 to support HBase 2.x
And I think the 1.4.x version can be retained for the time being, so that users who are currently using the 1.4.x version can have more time to evaluate whether their projects need to be upgraded and the cost of upgrading.
At 2020-06-19 12:35:36, "Jark Wu" <im...@gmail.com> wrote:
+1 to support HBase 2.x
But not sure about dropping support for 1.4.x
I cc'ed to user@ and user-zh@ to hear more feedback from users.
Best,
Jark
On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
Hi All!
I would like to revive an old ticket
<https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
upgrading the HBase version of the connector.
The current HBase version is 1.4.3 which is over 2 years old at this point
and incompatible with the newer HBase versions used at many companies.
We propose to upgrade the connector to the latest version and drop support
for the old version starting from the 1.12 Flink release. This would help
us maintain and improve the HBase connector over time.
If the community agrees we are happy to contribute this upgrade as we have
already developed and tested the updated version.
Cheers,
Gyula
Re:Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by chaojianok <ch...@163.com>.
+1 to support HBase 2.x
And I think the 1.4.x version can be retained for the time being, so that users who are currently using the 1.4.x version can have more time to evaluate whether their projects need to be upgraded and the cost of upgrading.
At 2020-06-19 12:35:36, "Jark Wu" <im...@gmail.com> wrote:
+1 to support HBase 2.x
But not sure about dropping support for 1.4.x
I cc'ed to user@ and user-zh@ to hear more feedback from users.
Best,
Jark
On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
Hi All!
I would like to revive an old ticket
<https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
upgrading the HBase version of the connector.
The current HBase version is 1.4.3 which is over 2 years old at this point
and incompatible with the newer HBase versions used at many companies.
We propose to upgrade the connector to the latest version and drop support
for the old version starting from the 1.12 Flink release. This would help
us maintain and improve the HBase connector over time.
If the community agrees we are happy to contribute this upgrade as we have
already developed and tested the updated version.
Cheers,
Gyula
Re:Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by chaojianok <ch...@163.com>.
+1 to support HBase 2.x
And I think the 1.4.x version can be retained for the time being, so that users who are currently using the 1.4.x version can have more time to evaluate whether their projects need to be upgraded and the cost of upgrading.
At 2020-06-19 12:35:36, "Jark Wu" <im...@gmail.com> wrote:
+1 to support HBase 2.x
But not sure about dropping support for 1.4.x
I cc'ed to user@ and user-zh@ to hear more feedback from users.
Best,
Jark
On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
Hi All!
I would like to revive an old ticket
<https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
upgrading the HBase version of the connector.
The current HBase version is 1.4.3 which is over 2 years old at this point
and incompatible with the newer HBase versions used at many companies.
We propose to upgrade the connector to the latest version and drop support
for the old version starting from the 1.12 Flink release. This would help
us maintain and improve the HBase connector over time.
If the community agrees we are happy to contribute this upgrade as we have
already developed and tested the updated version.
Cheers,
Gyula
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Jark Wu <im...@gmail.com>.
+1 to support HBase 2.x
But not sure about dropping support for 1.4.x
I cc'ed to user@ and user-zh@ to hear more feedback from users.
Best,
Jark
On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
> Hi All!
>
> I would like to revive an old ticket
> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
> upgrading the HBase version of the connector.
>
> The current HBase version is 1.4.3 which is over 2 years old at this point
> and incompatible with the newer HBase versions used at many companies.
>
> We propose to upgrade the connector to the latest version and drop support
> for the old version starting from the 1.12 Flink release. This would help
> us maintain and improve the HBase connector over time.
>
> If the community agrees we are happy to contribute this upgrade as we have
> already developed and tested the updated version.
>
> Cheers,
> Gyula
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Jark Wu <im...@gmail.com>.
+1 to support HBase 2.x
But not sure about dropping support for 1.4.x
I cc'ed to user@ and user-zh@ to hear more feedback from users.
Best,
Jark
On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
> Hi All!
>
> I would like to revive an old ticket
> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
> upgrading the HBase version of the connector.
>
> The current HBase version is 1.4.3 which is over 2 years old at this point
> and incompatible with the newer HBase versions used at many companies.
>
> We propose to upgrade the connector to the latest version and drop support
> for the old version starting from the 1.12 Flink release. This would help
> us maintain and improve the HBase connector over time.
>
> If the community agrees we are happy to contribute this upgrade as we have
> already developed and tested the updated version.
>
> Cheers,
> Gyula
>
Re: [DISCUSS] Upgrade HBase connector to 2.2.x
Posted by Jark Wu <im...@gmail.com>.
+1 to support HBase 2.x
But not sure about dropping support for 1.4.x
I cc'ed to user@ and user-zh@ to hear more feedback from users.
Best,
Jark
On Thu, 18 Jun 2020 at 21:25, Gyula Fóra <gy...@gmail.com> wrote:
> Hi All!
>
> I would like to revive an old ticket
> <https://issues.apache.org/jira/browse/FLINK-9849> and discussion around
> upgrading the HBase version of the connector.
>
> The current HBase version is 1.4.3 which is over 2 years old at this point
> and incompatible with the newer HBase versions used at many companies.
>
> We propose to upgrade the connector to the latest version and drop support
> for the old version starting from the 1.12 Flink release. This would help
> us maintain and improve the HBase connector over time.
>
> If the community agrees we are happy to contribute this upgrade as we have
> already developed and tested the updated version.
>
> Cheers,
> Gyula
>