You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "张铎(Duo Zhang)" <pa...@gmail.com> on 2022/03/01 14:07:27 UTC
Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support
So after HBASE-26773, we still have another problem that we can not
use sun.nio.ch.DirectBuffer. The main usage is to get the address.
Let me provide a new util method in the hbase-unsafe module.
Andrew Purtell <an...@gmail.com> 于2022年2月26日周六 13:58写道：
> That sounds good, looking forward to it. Please let me know if you want
> help.
>
> > On Feb 25, 2022, at 9:56 PM, 张铎 <pa...@gmail.com> wrote:
> >
> > Filed HBASE-26773 and tried, the result is not very good. Netty's
> > PlatformDependent can not meet all the requirements.
> >
> > So now I prefer we have a new hbase-thirdparty-unsafe module to provide
> the
> > Unsafe ability without exposing sun.misc.Unsafe directly.
> >
> > Will give it a try.
> >
> > Thanks.
> >
> > Andrew Purtell <ap...@apache.org> 于2022年2月24日周四 10:46写道：
> >
> >> Ok, a bridge in one place would minimize the risk.
> >>
> >>> On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <pa...@gmail.com>
> >>> wrote:
> >>>
> >>> I think this is a reasonable concern, we could do this on our own.
> >>>
> >>> But checking the commit history of PlatformDependent[1], the API is
> >> pretty
> >>> stable, so I think the risk of hitting a security issue but we can not
> >>> upgrade due to API conflicts is very low. And if a security issue
> >> requires
> >>> big changes on PlatformDependent, then I assume we have to fix our own
> >>> version as well...
> >>>
> >>> And we will not spread PlatformDependent references everywhere in our
> >> code,
> >>> we will still have a bridge, like the current UnsafeAccess. So if later
> >> we
> >>> decide to implement our own version, it will not introduce a big impact
> >> to
> >>> our code base.
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> [1]
> >>>
> >>>
> >>
> https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java
> >>>
> >>> Andrew Purtell <ap...@apache.org> 于2022年2月24日周四 01:12写道：
> >>>
> >>>> We often upgrade netty after there is a CVE announcement. This
> >> produces a
> >>>> pressure to upgrade in production. Use of an internal API is not a
> good
> >>>> practice because it can cause friction because of breaking changes
> when
> >>>> there is a need to upgrade all of a sudden. This is painful and
> >> hopefully
> >>>> we can exclude it as a possibility now when coming up with the plan.
> We
> >>> can
> >>>> do the same thing they do in our own hbase-thirdparty without taking
> >> the
> >>>> dependency and achieve the same goal, right?
> >>>>
> >>>> On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <pa...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> For me, since we shade netty, there will be no conflict with other
> >>> netty
> >>>>> dependencies, so we can make sure the netty version we used is
> >> exactly
> >>>> the
> >>>>> one we shaded in hbase-thirdparty.
> >>>>>
> >>>>> If later we want to upgrade netty version in hbase-thirdparty, and it
> >>>>> breaks the api, then we can release a new hbase-thirdparty, and
> >> upgrade
> >>>> the
> >>>>> main hbase repo to depend on the new hbase-thirdparty version, and
> >> also
> >>>>> modify the code in hbase accordingly. There will be no user visible
> >>>>> affects.
> >>>>>
> >>>>> Anyway, let me have a try first. Even with netty's PaltformDependent
> >>>> class,
> >>>>> since it still references sun.misc.Unsafe, maybe we will still get an
> >>>>> compile error...
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> Andrew Purtell <ap...@apache.org> 于2022年2月22日周二 08:08写道：
> >>>>>
> >>>>>> Thanks for that. I understand better what you are proposing now. It
> >>> is
> >>>>>> clear that you have thought it through.
> >>>>>>
> >>>>>> So then the only question I have is this: Do you think it is a
> >>> concern
> >>>>> that
> >>>>>> Netty's PlatformDependent class is in their
> >> "io.netty.util.internal"
> >>>>>> package? The 'internal' designation suggests to me they won't apply
> >>> the
> >>>>>> same care to not breaking users of it as they would for their
> >> public
> >>>>>> (non-"internal") API. I noticed this earlier but had other
> >> questions
> >>> at
> >>>>>> that time.
> >>>>>>
> >>>>>> If it is a concern, we have the option of doing the same thing that
> >>>> Netty
> >>>>>> has done with their PlatformDependent class as a new utility class
> >> of
> >>>> our
> >>>>>> own in hbase-thirdparty. This may impose special requirements on
> >>>> building
> >>>>>> hbase-thirdparty, or, at least, the module containing this wrapper,
> >>> but
> >>>>> it
> >>>>>> is an option that would help us avoid external breaking changes.
> >> For
> >>>> your
> >>>>>> consideration.
> >>>>>>
> >>>>>> On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) <
> >> palomino219@gmail.com
> >>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> What I proposed here is for solving the problem…
> >>>>>>>
> >>>>>>> The problem is that when using —release 8, javac will not export
> >>> the
> >>>>>>> jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a
> >>>>> compile
> >>>>>>> error. But at runtime if you export jdk.unsupported you can still
> >>> use
> >>>>>>> sun.misc.Unsafe. The related jdk issue said —release is only for
> >>>> public
> >>>>>> API
> >>>>>>> so not exporting jdk.unsupported is the expected behavior.
> >>>>>>>
> >>>>>>> So what I proposed here is to remove direct dependency on
> >>>>> sun.misc.Unsafe
> >>>>>>> in HBase code, so at compile time there will be no problem. Betty
> >>>> has a
> >>>>>>> wrapper of Unsafe so I think we could have a try. At runtime if
> >> we
> >>>> have
> >>>>>>> jdk.unsupported exported we could still actually make use of
> >>>>>>> sun.misc.Unsafe.
> >>>>>>>
> >>>>>>> Feel free to correct me if I missed something or there are still
> >>>>>> uncleared
> >>>>>>> things.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Andrew Purtell <an...@gmail.com>于2022年2月22日 周二00:56写道：
> >>>>>>>
> >>>>>>>> As Nick discovered ‘—release’ doesn’t work for version 8
> >> anyway.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Feb 20, 2022, at 3:47 PM, 张铎 <pa...@gmail.com>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>> I know, with —release 8 ByteBuffer will not be an problem
> >> but
> >>>> then
> >>>>>>>>> sun.misc.Unsafe can not be used any more right? So the
> >> problem
> >>>> here
> >>>>>> is
> >>>>>>>> how
> >>>>>>>>> to make use of Unsafe with —release 8. What I proposed here
> >> is
> >>>> that
> >>>>>> we
> >>>>>>>> just
> >>>>>>>>> do not use it, then we can make use of —release 8 to address
> >>> the
> >>>>>>>> ByteBuffer
> >>>>>>>>> problem.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>> Andrew Purtell <ap...@apache.org>于2022年2月21日 周一03:32写道：
> >>>>>>>>>
> >>>>>>>>>> The problem that leads us to consider '--release 8', or what
> >>> it
> >>>>>> would
> >>>>>>>>>> promise if it worked, isn't Unsafe, it is ByteBuffer.
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/
> >>>>>>>>>> . Although to your point Netty has ByteBuf...
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) <
> >>>>>> palomino219@gmail.com
> >>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Since we have shaded netty, I think we could make use of
> >>>>>>>>>>> netty's PlatformDependent[1] class to avoid referencing
> >>> Unsafe
> >>>>>>>> directly,
> >>>>>>>>>>> then there will be no problem for us to use the '--release
> >> 8'
> >>>>>> option.
> >>>>>>>>>>>
> >>>>>>>>>>> If no big concerns, I will file an issue to remove all the
> >>>>>>>>>> sun.misc.Unsafe
> >>>>>>>>>>> references in our code base and make use of
> >> PlatformDependent
> >>>>> from
> >>>>>>> the
> >>>>>>>>>>> shaded netty.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks.
> >>>>>>>>>>>
> >>>>>>>>>>> 1.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java
> >>>>>>>>>>>
> >>>>>>>>>>> 张铎(Duo Zhang) <pa...@gmail.com> 于2022年2月16日周三
> >> 14:12写道：
> >>>>>>>>>>>
> >>>>>>>>>>>> I think this could be done by some module tricks, where we
> >>>>> build a
> >>>>>>>>>>> special
> >>>>>>>>>>>> module with -source 8 and -target 8.We have done similar
> >>>> things
> >>>>> in
> >>>>>>> the
> >>>>>>>>>>> past
> >>>>>>>>>>>> to have hadoop1-compat and hadoop2-compact.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Let me have a try.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Sean Busbey <bu...@apache.org> 于2022年2月16日周三 13:31写道：
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Unfortunately if we want to keep jdk8 working we can't
> >> rely
> >>>> on
> >>>>>>>>>> building
> >>>>>>>>>>>>> with the jdk release target option on newer jdks.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We ran into this recently in HBASE-25465 where a JDK bug
> >>> came
> >>>>> up.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) <
> >>>>>>> palomino219@gmail.com
> >>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> +1 on moving the minimum required java version for
> >>> compiling
> >>>>>> HBase
> >>>>>>>>>> to
> >>>>>>>>>>>>> Java
> >>>>>>>>>>>>>> 11. But we could still use --release=8 to still support
> >>> jdk8
> >>>>> at
> >>>>>>>>>>> runtime.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Java 8 is still widely used, and I believe this will
> >> last
> >>>> for
> >>>>>> even
> >>>>>>>>>>> more
> >>>>>>>>>>>>>> years, as lots of big companies such as RedHat,
> >> Microsoft
> >>>> and
> >>>>>>> Amazon
> >>>>>>>>>>> are
> >>>>>>>>>>>>>> still shipping new jdk8 releases, I do not think it is
> >>> time
> >>>> to
> >>>>>>> fully
> >>>>>>>>>>>>> drop
> >>>>>>>>>>>>>> the support of jdk8.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sean Busbey <bu...@apache.org> 于2022年2月16日周三 09:57写道：
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> If we set the minJDK to 11 I believe that will
> >>> effectively
> >>>>>> remove
> >>>>>>>>>>> jdk8
> >>>>>>>>>>>>>>> support, rather than "just" deprecate it.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> As we build out more robust testing I would be in favor
> >>> of
> >>>>>>> running
> >>>>>>>>>>>>> less
> >>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>> it on deprecated JDKs.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser <
> >>> elserj@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11
> >>> seems
> >>>>>>>>>>>>> reasonable
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>> me.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Gotta start pushing the issue somehow.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote:
> >>>>>>>>>>>>>>>>> Hi folks!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's been some time since we decided to stick to LTS
> >>> JDK
> >>>>>>>>>>> releases
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>> way
> >>>>>>>>>>>>>>>>> of getting a handle on the JDK treadmill.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> What do folks think about deprecating JDK8? The
> >>> openjdk8u
> >>>>>>>>>>> project
> >>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>> going and there are commercial support options at
> >> least
> >>>>>>>>>> through
> >>>>>>>>>>>>> 2030.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Deprecating it in HBase 3 would mean we could remove
> >> it
> >>>> in
> >>>>>>>>>> HBase
> >>>>>>>>>>>>> 4,
> >>>>>>>>>>>>>> not
> >>>>>>>>>>>>>>>>> that we would _have_ to remove it. The way I think
> >>> about
> >>>>>>>>>> likely
> >>>>>>>>>>>>>> timing
> >>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>> these events goes like this:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> * HBase 2 started alphas in June 2017, betas in
> >> January
> >>>>> 2018,
> >>>>>>>>>>> and
> >>>>>>>>>>>>>> came
> >>>>>>>>>>>>>>>> out
> >>>>>>>>>>>>>>>>> in April 2018
> >>>>>>>>>>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb
> >>> 2022
> >>>>> we
> >>>>>>>>>>>>> haven't
> >>>>>>>>>>>>>>>>> discussed how close we are to our stated beta goals
> >>>>> (upgrades
> >>>>>>>>>>> from
> >>>>>>>>>>>>>>> active
> >>>>>>>>>>>>>>>>> 2.x releases and removal of not-ready features).
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Given the above, in the absence of us specifically
> >>>> pushing
> >>>>> to
> >>>>>>>>>>> roll
> >>>>>>>>>>>>>>>> through
> >>>>>>>>>>>>>>>>> major version numbers for some reason, I think a
> >>>> reasonably
> >>>>>>>>>>>>>>> conservative
> >>>>>>>>>>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or
> >> early
> >>>>> 2023
> >>>>>>>>>> and
> >>>>>>>>>>>>> then
> >>>>>>>>>>>>>>>> HBase
> >>>>>>>>>>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030
> >>>> seems
> >>>>>>>>>>>>> unlikely.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> That all said, our current reference guide section on
> >>>> java
> >>>>>>>>>>>>> versions
> >>>>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>>> not sound very confident about JDK11 support.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> A Note on JDK11 *
> >>>>>>>>>>>>>>>>>> Preliminary support for JDK11 is introduced with
> >> HBase
> >>>>>> 2.3.0.
> >>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>> support is limited to
> >>>>>>>>>>>>>>>>>> compilation and running the full test suite. There
> >> are
> >>>>> open
> >>>>>>>>>>>>>> questions
> >>>>>>>>>>>>>>>>> regarding the runtime
> >>>>>>>>>>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and
> >>> Apache
> >>>>>>>>>> Hadoop
> >>>>>>>>>>>>>>>>> (HADOOP-15338).
> >>>>>>>>>>>>>>>>>> Significantly, neither project has yet released a
> >>>> version
> >>>>>>>>>> with
> >>>>>>>>>>>>>>> explicit
> >>>>>>>>>>>>>>>>> runtime support for
> >>>>>>>>>>>>>>>>>> JDK11. The remaining known issues in HBase are
> >>>> catalogued
> >>>>> in
> >>>>>>>>>>>>>>>> HBASE-22972.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Since that blurb was written, Hadoop has added JDK11
> >>>>> support
> >>>>>>>>>> [1]
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>> has
> >>>>>>>>>>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11
> >>> support
> >>>>> we
> >>>>>>>>>>> could
> >>>>>>>>>>>>>>> update
> >>>>>>>>>>>>>>>>> our minimum supported versions of these projects to
> >>> match
> >>>>>> that
> >>>>>>>>>>>>>> support.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> What do folks think?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [1]:
> >> https://hadoop.apache.org/docs/r3.3.0/index.html
> >>>>>>>>>>>>>>>>> [2]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> Best regards,
> >>>>>>>>>> Andrew
> >>>>>>>>>>
> >>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>>>>>>>>> It's what we’ve earned
> >>>>>>>>>> Welcome, apocalypse, what’s taken you so long?
> >>>>>>>>>> Bring us the fitting end that we’ve been counting on
> >>>>>>>>>> - A23, Welcome, Apocalypse
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Best regards,
> >>>>>> Andrew
> >>>>>>
> >>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>>>>> It's what we’ve earned
> >>>>>> Welcome, apocalypse, what’s taken you so long?
> >>>>>> Bring us the fitting end that we’ve been counting on
> >>>>>> - A23, Welcome, Apocalypse
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Andrew
> >>>>
> >>>> Unrest, ignorance distilled, nihilistic imbeciles -
> >>>> It's what we’ve earned
> >>>> Welcome, apocalypse, what’s taken you so long?
> >>>> Bring us the fitting end that we’ve been counting on
> >>>> - A23, Welcome, Apocalypse
> >>>>
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >> Unrest, ignorance distilled, nihilistic imbeciles -
> >> It's what we’ve earned
> >> Welcome, apocalypse, what’s taken you so long?
> >> Bring us the fitting end that we’ve been counting on
> >> - A23, Welcome, Apocalypse
> >>
>
Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Filed HBASE-26793 for making a new hbase-thirdparty release.
Then will put the PR for HBASE-25465.
Thanks.
张铎(Duo Zhang) <pa...@gmail.com> 于2022年3月2日周三 11:19写道：
> Good news, with HBASE-26781 in place[1], I successfully built hbase with
> java 11 --release 8. Of course I need to modify the hbase main code a bit,
> and also make use of netty's PlatformDependent for processing
> DirectByteBuffer.
>
> 1. https://github.com/apache/hbase-thirdparty/pull/79
>
> 张铎(Duo Zhang) <pa...@gmail.com> 于2022年3月1日周二 22:07写道：
>
>> So after HBASE-26773, we still have another problem that we can not
>> use sun.nio.ch.DirectBuffer. The main usage is to get the address.
>>
>> Let me provide a new util method in the hbase-unsafe module.
>>
>> Andrew Purtell <an...@gmail.com> 于2022年2月26日周六 13:58写道：
>>
>>> That sounds good, looking forward to it. Please let me know if you want
>>> help.
>>>
>>> > On Feb 25, 2022, at 9:56 PM, 张铎 <pa...@gmail.com> wrote:
>>> >
>>> > Filed HBASE-26773 and tried, the result is not very good. Netty's
>>> > PlatformDependent can not meet all the requirements.
>>> >
>>> > So now I prefer we have a new hbase-thirdparty-unsafe module to
>>> provide the
>>> > Unsafe ability without exposing sun.misc.Unsafe directly.
>>> >
>>> > Will give it a try.
>>> >
>>> > Thanks.
>>> >
>>> > Andrew Purtell <ap...@apache.org> 于2022年2月24日周四 10:46写道：
>>> >
>>> >> Ok, a bridge in one place would minimize the risk.
>>> >>
>>> >>> On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <palomino219@gmail.com
>>> >
>>> >>> wrote:
>>> >>>
>>> >>> I think this is a reasonable concern, we could do this on our own.
>>> >>>
>>> >>> But checking the commit history of PlatformDependent[1], the API is
>>> >> pretty
>>> >>> stable, so I think the risk of hitting a security issue but we can
>>> not
>>> >>> upgrade due to API conflicts is very low. And if a security issue
>>> >> requires
>>> >>> big changes on PlatformDependent, then I assume we have to fix our
>>> own
>>> >>> version as well...
>>> >>>
>>> >>> And we will not spread PlatformDependent references everywhere in our
>>> >> code,
>>> >>> we will still have a bridge, like the current UnsafeAccess. So if
>>> later
>>> >> we
>>> >>> decide to implement our own version, it will not introduce a big
>>> impact
>>> >> to
>>> >>> our code base.
>>> >>>
>>> >>> Thanks.
>>> >>>
>>> >>>
>>> >>> [1]
>>> >>>
>>> >>>
>>> >>
>>> https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java
>>> >>>
>>> >>> Andrew Purtell <ap...@apache.org> 于2022年2月24日周四 01:12写道：
>>> >>>
>>> >>>> We often upgrade netty after there is a CVE announcement. This
>>> >> produces a
>>> >>>> pressure to upgrade in production. Use of an internal API is not a
>>> good
>>> >>>> practice because it can cause friction because of breaking changes
>>> when
>>> >>>> there is a need to upgrade all of a sudden. This is painful and
>>> >> hopefully
>>> >>>> we can exclude it as a possibility now when coming up with the
>>> plan. We
>>> >>> can
>>> >>>> do the same thing they do in our own hbase-thirdparty without taking
>>> >> the
>>> >>>> dependency and achieve the same goal, right?
>>> >>>>
>>> >>>> On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <
>>> palomino219@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> For me, since we shade netty, there will be no conflict with other
>>> >>> netty
>>> >>>>> dependencies, so we can make sure the netty version we used is
>>> >> exactly
>>> >>>> the
>>> >>>>> one we shaded in hbase-thirdparty.
>>> >>>>>
>>> >>>>> If later we want to upgrade netty version in hbase-thirdparty, and
>>> it
>>> >>>>> breaks the api, then we can release a new hbase-thirdparty, and
>>> >> upgrade
>>> >>>> the
>>> >>>>> main hbase repo to depend on the new hbase-thirdparty version, and
>>> >> also
>>> >>>>> modify the code in hbase accordingly. There will be no user visible
>>> >>>>> affects.
>>> >>>>>
>>> >>>>> Anyway, let me have a try first. Even with netty's
>>> PaltformDependent
>>> >>>> class,
>>> >>>>> since it still references sun.misc.Unsafe, maybe we will still get
>>> an
>>> >>>>> compile error...
>>> >>>>>
>>> >>>>> Thanks.
>>> >>>>>
>>> >>>>> Andrew Purtell <ap...@apache.org> 于2022年2月22日周二 08:08写道：
>>> >>>>>
>>> >>>>>> Thanks for that. I understand better what you are proposing now.
>>> It
>>> >>> is
>>> >>>>>> clear that you have thought it through.
>>> >>>>>>
>>> >>>>>> So then the only question I have is this: Do you think it is a
>>> >>> concern
>>> >>>>> that
>>> >>>>>> Netty's PlatformDependent class is in their
>>> >> "io.netty.util.internal"
>>> >>>>>> package? The 'internal' designation suggests to me they won't
>>> apply
>>> >>> the
>>> >>>>>> same care to not breaking users of it as they would for their
>>> >> public
>>> >>>>>> (non-"internal") API. I noticed this earlier but had other
>>> >> questions
>>> >>> at
>>> >>>>>> that time.
>>> >>>>>>
>>> >>>>>> If it is a concern, we have the option of doing the same thing
>>> that
>>> >>>> Netty
>>> >>>>>> has done with their PlatformDependent class as a new utility class
>>> >> of
>>> >>>> our
>>> >>>>>> own in hbase-thirdparty. This may impose special requirements on
>>> >>>> building
>>> >>>>>> hbase-thirdparty, or, at least, the module containing this
>>> wrapper,
>>> >>> but
>>> >>>>> it
>>> >>>>>> is an option that would help us avoid external breaking changes.
>>> >> For
>>> >>>> your
>>> >>>>>> consideration.
>>> >>>>>>
>>> >>>>>> On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) <
>>> >> palomino219@gmail.com
>>> >>>>
>>> >>>>>> wrote:
>>> >>>>>>
>>> >>>>>>> What I proposed here is for solving the problem…
>>> >>>>>>>
>>> >>>>>>> The problem is that when using —release 8, javac will not export
>>> >>> the
>>> >>>>>>> jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a
>>> >>>>> compile
>>> >>>>>>> error. But at runtime if you export jdk.unsupported you can still
>>> >>> use
>>> >>>>>>> sun.misc.Unsafe. The related jdk issue said —release is only for
>>> >>>> public
>>> >>>>>> API
>>> >>>>>>> so not exporting jdk.unsupported is the expected behavior.
>>> >>>>>>>
>>> >>>>>>> So what I proposed here is to remove direct dependency on
>>> >>>>> sun.misc.Unsafe
>>> >>>>>>> in HBase code, so at compile time there will be no problem. Betty
>>> >>>> has a
>>> >>>>>>> wrapper of Unsafe so I think we could have a try. At runtime if
>>> >> we
>>> >>>> have
>>> >>>>>>> jdk.unsupported exported we could still actually make use of
>>> >>>>>>> sun.misc.Unsafe.
>>> >>>>>>>
>>> >>>>>>> Feel free to correct me if I missed something or there are still
>>> >>>>>> uncleared
>>> >>>>>>> things.
>>> >>>>>>>
>>> >>>>>>> Thanks.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Andrew Purtell <an...@gmail.com>于2022年2月22日 周二00:56写道：
>>> >>>>>>>
>>> >>>>>>>> As Nick discovered ‘—release’ doesn’t work for version 8
>>> >> anyway.
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>>> On Feb 20, 2022, at 3:47 PM, 张铎 <pa...@gmail.com>
>>> >> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> I know, with —release 8 ByteBuffer will not be an problem
>>> >> but
>>> >>>> then
>>> >>>>>>>>> sun.misc.Unsafe can not be used any more right? So the
>>> >> problem
>>> >>>> here
>>> >>>>>> is
>>> >>>>>>>> how
>>> >>>>>>>>> to make use of Unsafe with —release 8. What I proposed here
>>> >> is
>>> >>>> that
>>> >>>>>> we
>>> >>>>>>>> just
>>> >>>>>>>>> do not use it, then we can make use of —release 8 to address
>>> >>> the
>>> >>>>>>>> ByteBuffer
>>> >>>>>>>>> problem.
>>> >>>>>>>>>
>>> >>>>>>>>> Thanks.
>>> >>>>>>>>>
>>> >>>>>>>>> Andrew Purtell <ap...@apache.org>于2022年2月21日 周一03:32写道：
>>> >>>>>>>>>
>>> >>>>>>>>>> The problem that leads us to consider '--release 8', or what
>>> >>> it
>>> >>>>>> would
>>> >>>>>>>>>> promise if it worked, isn't Unsafe, it is ByteBuffer.
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/
>>> >>>>>>>>>> . Although to your point Netty has ByteBuf...
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) <
>>> >>>>>> palomino219@gmail.com
>>> >>>>>>>>
>>> >>>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Since we have shaded netty, I think we could make use of
>>> >>>>>>>>>>> netty's PlatformDependent[1] class to avoid referencing
>>> >>> Unsafe
>>> >>>>>>>> directly,
>>> >>>>>>>>>>> then there will be no problem for us to use the '--release
>>> >> 8'
>>> >>>>>> option.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> If no big concerns, I will file an issue to remove all the
>>> >>>>>>>>>> sun.misc.Unsafe
>>> >>>>>>>>>>> references in our code base and make use of
>>> >> PlatformDependent
>>> >>>>> from
>>> >>>>>>> the
>>> >>>>>>>>>>> shaded netty.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Thanks.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> 1.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>
>>> >>
>>> https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> 张铎(Duo Zhang) <pa...@gmail.com> 于2022年2月16日周三
>>> >> 14:12写道：
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> I think this could be done by some module tricks, where we
>>> >>>>> build a
>>> >>>>>>>>>>> special
>>> >>>>>>>>>>>> module with -source 8 and -target 8.We have done similar
>>> >>>> things
>>> >>>>> in
>>> >>>>>>> the
>>> >>>>>>>>>>> past
>>> >>>>>>>>>>>> to have hadoop1-compat and hadoop2-compact.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Let me have a try.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Sean Busbey <bu...@apache.org> 于2022年2月16日周三 13:31写道：
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>> Unfortunately if we want to keep jdk8 working we can't
>>> >> rely
>>> >>>> on
>>> >>>>>>>>>> building
>>> >>>>>>>>>>>>> with the jdk release target option on newer jdks.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> We ran into this recently in HBASE-25465 where a JDK bug
>>> >>> came
>>> >>>>> up.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) <
>>> >>>>>>> palomino219@gmail.com
>>> >>>>>>>>>
>>> >>>>>>>>>>>>> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> +1 on moving the minimum required java version for
>>> >>> compiling
>>> >>>>>> HBase
>>> >>>>>>>>>> to
>>> >>>>>>>>>>>>> Java
>>> >>>>>>>>>>>>>> 11. But we could still use --release=8 to still support
>>> >>> jdk8
>>> >>>>> at
>>> >>>>>>>>>>> runtime.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Java 8 is still widely used, and I believe this will
>>> >> last
>>> >>>> for
>>> >>>>>> even
>>> >>>>>>>>>>> more
>>> >>>>>>>>>>>>>> years, as lots of big companies such as RedHat,
>>> >> Microsoft
>>> >>>> and
>>> >>>>>>> Amazon
>>> >>>>>>>>>>> are
>>> >>>>>>>>>>>>>> still shipping new jdk8 releases, I do not think it is
>>> >>> time
>>> >>>> to
>>> >>>>>>> fully
>>> >>>>>>>>>>>>> drop
>>> >>>>>>>>>>>>>> the support of jdk8.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Thanks.
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Sean Busbey <bu...@apache.org> 于2022年2月16日周三 09:57写道：
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> If we set the minJDK to 11 I believe that will
>>> >>> effectively
>>> >>>>>> remove
>>> >>>>>>>>>>> jdk8
>>> >>>>>>>>>>>>>>> support, rather than "just" deprecate it.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> As we build out more robust testing I would be in favor
>>> >>> of
>>> >>>>>>> running
>>> >>>>>>>>>>>>> less
>>> >>>>>>>>>>>>>> of
>>> >>>>>>>>>>>>>>> it on deprecated JDKs.
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser <
>>> >>> elserj@apache.org>
>>> >>>>>>> wrote:
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11
>>> >>> seems
>>> >>>>>>>>>>>>> reasonable
>>> >>>>>>>>>>>>>> to
>>> >>>>>>>>>>>>>>>> me.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> Gotta start pushing the issue somehow.
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote:
>>> >>>>>>>>>>>>>>>>> Hi folks!
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> It's been some time since we decided to stick to LTS
>>> >>> JDK
>>> >>>>>>>>>>> releases
>>> >>>>>>>>>>>>> as
>>> >>>>>>>>>>>>>> a
>>> >>>>>>>>>>>>>>>> way
>>> >>>>>>>>>>>>>>>>> of getting a handle on the JDK treadmill.
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> What do folks think about deprecating JDK8? The
>>> >>> openjdk8u
>>> >>>>>>>>>>> project
>>> >>>>>>>>>>>>> is
>>> >>>>>>>>>>>>>>>> still
>>> >>>>>>>>>>>>>>>>> going and there are commercial support options at
>>> >> least
>>> >>>>>>>>>> through
>>> >>>>>>>>>>>>> 2030.
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> Deprecating it in HBase 3 would mean we could remove
>>> >> it
>>> >>>> in
>>> >>>>>>>>>> HBase
>>> >>>>>>>>>>>>> 4,
>>> >>>>>>>>>>>>>> not
>>> >>>>>>>>>>>>>>>>> that we would _have_ to remove it. The way I think
>>> >>> about
>>> >>>>>>>>>> likely
>>> >>>>>>>>>>>>>> timing
>>> >>>>>>>>>>>>>>> of
>>> >>>>>>>>>>>>>>>>> these events goes like this:
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> * HBase 2 started alphas in June 2017, betas in
>>> >> January
>>> >>>>> 2018,
>>> >>>>>>>>>>> and
>>> >>>>>>>>>>>>>> came
>>> >>>>>>>>>>>>>>>> out
>>> >>>>>>>>>>>>>>>>> in April 2018
>>> >>>>>>>>>>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb
>>> >>> 2022
>>> >>>>> we
>>> >>>>>>>>>>>>> haven't
>>> >>>>>>>>>>>>>>>>> discussed how close we are to our stated beta goals
>>> >>>>> (upgrades
>>> >>>>>>>>>>> from
>>> >>>>>>>>>>>>>>> active
>>> >>>>>>>>>>>>>>>>> 2.x releases and removal of not-ready features).
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> Given the above, in the absence of us specifically
>>> >>>> pushing
>>> >>>>> to
>>> >>>>>>>>>>> roll
>>> >>>>>>>>>>>>>>>> through
>>> >>>>>>>>>>>>>>>>> major version numbers for some reason, I think a
>>> >>>> reasonably
>>> >>>>>>>>>>>>>>> conservative
>>> >>>>>>>>>>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or
>>> >> early
>>> >>>>> 2023
>>> >>>>>>>>>> and
>>> >>>>>>>>>>>>> then
>>> >>>>>>>>>>>>>>>> HBase
>>> >>>>>>>>>>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030
>>> >>>> seems
>>> >>>>>>>>>>>>> unlikely.
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> That all said, our current reference guide section on
>>> >>>> java
>>> >>>>>>>>>>>>> versions
>>> >>>>>>>>>>>>>>> does
>>> >>>>>>>>>>>>>>>>> not sound very confident about JDK11 support.
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>> A Note on JDK11 *
>>> >>>>>>>>>>>>>>>>>> Preliminary support for JDK11 is introduced with
>>> >> HBase
>>> >>>>>> 2.3.0.
>>> >>>>>>>>>>>>> This
>>> >>>>>>>>>>>>>>>>> support is limited to
>>> >>>>>>>>>>>>>>>>>> compilation and running the full test suite. There
>>> >> are
>>> >>>>> open
>>> >>>>>>>>>>>>>> questions
>>> >>>>>>>>>>>>>>>>> regarding the runtime
>>> >>>>>>>>>>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and
>>> >>> Apache
>>> >>>>>>>>>> Hadoop
>>> >>>>>>>>>>>>>>>>> (HADOOP-15338).
>>> >>>>>>>>>>>>>>>>>> Significantly, neither project has yet released a
>>> >>>> version
>>> >>>>>>>>>> with
>>> >>>>>>>>>>>>>>> explicit
>>> >>>>>>>>>>>>>>>>> runtime support for
>>> >>>>>>>>>>>>>>>>>> JDK11. The remaining known issues in HBase are
>>> >>>> catalogued
>>> >>>>> in
>>> >>>>>>>>>>>>>>>> HBASE-22972.
>>> >>>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> Since that blurb was written, Hadoop has added JDK11
>>> >>>>> support
>>> >>>>>>>>>> [1]
>>> >>>>>>>>>>>>> as
>>> >>>>>>>>>>>>>> has
>>> >>>>>>>>>>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11
>>> >>> support
>>> >>>>> we
>>> >>>>>>>>>>> could
>>> >>>>>>>>>>>>>>> update
>>> >>>>>>>>>>>>>>>>> our minimum supported versions of these projects to
>>> >>> match
>>> >>>>>> that
>>> >>>>>>>>>>>>>> support.
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> What do folks think?
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>> [1]:
>>> >> https://hadoop.apache.org/docs/r3.3.0/index.html
>>> >>>>>>>>>>>>>>>>> [2]:
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
>>> >>>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> --
>>> >>>>>>>>>> Best regards,
>>> >>>>>>>>>> Andrew
>>> >>>>>>>>>>
>>> >>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>> >>>>>>>>>> It's what we’ve earned
>>> >>>>>>>>>> Welcome, apocalypse, what’s taken you so long?
>>> >>>>>>>>>> Bring us the fitting end that we’ve been counting on
>>> >>>>>>>>>> - A23, Welcome, Apocalypse
>>> >>>>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Best regards,
>>> >>>>>> Andrew
>>> >>>>>>
>>> >>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>> >>>>>> It's what we’ve earned
>>> >>>>>> Welcome, apocalypse, what’s taken you so long?
>>> >>>>>> Bring us the fitting end that we’ve been counting on
>>> >>>>>> - A23, Welcome, Apocalypse
>>> >>>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Best regards,
>>> >>>> Andrew
>>> >>>>
>>> >>>> Unrest, ignorance distilled, nihilistic imbeciles -
>>> >>>> It's what we’ve earned
>>> >>>> Welcome, apocalypse, what’s taken you so long?
>>> >>>> Bring us the fitting end that we’ve been counting on
>>> >>>> - A23, Welcome, Apocalypse
>>> >>>>
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >> Best regards,
>>> >> Andrew
>>> >>
>>> >> Unrest, ignorance distilled, nihilistic imbeciles -
>>> >> It's what we’ve earned
>>> >> Welcome, apocalypse, what’s taken you so long?
>>> >> Bring us the fitting end that we’ve been counting on
>>> >> - A23, Welcome, Apocalypse
>>> >>
>>>
>>
Re: [DISCUSS] deprecating jdk8, progress on LTS jdk support
Posted by "张铎(Duo Zhang)" <pa...@gmail.com>.
Good news, with HBASE-26781 in place[1], I successfully built hbase with
java 11 --release 8. Of course I need to modify the hbase main code a bit,
and also make use of netty's PlatformDependent for processing
DirectByteBuffer.
1. https://github.com/apache/hbase-thirdparty/pull/79
张铎(Duo Zhang) <pa...@gmail.com> 于2022年3月1日周二 22:07写道：
> So after HBASE-26773, we still have another problem that we can not
> use sun.nio.ch.DirectBuffer. The main usage is to get the address.
>
> Let me provide a new util method in the hbase-unsafe module.
>
> Andrew Purtell <an...@gmail.com> 于2022年2月26日周六 13:58写道：
>
>> That sounds good, looking forward to it. Please let me know if you want
>> help.
>>
>> > On Feb 25, 2022, at 9:56 PM, 张铎 <pa...@gmail.com> wrote:
>> >
>> > Filed HBASE-26773 and tried, the result is not very good. Netty's
>> > PlatformDependent can not meet all the requirements.
>> >
>> > So now I prefer we have a new hbase-thirdparty-unsafe module to provide
>> the
>> > Unsafe ability without exposing sun.misc.Unsafe directly.
>> >
>> > Will give it a try.
>> >
>> > Thanks.
>> >
>> > Andrew Purtell <ap...@apache.org> 于2022年2月24日周四 10:46写道：
>> >
>> >> Ok, a bridge in one place would minimize the risk.
>> >>
>> >>> On Wed, Feb 23, 2022 at 6:07 PM 张铎(Duo Zhang) <pa...@gmail.com>
>> >>> wrote:
>> >>>
>> >>> I think this is a reasonable concern, we could do this on our own.
>> >>>
>> >>> But checking the commit history of PlatformDependent[1], the API is
>> >> pretty
>> >>> stable, so I think the risk of hitting a security issue but we can not
>> >>> upgrade due to API conflicts is very low. And if a security issue
>> >> requires
>> >>> big changes on PlatformDependent, then I assume we have to fix our own
>> >>> version as well...
>> >>>
>> >>> And we will not spread PlatformDependent references everywhere in our
>> >> code,
>> >>> we will still have a bridge, like the current UnsafeAccess. So if
>> later
>> >> we
>> >>> decide to implement our own version, it will not introduce a big
>> impact
>> >> to
>> >>> our code base.
>> >>>
>> >>> Thanks.
>> >>>
>> >>>
>> >>> [1]
>> >>>
>> >>>
>> >>
>> https://github.com/netty/netty/commits/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java
>> >>>
>> >>> Andrew Purtell <ap...@apache.org> 于2022年2月24日周四 01:12写道：
>> >>>
>> >>>> We often upgrade netty after there is a CVE announcement. This
>> >> produces a
>> >>>> pressure to upgrade in production. Use of an internal API is not a
>> good
>> >>>> practice because it can cause friction because of breaking changes
>> when
>> >>>> there is a need to upgrade all of a sudden. This is painful and
>> >> hopefully
>> >>>> we can exclude it as a possibility now when coming up with the plan.
>> We
>> >>> can
>> >>>> do the same thing they do in our own hbase-thirdparty without taking
>> >> the
>> >>>> dependency and achieve the same goal, right?
>> >>>>
>> >>>> On Mon, Feb 21, 2022 at 7:35 PM 张铎(Duo Zhang) <palomino219@gmail.com
>> >
>> >>>> wrote:
>> >>>>
>> >>>>> For me, since we shade netty, there will be no conflict with other
>> >>> netty
>> >>>>> dependencies, so we can make sure the netty version we used is
>> >> exactly
>> >>>> the
>> >>>>> one we shaded in hbase-thirdparty.
>> >>>>>
>> >>>>> If later we want to upgrade netty version in hbase-thirdparty, and
>> it
>> >>>>> breaks the api, then we can release a new hbase-thirdparty, and
>> >> upgrade
>> >>>> the
>> >>>>> main hbase repo to depend on the new hbase-thirdparty version, and
>> >> also
>> >>>>> modify the code in hbase accordingly. There will be no user visible
>> >>>>> affects.
>> >>>>>
>> >>>>> Anyway, let me have a try first. Even with netty's PaltformDependent
>> >>>> class,
>> >>>>> since it still references sun.misc.Unsafe, maybe we will still get
>> an
>> >>>>> compile error...
>> >>>>>
>> >>>>> Thanks.
>> >>>>>
>> >>>>> Andrew Purtell <ap...@apache.org> 于2022年2月22日周二 08:08写道：
>> >>>>>
>> >>>>>> Thanks for that. I understand better what you are proposing now. It
>> >>> is
>> >>>>>> clear that you have thought it through.
>> >>>>>>
>> >>>>>> So then the only question I have is this: Do you think it is a
>> >>> concern
>> >>>>> that
>> >>>>>> Netty's PlatformDependent class is in their
>> >> "io.netty.util.internal"
>> >>>>>> package? The 'internal' designation suggests to me they won't apply
>> >>> the
>> >>>>>> same care to not breaking users of it as they would for their
>> >> public
>> >>>>>> (non-"internal") API. I noticed this earlier but had other
>> >> questions
>> >>> at
>> >>>>>> that time.
>> >>>>>>
>> >>>>>> If it is a concern, we have the option of doing the same thing that
>> >>>> Netty
>> >>>>>> has done with their PlatformDependent class as a new utility class
>> >> of
>> >>>> our
>> >>>>>> own in hbase-thirdparty. This may impose special requirements on
>> >>>> building
>> >>>>>> hbase-thirdparty, or, at least, the module containing this wrapper,
>> >>> but
>> >>>>> it
>> >>>>>> is an option that would help us avoid external breaking changes.
>> >> For
>> >>>> your
>> >>>>>> consideration.
>> >>>>>>
>> >>>>>> On Mon, Feb 21, 2022 at 3:48 PM 张铎(Duo Zhang) <
>> >> palomino219@gmail.com
>> >>>>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>>> What I proposed here is for solving the problem…
>> >>>>>>>
>> >>>>>>> The problem is that when using —release 8, javac will not export
>> >>> the
>> >>>>>>> jdk.unsupported symbols, so using sun.misc.Unsafe will trigger a
>> >>>>> compile
>> >>>>>>> error. But at runtime if you export jdk.unsupported you can still
>> >>> use
>> >>>>>>> sun.misc.Unsafe. The related jdk issue said —release is only for
>> >>>> public
>> >>>>>> API
>> >>>>>>> so not exporting jdk.unsupported is the expected behavior.
>> >>>>>>>
>> >>>>>>> So what I proposed here is to remove direct dependency on
>> >>>>> sun.misc.Unsafe
>> >>>>>>> in HBase code, so at compile time there will be no problem. Betty
>> >>>> has a
>> >>>>>>> wrapper of Unsafe so I think we could have a try. At runtime if
>> >> we
>> >>>> have
>> >>>>>>> jdk.unsupported exported we could still actually make use of
>> >>>>>>> sun.misc.Unsafe.
>> >>>>>>>
>> >>>>>>> Feel free to correct me if I missed something or there are still
>> >>>>>> uncleared
>> >>>>>>> things.
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Andrew Purtell <an...@gmail.com>于2022年2月22日 周二00:56写道：
>> >>>>>>>
>> >>>>>>>> As Nick discovered ‘—release’ doesn’t work for version 8
>> >> anyway.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> On Feb 20, 2022, at 3:47 PM, 张铎 <pa...@gmail.com>
>> >> wrote:
>> >>>>>>>>>
>> >>>>>>>>> I know, with —release 8 ByteBuffer will not be an problem
>> >> but
>> >>>> then
>> >>>>>>>>> sun.misc.Unsafe can not be used any more right? So the
>> >> problem
>> >>>> here
>> >>>>>> is
>> >>>>>>>> how
>> >>>>>>>>> to make use of Unsafe with —release 8. What I proposed here
>> >> is
>> >>>> that
>> >>>>>> we
>> >>>>>>>> just
>> >>>>>>>>> do not use it, then we can make use of —release 8 to address
>> >>> the
>> >>>>>>>> ByteBuffer
>> >>>>>>>>> problem.
>> >>>>>>>>>
>> >>>>>>>>> Thanks.
>> >>>>>>>>>
>> >>>>>>>>> Andrew Purtell <ap...@apache.org>于2022年2月21日 周一03:32写道：
>> >>>>>>>>>
>> >>>>>>>>>> The problem that leads us to consider '--release 8', or what
>> >>> it
>> >>>>>> would
>> >>>>>>>>>> promise if it worked, isn't Unsafe, it is ByteBuffer.
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/
>> >>>>>>>>>> . Although to your point Netty has ByteBuf...
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>> On Sun, Feb 20, 2022 at 5:23 AM 张铎(Duo Zhang) <
>> >>>>>> palomino219@gmail.com
>> >>>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Since we have shaded netty, I think we could make use of
>> >>>>>>>>>>> netty's PlatformDependent[1] class to avoid referencing
>> >>> Unsafe
>> >>>>>>>> directly,
>> >>>>>>>>>>> then there will be no problem for us to use the '--release
>> >> 8'
>> >>>>>> option.
>> >>>>>>>>>>>
>> >>>>>>>>>>> If no big concerns, I will file an issue to remove all the
>> >>>>>>>>>> sun.misc.Unsafe
>> >>>>>>>>>>> references in our code base and make use of
>> >> PlatformDependent
>> >>>>> from
>> >>>>>>> the
>> >>>>>>>>>>> shaded netty.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks.
>> >>>>>>>>>>>
>> >>>>>>>>>>> 1.
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> https://github.com/netty/netty/blob/4.1/common/src/main/java/io/netty/util/internal/PlatformDependent.java
>> >>>>>>>>>>>
>> >>>>>>>>>>> 张铎(Duo Zhang) <pa...@gmail.com> 于2022年2月16日周三
>> >> 14:12写道：
>> >>>>>>>>>>>
>> >>>>>>>>>>>> I think this could be done by some module tricks, where we
>> >>>>> build a
>> >>>>>>>>>>> special
>> >>>>>>>>>>>> module with -source 8 and -target 8.We have done similar
>> >>>> things
>> >>>>> in
>> >>>>>>> the
>> >>>>>>>>>>> past
>> >>>>>>>>>>>> to have hadoop1-compat and hadoop2-compact.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Let me have a try.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Sean Busbey <bu...@apache.org> 于2022年2月16日周三 13:31写道：
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Unfortunately if we want to keep jdk8 working we can't
>> >> rely
>> >>>> on
>> >>>>>>>>>> building
>> >>>>>>>>>>>>> with the jdk release target option on newer jdks.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> We ran into this recently in HBASE-25465 where a JDK bug
>> >>> came
>> >>>>> up.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Tue, Feb 15, 2022 at 8:12 PM 张铎(Duo Zhang) <
>> >>>>>>> palomino219@gmail.com
>> >>>>>>>>>
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> +1 on moving the minimum required java version for
>> >>> compiling
>> >>>>>> HBase
>> >>>>>>>>>> to
>> >>>>>>>>>>>>> Java
>> >>>>>>>>>>>>>> 11. But we could still use --release=8 to still support
>> >>> jdk8
>> >>>>> at
>> >>>>>>>>>>> runtime.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Java 8 is still widely used, and I believe this will
>> >> last
>> >>>> for
>> >>>>>> even
>> >>>>>>>>>>> more
>> >>>>>>>>>>>>>> years, as lots of big companies such as RedHat,
>> >> Microsoft
>> >>>> and
>> >>>>>>> Amazon
>> >>>>>>>>>>> are
>> >>>>>>>>>>>>>> still shipping new jdk8 releases, I do not think it is
>> >>> time
>> >>>> to
>> >>>>>>> fully
>> >>>>>>>>>>>>> drop
>> >>>>>>>>>>>>>> the support of jdk8.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Thanks.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Sean Busbey <bu...@apache.org> 于2022年2月16日周三 09:57写道：
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> If we set the minJDK to 11 I believe that will
>> >>> effectively
>> >>>>>> remove
>> >>>>>>>>>>> jdk8
>> >>>>>>>>>>>>>>> support, rather than "just" deprecate it.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> As we build out more robust testing I would be in favor
>> >>> of
>> >>>>>>> running
>> >>>>>>>>>>>>> less
>> >>>>>>>>>>>>>> of
>> >>>>>>>>>>>>>>> it on deprecated JDKs.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Tue, Feb 15, 2022, 16:10 Josh Elser <
>> >>> elserj@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Deprecating jdk8 for HBase 3 and requiring minJdk=11
>> >>> seems
>> >>>>>>>>>>>>> reasonable
>> >>>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>> me.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Gotta start pushing the issue somehow.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On 2/15/22 1:47 PM, Sean Busbey wrote:
>> >>>>>>>>>>>>>>>>> Hi folks!
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> It's been some time since we decided to stick to LTS
>> >>> JDK
>> >>>>>>>>>>> releases
>> >>>>>>>>>>>>> as
>> >>>>>>>>>>>>>> a
>> >>>>>>>>>>>>>>>> way
>> >>>>>>>>>>>>>>>>> of getting a handle on the JDK treadmill.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> What do folks think about deprecating JDK8? The
>> >>> openjdk8u
>> >>>>>>>>>>> project
>> >>>>>>>>>>>>> is
>> >>>>>>>>>>>>>>>> still
>> >>>>>>>>>>>>>>>>> going and there are commercial support options at
>> >> least
>> >>>>>>>>>> through
>> >>>>>>>>>>>>> 2030.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Deprecating it in HBase 3 would mean we could remove
>> >> it
>> >>>> in
>> >>>>>>>>>> HBase
>> >>>>>>>>>>>>> 4,
>> >>>>>>>>>>>>>> not
>> >>>>>>>>>>>>>>>>> that we would _have_ to remove it. The way I think
>> >>> about
>> >>>>>>>>>> likely
>> >>>>>>>>>>>>>> timing
>> >>>>>>>>>>>>>>> of
>> >>>>>>>>>>>>>>>>> these events goes like this:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> * HBase 2 started alphas in June 2017, betas in
>> >> January
>> >>>>> 2018,
>> >>>>>>>>>>> and
>> >>>>>>>>>>>>>> came
>> >>>>>>>>>>>>>>>> out
>> >>>>>>>>>>>>>>>>> in April 2018
>> >>>>>>>>>>>>>>>>> * HBase 3 started alphas in July 2021, and as of Feb
>> >>> 2022
>> >>>>> we
>> >>>>>>>>>>>>> haven't
>> >>>>>>>>>>>>>>>>> discussed how close we are to our stated beta goals
>> >>>>> (upgrades
>> >>>>>>>>>>> from
>> >>>>>>>>>>>>>>> active
>> >>>>>>>>>>>>>>>>> 2.x releases and removal of not-ready features).
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Given the above, in the absence of us specifically
>> >>>> pushing
>> >>>>> to
>> >>>>>>>>>>> roll
>> >>>>>>>>>>>>>>>> through
>> >>>>>>>>>>>>>>>>> major version numbers for some reason, I think a
>> >>>> reasonably
>> >>>>>>>>>>>>>>> conservative
>> >>>>>>>>>>>>>>>>> estimate is for HBase 3 to arrive in late 2022 or
>> >> early
>> >>>>> 2023
>> >>>>>>>>>> and
>> >>>>>>>>>>>>> then
>> >>>>>>>>>>>>>>>> HBase
>> >>>>>>>>>>>>>>>>> 4 to start alphas in ~2025. An HBase 5 prior to 2030
>> >>>> seems
>> >>>>>>>>>>>>> unlikely.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> That all said, our current reference guide section on
>> >>>> java
>> >>>>>>>>>>>>> versions
>> >>>>>>>>>>>>>>> does
>> >>>>>>>>>>>>>>>>> not sound very confident about JDK11 support.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>> A Note on JDK11 *
>> >>>>>>>>>>>>>>>>>> Preliminary support for JDK11 is introduced with
>> >> HBase
>> >>>>>> 2.3.0.
>> >>>>>>>>>>>>> This
>> >>>>>>>>>>>>>>>>> support is limited to
>> >>>>>>>>>>>>>>>>>> compilation and running the full test suite. There
>> >> are
>> >>>>> open
>> >>>>>>>>>>>>>> questions
>> >>>>>>>>>>>>>>>>> regarding the runtime
>> >>>>>>>>>>>>>>>>>> compatibility of JDK11 with Apache ZooKeeper and
>> >>> Apache
>> >>>>>>>>>> Hadoop
>> >>>>>>>>>>>>>>>>> (HADOOP-15338).
>> >>>>>>>>>>>>>>>>>> Significantly, neither project has yet released a
>> >>>> version
>> >>>>>>>>>> with
>> >>>>>>>>>>>>>>> explicit
>> >>>>>>>>>>>>>>>>> runtime support for
>> >>>>>>>>>>>>>>>>>> JDK11. The remaining known issues in HBase are
>> >>>> catalogued
>> >>>>> in
>> >>>>>>>>>>>>>>>> HBASE-22972.
>> >>>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Since that blurb was written, Hadoop has added JDK11
>> >>>>> support
>> >>>>>>>>>> [1]
>> >>>>>>>>>>>>> as
>> >>>>>>>>>>>>>> has
>> >>>>>>>>>>>>>>>>> ZooKeeper[2]. As a part of buttoning up our JDK11
>> >>> support
>> >>>>> we
>> >>>>>>>>>>> could
>> >>>>>>>>>>>>>>> update
>> >>>>>>>>>>>>>>>>> our minimum supported versions of these projects to
>> >>> match
>> >>>>>> that
>> >>>>>>>>>>>>>> support.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> What do folks think?
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> [1]:
>> >> https://hadoop.apache.org/docs/r3.3.0/index.html
>> >>>>>>>>>>>>>>>>> [2]:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>> https://zookeeper.apache.org/doc/r3.6.0/zookeeperAdmin.html#sc_systemReq
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> Best regards,
>> >>>>>>>>>> Andrew
>> >>>>>>>>>>
>> >>>>>>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>> >>>>>>>>>> It's what we’ve earned
>> >>>>>>>>>> Welcome, apocalypse, what’s taken you so long?
>> >>>>>>>>>> Bring us the fitting end that we’ve been counting on
>> >>>>>>>>>> - A23, Welcome, Apocalypse
>> >>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Best regards,
>> >>>>>> Andrew
>> >>>>>>
>> >>>>>> Unrest, ignorance distilled, nihilistic imbeciles -
>> >>>>>> It's what we’ve earned
>> >>>>>> Welcome, apocalypse, what’s taken you so long?
>> >>>>>> Bring us the fitting end that we’ve been counting on
>> >>>>>> - A23, Welcome, Apocalypse
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Best regards,
>> >>>> Andrew
>> >>>>
>> >>>> Unrest, ignorance distilled, nihilistic imbeciles -
>> >>>> It's what we’ve earned
>> >>>> Welcome, apocalypse, what’s taken you so long?
>> >>>> Bring us the fitting end that we’ve been counting on
>> >>>> - A23, Welcome, Apocalypse
>> >>>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >> Andrew
>> >>
>> >> Unrest, ignorance distilled, nihilistic imbeciles -
>> >> It's what we’ve earned
>> >> Welcome, apocalypse, what’s taken you so long?
>> >> Bring us the fitting end that we’ve been counting on
>> >> - A23, Welcome, Apocalypse
>> >>
>>
>