You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@datasketches.apache.org by leerho <le...@gmail.com> on 2019/10/11 22:13:18 UTC
Help with Migration to OpenJDK 11
Roman,
I hope you are doing well.
I really appreciate the contributions you made to our Memory component!
Given that Druid has made its code base compatible with OpenJDK 11, we
could use your help in what changes do we need to make to make that happen
for DataSketches. As I recall, our DataSketches library was not the only
Druid dependency that took advantage of Unsafe. So I assume you have been
down this path :)
Currently, I have a check in the static initializer in Memory/UnsafeUtil:
parseJavaVersion(...) that checks the string returned from
System.getProperty("java.version"). If that string does not contain "1.8"
or "8" it will throw an error. Given that we have not tested with JDK 9,
10, 11 or 12, I felt it was safer to explicitly throw rather than having
the users experience some weird failure later that may be difficult to
debug.
With only one minor exception (which we could easily fix) the Memory
component is the only place where Unsafe is used. However, we do use
reflection to gain internal access to a number of other classes including
ByteBuffer, Cleaner, and Bits.
Looking forward to hearing from you.
Cheers,
Lee.
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Dave, sorry, clearly I didn’t carefully read your email
:( but I did figure it out!
On Fri, Oct 18, 2019 at 5:42 PM Dave Fisher <wa...@comcast.net> wrote:
> Hi Lee,
>
> DATASKETCHES Jira issues are for the project.
>
> If we want Infrastructure to do something of service to the community then
> you need to open an INFRA ticket. Got it?
>
> Regards?
> Dave
>
> Sent from my iPhone
>
> > On Oct 18, 2019, at 3:53 PM, leerho <le...@gmail.com> wrote:
> >
> > I opened a ticket (DATASKETCHES-3) with infra to see if we could add this
> > capability to our slack channel.
> >
> >> On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
> >>
> >> Hi -
> >>
> >> Apache Pulsar uses this: https://github.com/merlimat/slack-email-digest
> >>
> >> They don’t have their slack channels at the-asf.slack.com but it does
> >> work. You can always ask for infra’s help.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
> >>>
> >>> I wonder if there is a way to have slack auto-post to an email list?
> >>>
> >>>> On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
> >>>>
> >>>> I don't have the expertise on jdk 11 but would love know more about
> >>>> datasketches and specifically the java lib. Would be great if you can
> >> bring
> >>>> back the result of what you have discussed in the slack dev channel.
> >> This
> >>>> helps other contributors easily to search and get to know what
> happened
> >> in
> >>>> the community and get involved :)
> >>>>
> >>>> Best
> >>>> Evans
> >>>>
> >>>>> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
> >>>>>
> >>>>> Christopher,
> >>>>>
> >>>>> This would be fabulous. I have tons of questions, whenever you are
> >>>> ready.
> >>>>> I might be easier to interact via the ASF slack channel
> "datasketches"
> >> or
> >>>>> "datasketches-dev".
> >>>>>
> >>>>> Lee.
> >>>>>
> >>>>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> The memory repository is the one that needs the most conversion
> help.
> >>>>>> Everything using Unsafe in datasketches-java runs through the memory
> >>>>>> package.
> >>>>>>
> >>>>>> I'm not sure off-hand if the characterization repo has low level
> tests
> >>>> of
> >>>>>> Memory (on my phone at the moment), but we'd be looking to minimize
> >> the
> >>>>>> speed impact in the process since it very directly impacts sketch
> >>>> speed.
> >>>>>>
> >>>>>> jon
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> I've converted several other projects to using JDK 11, and can help
> >>>>>>> with this if there's still a need for help. I've actually been
> >>>> looking
> >>>>>>> at ways I can contribute to DataSketches since ApacheCon, and this
> >>>>>>> seems to be something I can probably do.
> >>>>>>>
> >>>>>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Gian,
> >>>>>>>>
> >>>>>>>> If I were to supply you with a link to a snapshot or a Release
> >>>>>> Candidate
> >>>>>>>> version of the datasketches jar (and Memory jar) could you test in
> >>>>> your
> >>>>>>>> environment under JDK 11 and see if it works?
> >>>>>>>>
> >>>>>>>> Lee
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Holy Cow! I have never seen such a complex travis config!
> >>>>>>>>>
> >>>>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I should have linked Druid's Travis config file for reference.
> >>>>> Here
> >>>>>>> it is:
> >>>>>>>>>>
> >>>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >>>>> does
> >>>>>>>>>> compile
> >>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >>>>> 8.
> >>>>>>>>>>>
> >>>>>>>>>>> It depends on who you ask, I guess :)
> >>>>>>>>>>>
> >>>>>>>>>>> From Druid's perspective, we aren't going to be compiling your
> >>>>>> code,
> >>>>>>>>>> only
> >>>>>>>>>>> running it. So the only requirement we would ask for is that
> >>>> the
> >>>>>>> code
> >>>>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
> >>>> Java
> >>>>>> 11
> >>>>>>> is
> >>>>>>>>>> good
> >>>>>>>>>>> for future-proofing your code, though. It is starting to
> >>>> become
> >>>>>> more
> >>>>>>>>>>> prevalent.
> >>>>>>>>>>>
> >>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
> >>>> all
> >>>>>> the
> >>>>>>>>>> direct
> >>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >>>>> what
> >>>>>> is
> >>>>>>>>>>> required.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Is this correct?
> >>>>>>>>>>>
> >>>>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
> >>>>>> that's
> >>>>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
> >>>>> run
> >>>>>>> our
> >>>>>>>>>> tests
> >>>>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
> >>>>> testing
> >>>>>>>>>> easier,
> >>>>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
> >>>> even
> >>>>>>> though
> >>>>>>>>>> we
> >>>>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
> >>>>> done
> >>>>>>> this
> >>>>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
> >>>>> Travis
> >>>>>>>>>> doesn't
> >>>>>>>>>>> easily support.)
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks, Gian, this is VERY helpful!
> >>>>>>>>>>>>
> >>>>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
> >>>>>> Unsafe
> >>>>>>> and
> >>>>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
> >>>>> this
> >>>>>>> would
> >>>>>>>>>>>> work
> >>>>>>>>>>>> with 9+ ), and then defines some method-handles that the
> >>>> other
> >>>>>>> classes
> >>>>>>>>>>>> use.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Will this approach allow code to compile under JDK11 and
> >>>> JDK8?
> >>>>>>>>>>>>
> >>>>>>>>>>>> It also seems that compiling with JDK11 is still a work in
> >>>>>>> progress as
> >>>>>>>>>> the
> >>>>>>>>>>>> pom.xml clearly specifies JDK8.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >>>> does
> >>>>>>> compile
> >>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >>>> 8.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
> >>>>> the
> >>>>>>>>>> direct
> >>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >>>> what
> >>>>>> is
> >>>>>>>>>>>> required.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Is this correct?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Lee.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
> >>>> gian@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> And nobody should be using the Oracle JDK without a
> >>>>> commercial
> >>>>>>>>>>>> relationship
> >>>>>>>>>>>>> with Oracle, its license is a nightmare.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
> >>>>> gian@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
> >>>>> now.
> >>>>>>> The
> >>>>>>>>>>>>>> differences are in terms of licensing, length of support,
> >>>>> and
> >>>>>>>>>> release
> >>>>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
> >>>>>>> third-party
> >>>>>>>>>>>>> OpenJDK
> >>>>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
> >>>>> than
> >>>>>>>>>> vanilla
> >>>>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
> >>>> that
> >>>>>>> public
> >>>>>>>>>>>> updates
> >>>>>>>>>>>>>> will cease for major versions after 6 months.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
> >>>> leerho@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, Gian,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Your references are helpful and I am studying them.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
> >>>>> other
> >>>>>>> hidden
> >>>>>>>>>>>>>>> classes)
> >>>>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
> >>>> far
> >>>>> as
> >>>>>>>>>> Druid is
> >>>>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
> >>>>>> OpenJDK
> >>>>>>>>>> always
> >>>>>>>>>>>>>>> aligned?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
> >>>>>> gian@apache.org
> >>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> You didn't address this message to me, but in Druid
> >>>> most
> >>>>>> of
> >>>>>>> the
> >>>>>>>>>>>> work
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
> >>>> issue,
> >>>>>>> which
> >>>>>>>>>> you
> >>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>> find helpful:
> >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
> >>>>>>>>>>>>>>>> Skimming the list, these might be particularly
> >>>> relevant
> >>>>> to
> >>>>>>>>>>>>> DataSketches:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
> >>>>>>> (cleaner
> >>>>>>>>>>>>>>> operations)
> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
> >>>>>>>>>> (ByteBuffer
> >>>>>>>>>>>>> unmap
> >>>>>>>>>>>>>>>> operation)
> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
> >>>>>>> (Remove
> >>>>>>>>>>>> direct
> >>>>>>>>>>>>>>>> references to Unsafe)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
> >>>>> the
> >>>>>>> list
> >>>>>>>>>> at
> >>>>>>>>>>>>>>> large.
> >>>>>>>>>>>>>>>> It encourages more people to participate. If Roman is
> >>>> a
> >>>>>>>>>> subscriber
> >>>>>>>>>>>> he
> >>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
> >>>>>>> subscribe.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
> >>>>> leerho@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Roman,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I hope you are doing well.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I really appreciate the contributions you made to
> >>>> our
> >>>>>>> Memory
> >>>>>>>>>>>>>>> component!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Given that Druid has made its code base compatible
> >>>>> with
> >>>>>>>>>> OpenJDK
> >>>>>>>>>>>> 11,
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> could use your help in what changes do we need to
> >>>> make
> >>>>>> to
> >>>>>>> make
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> happen
> >>>>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
> >>>>> library
> >>>>>>> was
> >>>>>>>>>> not
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
> >>>> I
> >>>>>>> assume
> >>>>>>>>>> you
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>> down this path :)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Currently, I have a check in the static initializer
> >>>> in
> >>>>>>>>>>>>>>> Memory/UnsafeUtil:
> >>>>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
> >>>> returned
> >>>>>> from
> >>>>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
> >>>>> does
> >>>>>>> not
> >>>>>>>>>>>> contain
> >>>>>>>>>>>>>>>> "1.8"
> >>>>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
> >>>> not
> >>>>>>> tested
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>>> JDK 9,
> >>>>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
> >>>> throw
> >>>>>>> rather
> >>>>>>>>>> than
> >>>>>>>>>>>>>>> having
> >>>>>>>>>>>>>>>>> the users experience some weird failure later that
> >>>> may
> >>>>>> be
> >>>>>>>>>>>> difficult
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> debug.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> With only one minor exception (which we could easily
> >>>>>> fix)
> >>>>>>> the
> >>>>>>>>>>>> Memory
> >>>>>>>>>>>>>>>>> component is the only place where Unsafe is used.
> >>>>>>> However,
> >>>>>>>>>> we do
> >>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> reflection to gain internal access to a number of
> >>>>> other
> >>>>>>>>>> classes
> >>>>>>>>>>>>>>> including
> >>>>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Looking forward to hearing from you.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>> --
> >>> From my cell phone.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> >> For additional commands, e-mail: dev-help@datasketches.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> For additional commands, e-mail: dev-help@datasketches.apache.org
>
> --
From my cell phone.
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
I guess not. I guess the "DATASKETCHES-XXX" tickets are against our own
team?
I moved the "slack" ticket to an INFRA-19303.
On Fri, Oct 18, 2019 at 5:48 PM leerho <le...@gmail.com> wrote:
> I thought that is what I did. Isn't the DATASKETCHES-3 ticket I opened
> the same thing?
>
>
> On Fri, Oct 18, 2019 at 5:42 PM Dave Fisher <wa...@comcast.net> wrote:
>
>> Hi Lee,
>>
>> DATASKETCHES Jira issues are for the project.
>>
>> If we want Infrastructure to do something of service to the community
>> then you need to open an INFRA ticket. Got it?
>>
>> Regards?
>> Dave
>>
>> Sent from my iPhone
>>
>> > On Oct 18, 2019, at 3:53 PM, leerho <le...@gmail.com> wrote:
>> >
>> > I opened a ticket (DATASKETCHES-3) with infra to see if we could add
>> this
>> > capability to our slack channel.
>> >
>> >> On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
>> >>
>> >> Hi -
>> >>
>> >> Apache Pulsar uses this:
>> https://github.com/merlimat/slack-email-digest
>> >>
>> >> They don’t have their slack channels at the-asf.slack.com but it does
>> >> work. You can always ask for infra’s help.
>> >>
>> >> Regards,
>> >> Dave
>> >>
>> >>> On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
>> >>>
>> >>> I wonder if there is a way to have slack auto-post to an email list?
>> >>>
>> >>>> On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
>> >>>>
>> >>>> I don't have the expertise on jdk 11 but would love know more about
>> >>>> datasketches and specifically the java lib. Would be great if you can
>> >> bring
>> >>>> back the result of what you have discussed in the slack dev channel.
>> >> This
>> >>>> helps other contributors easily to search and get to know what
>> happened
>> >> in
>> >>>> the community and get involved :)
>> >>>>
>> >>>> Best
>> >>>> Evans
>> >>>>
>> >>>>> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
>> >>>>>
>> >>>>> Christopher,
>> >>>>>
>> >>>>> This would be fabulous. I have tons of questions, whenever you are
>> >>>> ready.
>> >>>>> I might be easier to interact via the ASF slack channel
>> "datasketches"
>> >> or
>> >>>>> "datasketches-dev".
>> >>>>>
>> >>>>> Lee.
>> >>>>>
>> >>>>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
>> >> wrote:
>> >>>>>
>> >>>>>> The memory repository is the one that needs the most conversion
>> help.
>> >>>>>> Everything using Unsafe in datasketches-java runs through the
>> memory
>> >>>>>> package.
>> >>>>>>
>> >>>>>> I'm not sure off-hand if the characterization repo has low level
>> tests
>> >>>> of
>> >>>>>> Memory (on my phone at the moment), but we'd be looking to minimize
>> >> the
>> >>>>>> speed impact in the process since it very directly impacts sketch
>> >>>> speed.
>> >>>>>>
>> >>>>>> jon
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
>> >> wrote:
>> >>>>>>
>> >>>>>>> I've converted several other projects to using JDK 11, and can
>> help
>> >>>>>>> with this if there's still a need for help. I've actually been
>> >>>> looking
>> >>>>>>> at ways I can contribute to DataSketches since ApacheCon, and this
>> >>>>>>> seems to be something I can probably do.
>> >>>>>>>
>> >>>>>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>> Gian,
>> >>>>>>>>
>> >>>>>>>> If I were to supply you with a link to a snapshot or a Release
>> >>>>>> Candidate
>> >>>>>>>> version of the datasketches jar (and Memory jar) could you test
>> in
>> >>>>> your
>> >>>>>>>> environment under JDK 11 and see if it works?
>> >>>>>>>>
>> >>>>>>>> Lee
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com>
>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Holy Cow! I have never seen such a complex travis config!
>> >>>>>>>>>
>> >>>>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
>> >>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> I should have linked Druid's Travis config file for reference.
>> >>>>> Here
>> >>>>>>> it is:
>> >>>>>>>>>>
>> >>>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
>> >>>>>>>>>>
>> >>>>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
>> >>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>> >>>>> does
>> >>>>>>>>>> compile
>> >>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>> >>>>> 8.
>> >>>>>>>>>>>
>> >>>>>>>>>>> It depends on who you ask, I guess :)
>> >>>>>>>>>>>
>> >>>>>>>>>>> From Druid's perspective, we aren't going to be compiling your
>> >>>>>> code,
>> >>>>>>>>>> only
>> >>>>>>>>>>> running it. So the only requirement we would ask for is that
>> >>>> the
>> >>>>>>> code
>> >>>>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
>> >>>> Java
>> >>>>>> 11
>> >>>>>>> is
>> >>>>>>>>>> good
>> >>>>>>>>>>> for future-proofing your code, though. It is starting to
>> >>>> become
>> >>>>>> more
>> >>>>>>>>>>> prevalent.
>> >>>>>>>>>>>
>> >>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
>> >>>> all
>> >>>>>> the
>> >>>>>>>>>> direct
>> >>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>> >>>>> what
>> >>>>>> is
>> >>>>>>>>>>> required.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Is this correct?
>> >>>>>>>>>>>
>> >>>>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
>> >>>>>> that's
>> >>>>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
>> >>>>> run
>> >>>>>>> our
>> >>>>>>>>>> tests
>> >>>>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
>> >>>>> testing
>> >>>>>>>>>> easier,
>> >>>>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
>> >>>> even
>> >>>>>>> though
>> >>>>>>>>>> we
>> >>>>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
>> >>>>> done
>> >>>>>>> this
>> >>>>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
>> >>>>> Travis
>> >>>>>>>>>> doesn't
>> >>>>>>>>>>> easily support.)
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
>> >>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Thanks, Gian, this is VERY helpful!
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
>> >>>>>> Unsafe
>> >>>>>>> and
>> >>>>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
>> >>>>> this
>> >>>>>>> would
>> >>>>>>>>>>>> work
>> >>>>>>>>>>>> with 9+ ), and then defines some method-handles that the
>> >>>> other
>> >>>>>>> classes
>> >>>>>>>>>>>> use.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Will this approach allow code to compile under JDK11 and
>> >>>> JDK8?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> It also seems that compiling with JDK11 is still a work in
>> >>>>>>> progress as
>> >>>>>>>>>> the
>> >>>>>>>>>>>> pom.xml clearly specifies JDK8.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>> >>>> does
>> >>>>>>> compile
>> >>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>> >>>> 8.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
>> >>>>> the
>> >>>>>>>>>> direct
>> >>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>> >>>> what
>> >>>>>> is
>> >>>>>>>>>>>> required.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Is this correct?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Lee.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
>> >>>> gian@apache.org>
>> >>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> And nobody should be using the Oracle JDK without a
>> >>>>> commercial
>> >>>>>>>>>>>> relationship
>> >>>>>>>>>>>>> with Oracle, its license is a nightmare.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
>> >>>>> gian@apache.org>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
>> >>>>> now.
>> >>>>>>> The
>> >>>>>>>>>>>>>> differences are in terms of licensing, length of support,
>> >>>>> and
>> >>>>>>>>>> release
>> >>>>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
>> >>>>>>> third-party
>> >>>>>>>>>>>>> OpenJDK
>> >>>>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
>> >>>>> than
>> >>>>>>>>>> vanilla
>> >>>>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
>> >>>> that
>> >>>>>>> public
>> >>>>>>>>>>>> updates
>> >>>>>>>>>>>>>> will cease for major versions after 6 months.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
>> >>>> leerho@gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Thanks, Gian,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Your references are helpful and I am studying them.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
>> >>>>> other
>> >>>>>>> hidden
>> >>>>>>>>>>>>>>> classes)
>> >>>>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
>> >>>> far
>> >>>>> as
>> >>>>>>>>>> Druid is
>> >>>>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
>> >>>>>> OpenJDK
>> >>>>>>>>>> always
>> >>>>>>>>>>>>>>> aligned?
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Lee.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
>> >>>>>> gian@apache.org
>> >>>>>>>>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> You didn't address this message to me, but in Druid
>> >>>> most
>> >>>>>> of
>> >>>>>>> the
>> >>>>>>>>>>>> work
>> >>>>>>>>>>>>> for
>> >>>>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
>> >>>> issue,
>> >>>>>>> which
>> >>>>>>>>>> you
>> >>>>>>>>>>>>> might
>> >>>>>>>>>>>>>>>> find helpful:
>> >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
>> >>>>>>>>>>>>>>>> Skimming the list, these might be particularly
>> >>>> relevant
>> >>>>> to
>> >>>>>>>>>>>>> DataSketches:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
>> >>>>>>> (cleaner
>> >>>>>>>>>>>>>>> operations)
>> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
>> >>>>>>>>>> (ByteBuffer
>> >>>>>>>>>>>>> unmap
>> >>>>>>>>>>>>>>>> operation)
>> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
>> >>>>>>> (Remove
>> >>>>>>>>>>>> direct
>> >>>>>>>>>>>>>>>> references to Unsafe)
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
>> >>>>> the
>> >>>>>>> list
>> >>>>>>>>>> at
>> >>>>>>>>>>>>>>> large.
>> >>>>>>>>>>>>>>>> It encourages more people to participate. If Roman is
>> >>>> a
>> >>>>>>>>>> subscriber
>> >>>>>>>>>>>> he
>> >>>>>>>>>>>>>>> would
>> >>>>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
>> >>>>>>> subscribe.
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
>> >>>>> leerho@gmail.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Roman,
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I hope you are doing well.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I really appreciate the contributions you made to
>> >>>> our
>> >>>>>>> Memory
>> >>>>>>>>>>>>>>> component!
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Given that Druid has made its code base compatible
>> >>>>> with
>> >>>>>>>>>> OpenJDK
>> >>>>>>>>>>>> 11,
>> >>>>>>>>>>>>> we
>> >>>>>>>>>>>>>>>>> could use your help in what changes do we need to
>> >>>> make
>> >>>>>> to
>> >>>>>>> make
>> >>>>>>>>>>>> that
>> >>>>>>>>>>>>>>>> happen
>> >>>>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
>> >>>>> library
>> >>>>>>> was
>> >>>>>>>>>> not
>> >>>>>>>>>>>> the
>> >>>>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
>> >>>> I
>> >>>>>>> assume
>> >>>>>>>>>> you
>> >>>>>>>>>>>>> have
>> >>>>>>>>>>>>>>>> been
>> >>>>>>>>>>>>>>>>> down this path :)
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Currently, I have a check in the static initializer
>> >>>> in
>> >>>>>>>>>>>>>>> Memory/UnsafeUtil:
>> >>>>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
>> >>>> returned
>> >>>>>> from
>> >>>>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
>> >>>>> does
>> >>>>>>> not
>> >>>>>>>>>>>> contain
>> >>>>>>>>>>>>>>>> "1.8"
>> >>>>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
>> >>>> not
>> >>>>>>> tested
>> >>>>>>>>>>>> with
>> >>>>>>>>>>>>>>> JDK 9,
>> >>>>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
>> >>>> throw
>> >>>>>>> rather
>> >>>>>>>>>> than
>> >>>>>>>>>>>>>>> having
>> >>>>>>>>>>>>>>>>> the users experience some weird failure later that
>> >>>> may
>> >>>>>> be
>> >>>>>>>>>>>> difficult
>> >>>>>>>>>>>>> to
>> >>>>>>>>>>>>>>>>> debug.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> With only one minor exception (which we could easily
>> >>>>>> fix)
>> >>>>>>> the
>> >>>>>>>>>>>> Memory
>> >>>>>>>>>>>>>>>>> component is the only place where Unsafe is used.
>> >>>>>>> However,
>> >>>>>>>>>> we do
>> >>>>>>>>>>>>> use
>> >>>>>>>>>>>>>>>>> reflection to gain internal access to a number of
>> >>>>> other
>> >>>>>>>>>> classes
>> >>>>>>>>>>>>>>> including
>> >>>>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Looking forward to hearing from you.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Lee.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>
>> >>>>>>>
>> ---------------------------------------------------------------------
>> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>> >>>>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>> --
>> >>> From my cell phone.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>> >> For additional commands, e-mail: dev-help@datasketches.apache.org
>> >>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>> For additional commands, e-mail: dev-help@datasketches.apache.org
>>
>>
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
I thought that is what I did. Isn't the DATASKETCHES-3 ticket I opened the
same thing?
On Fri, Oct 18, 2019 at 5:42 PM Dave Fisher <wa...@comcast.net> wrote:
> Hi Lee,
>
> DATASKETCHES Jira issues are for the project.
>
> If we want Infrastructure to do something of service to the community then
> you need to open an INFRA ticket. Got it?
>
> Regards?
> Dave
>
> Sent from my iPhone
>
> > On Oct 18, 2019, at 3:53 PM, leerho <le...@gmail.com> wrote:
> >
> > I opened a ticket (DATASKETCHES-3) with infra to see if we could add this
> > capability to our slack channel.
> >
> >> On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
> >>
> >> Hi -
> >>
> >> Apache Pulsar uses this: https://github.com/merlimat/slack-email-digest
> >>
> >> They don’t have their slack channels at the-asf.slack.com but it does
> >> work. You can always ask for infra’s help.
> >>
> >> Regards,
> >> Dave
> >>
> >>> On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
> >>>
> >>> I wonder if there is a way to have slack auto-post to an email list?
> >>>
> >>>> On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
> >>>>
> >>>> I don't have the expertise on jdk 11 but would love know more about
> >>>> datasketches and specifically the java lib. Would be great if you can
> >> bring
> >>>> back the result of what you have discussed in the slack dev channel.
> >> This
> >>>> helps other contributors easily to search and get to know what
> happened
> >> in
> >>>> the community and get involved :)
> >>>>
> >>>> Best
> >>>> Evans
> >>>>
> >>>>> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
> >>>>>
> >>>>> Christopher,
> >>>>>
> >>>>> This would be fabulous. I have tons of questions, whenever you are
> >>>> ready.
> >>>>> I might be easier to interact via the ASF slack channel
> "datasketches"
> >> or
> >>>>> "datasketches-dev".
> >>>>>
> >>>>> Lee.
> >>>>>
> >>>>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> The memory repository is the one that needs the most conversion
> help.
> >>>>>> Everything using Unsafe in datasketches-java runs through the memory
> >>>>>> package.
> >>>>>>
> >>>>>> I'm not sure off-hand if the characterization repo has low level
> tests
> >>>> of
> >>>>>> Memory (on my phone at the moment), but we'd be looking to minimize
> >> the
> >>>>>> speed impact in the process since it very directly impacts sketch
> >>>> speed.
> >>>>>>
> >>>>>> jon
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
> >> wrote:
> >>>>>>
> >>>>>>> I've converted several other projects to using JDK 11, and can help
> >>>>>>> with this if there's still a need for help. I've actually been
> >>>> looking
> >>>>>>> at ways I can contribute to DataSketches since ApacheCon, and this
> >>>>>>> seems to be something I can probably do.
> >>>>>>>
> >>>>>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Gian,
> >>>>>>>>
> >>>>>>>> If I were to supply you with a link to a snapshot or a Release
> >>>>>> Candidate
> >>>>>>>> version of the datasketches jar (and Memory jar) could you test in
> >>>>> your
> >>>>>>>> environment under JDK 11 and see if it works?
> >>>>>>>>
> >>>>>>>> Lee
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Holy Cow! I have never seen such a complex travis config!
> >>>>>>>>>
> >>>>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> I should have linked Druid's Travis config file for reference.
> >>>>> Here
> >>>>>>> it is:
> >>>>>>>>>>
> >>>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
> >>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >>>>> does
> >>>>>>>>>> compile
> >>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >>>>> 8.
> >>>>>>>>>>>
> >>>>>>>>>>> It depends on who you ask, I guess :)
> >>>>>>>>>>>
> >>>>>>>>>>> From Druid's perspective, we aren't going to be compiling your
> >>>>>> code,
> >>>>>>>>>> only
> >>>>>>>>>>> running it. So the only requirement we would ask for is that
> >>>> the
> >>>>>>> code
> >>>>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
> >>>> Java
> >>>>>> 11
> >>>>>>> is
> >>>>>>>>>> good
> >>>>>>>>>>> for future-proofing your code, though. It is starting to
> >>>> become
> >>>>>> more
> >>>>>>>>>>> prevalent.
> >>>>>>>>>>>
> >>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
> >>>> all
> >>>>>> the
> >>>>>>>>>> direct
> >>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >>>>> what
> >>>>>> is
> >>>>>>>>>>> required.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Is this correct?
> >>>>>>>>>>>
> >>>>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
> >>>>>> that's
> >>>>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
> >>>>> run
> >>>>>>> our
> >>>>>>>>>> tests
> >>>>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
> >>>>> testing
> >>>>>>>>>> easier,
> >>>>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
> >>>> even
> >>>>>>> though
> >>>>>>>>>> we
> >>>>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
> >>>>> done
> >>>>>>> this
> >>>>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
> >>>>> Travis
> >>>>>>>>>> doesn't
> >>>>>>>>>>> easily support.)
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Thanks, Gian, this is VERY helpful!
> >>>>>>>>>>>>
> >>>>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
> >>>>>> Unsafe
> >>>>>>> and
> >>>>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
> >>>>> this
> >>>>>>> would
> >>>>>>>>>>>> work
> >>>>>>>>>>>> with 9+ ), and then defines some method-handles that the
> >>>> other
> >>>>>>> classes
> >>>>>>>>>>>> use.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Will this approach allow code to compile under JDK11 and
> >>>> JDK8?
> >>>>>>>>>>>>
> >>>>>>>>>>>> It also seems that compiling with JDK11 is still a work in
> >>>>>>> progress as
> >>>>>>>>>> the
> >>>>>>>>>>>> pom.xml clearly specifies JDK8.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >>>> does
> >>>>>>> compile
> >>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >>>> 8.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
> >>>>> the
> >>>>>>>>>> direct
> >>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >>>> what
> >>>>>> is
> >>>>>>>>>>>> required.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Is this correct?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Lee.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
> >>>> gian@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> And nobody should be using the Oracle JDK without a
> >>>>> commercial
> >>>>>>>>>>>> relationship
> >>>>>>>>>>>>> with Oracle, its license is a nightmare.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
> >>>>> gian@apache.org>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
> >>>>> now.
> >>>>>>> The
> >>>>>>>>>>>>>> differences are in terms of licensing, length of support,
> >>>>> and
> >>>>>>>>>> release
> >>>>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
> >>>>>>> third-party
> >>>>>>>>>>>>> OpenJDK
> >>>>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
> >>>>> than
> >>>>>>>>>> vanilla
> >>>>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
> >>>> that
> >>>>>>> public
> >>>>>>>>>>>> updates
> >>>>>>>>>>>>>> will cease for major versions after 6 months.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
> >>>> leerho@gmail.com>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks, Gian,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Your references are helpful and I am studying them.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
> >>>>> other
> >>>>>>> hidden
> >>>>>>>>>>>>>>> classes)
> >>>>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
> >>>> far
> >>>>> as
> >>>>>>>>>> Druid is
> >>>>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
> >>>>>> OpenJDK
> >>>>>>>>>> always
> >>>>>>>>>>>>>>> aligned?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
> >>>>>> gian@apache.org
> >>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> You didn't address this message to me, but in Druid
> >>>> most
> >>>>>> of
> >>>>>>> the
> >>>>>>>>>>>> work
> >>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
> >>>> issue,
> >>>>>>> which
> >>>>>>>>>> you
> >>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>> find helpful:
> >>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
> >>>>>>>>>>>>>>>> Skimming the list, these might be particularly
> >>>> relevant
> >>>>> to
> >>>>>>>>>>>>> DataSketches:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
> >>>>>>> (cleaner
> >>>>>>>>>>>>>>> operations)
> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
> >>>>>>>>>> (ByteBuffer
> >>>>>>>>>>>>> unmap
> >>>>>>>>>>>>>>>> operation)
> >>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
> >>>>>>> (Remove
> >>>>>>>>>>>> direct
> >>>>>>>>>>>>>>>> references to Unsafe)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
> >>>>> the
> >>>>>>> list
> >>>>>>>>>> at
> >>>>>>>>>>>>>>> large.
> >>>>>>>>>>>>>>>> It encourages more people to participate. If Roman is
> >>>> a
> >>>>>>>>>> subscriber
> >>>>>>>>>>>> he
> >>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
> >>>>>>> subscribe.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
> >>>>> leerho@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Roman,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I hope you are doing well.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I really appreciate the contributions you made to
> >>>> our
> >>>>>>> Memory
> >>>>>>>>>>>>>>> component!
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Given that Druid has made its code base compatible
> >>>>> with
> >>>>>>>>>> OpenJDK
> >>>>>>>>>>>> 11,
> >>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> could use your help in what changes do we need to
> >>>> make
> >>>>>> to
> >>>>>>> make
> >>>>>>>>>>>> that
> >>>>>>>>>>>>>>>> happen
> >>>>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
> >>>>> library
> >>>>>>> was
> >>>>>>>>>> not
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
> >>>> I
> >>>>>>> assume
> >>>>>>>>>> you
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>> down this path :)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Currently, I have a check in the static initializer
> >>>> in
> >>>>>>>>>>>>>>> Memory/UnsafeUtil:
> >>>>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
> >>>> returned
> >>>>>> from
> >>>>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
> >>>>> does
> >>>>>>> not
> >>>>>>>>>>>> contain
> >>>>>>>>>>>>>>>> "1.8"
> >>>>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
> >>>> not
> >>>>>>> tested
> >>>>>>>>>>>> with
> >>>>>>>>>>>>>>> JDK 9,
> >>>>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
> >>>> throw
> >>>>>>> rather
> >>>>>>>>>> than
> >>>>>>>>>>>>>>> having
> >>>>>>>>>>>>>>>>> the users experience some weird failure later that
> >>>> may
> >>>>>> be
> >>>>>>>>>>>> difficult
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>> debug.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> With only one minor exception (which we could easily
> >>>>>> fix)
> >>>>>>> the
> >>>>>>>>>>>> Memory
> >>>>>>>>>>>>>>>>> component is the only place where Unsafe is used.
> >>>>>>> However,
> >>>>>>>>>> we do
> >>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>> reflection to gain internal access to a number of
> >>>>> other
> >>>>>>>>>> classes
> >>>>>>>>>>>>>>> including
> >>>>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Looking forward to hearing from you.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>> --
> >>> From my cell phone.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> >> For additional commands, e-mail: dev-help@datasketches.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> For additional commands, e-mail: dev-help@datasketches.apache.org
>
>
Re: Help with Migration to OpenJDK 11
Posted by Dave Fisher <wa...@comcast.net>.
Hi Lee,
DATASKETCHES Jira issues are for the project.
If we want Infrastructure to do something of service to the community then you need to open an INFRA ticket. Got it?
Regards?
Dave
Sent from my iPhone
> On Oct 18, 2019, at 3:53 PM, leerho <le...@gmail.com> wrote:
>
> I opened a ticket (DATASKETCHES-3) with infra to see if we could add this
> capability to our slack channel.
>
>> On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
>>
>> Hi -
>>
>> Apache Pulsar uses this: https://github.com/merlimat/slack-email-digest
>>
>> They don’t have their slack channels at the-asf.slack.com but it does
>> work. You can always ask for infra’s help.
>>
>> Regards,
>> Dave
>>
>>> On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
>>>
>>> I wonder if there is a way to have slack auto-post to an email list?
>>>
>>>> On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
>>>>
>>>> I don't have the expertise on jdk 11 but would love know more about
>>>> datasketches and specifically the java lib. Would be great if you can
>> bring
>>>> back the result of what you have discussed in the slack dev channel.
>> This
>>>> helps other contributors easily to search and get to know what happened
>> in
>>>> the community and get involved :)
>>>>
>>>> Best
>>>> Evans
>>>>
>>>>> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
>>>>>
>>>>> Christopher,
>>>>>
>>>>> This would be fabulous. I have tons of questions, whenever you are
>>>> ready.
>>>>> I might be easier to interact via the ASF slack channel "datasketches"
>> or
>>>>> "datasketches-dev".
>>>>>
>>>>> Lee.
>>>>>
>>>>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
>> wrote:
>>>>>
>>>>>> The memory repository is the one that needs the most conversion help.
>>>>>> Everything using Unsafe in datasketches-java runs through the memory
>>>>>> package.
>>>>>>
>>>>>> I'm not sure off-hand if the characterization repo has low level tests
>>>> of
>>>>>> Memory (on my phone at the moment), but we'd be looking to minimize
>> the
>>>>>> speed impact in the process since it very directly impacts sketch
>>>> speed.
>>>>>>
>>>>>> jon
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
>> wrote:
>>>>>>
>>>>>>> I've converted several other projects to using JDK 11, and can help
>>>>>>> with this if there's still a need for help. I've actually been
>>>> looking
>>>>>>> at ways I can contribute to DataSketches since ApacheCon, and this
>>>>>>> seems to be something I can probably do.
>>>>>>>
>>>>>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Gian,
>>>>>>>>
>>>>>>>> If I were to supply you with a link to a snapshot or a Release
>>>>>> Candidate
>>>>>>>> version of the datasketches jar (and Memory jar) could you test in
>>>>> your
>>>>>>>> environment under JDK 11 and see if it works?
>>>>>>>>
>>>>>>>> Lee
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Holy Cow! I have never seen such a complex travis config!
>>>>>>>>>
>>>>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I should have linked Druid's Travis config file for reference.
>>>>> Here
>>>>>>> it is:
>>>>>>>>>>
>>>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>>>>> does
>>>>>>>>>> compile
>>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>>>>> 8.
>>>>>>>>>>>
>>>>>>>>>>> It depends on who you ask, I guess :)
>>>>>>>>>>>
>>>>>>>>>>> From Druid's perspective, we aren't going to be compiling your
>>>>>> code,
>>>>>>>>>> only
>>>>>>>>>>> running it. So the only requirement we would ask for is that
>>>> the
>>>>>>> code
>>>>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
>>>> Java
>>>>>> 11
>>>>>>> is
>>>>>>>>>> good
>>>>>>>>>>> for future-proofing your code, though. It is starting to
>>>> become
>>>>>> more
>>>>>>>>>>> prevalent.
>>>>>>>>>>>
>>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
>>>> all
>>>>>> the
>>>>>>>>>> direct
>>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>>>>> what
>>>>>> is
>>>>>>>>>>> required.
>>>>>>>>>>>>
>>>>>>>>>>>> Is this correct?
>>>>>>>>>>>
>>>>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
>>>>>> that's
>>>>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
>>>>> run
>>>>>>> our
>>>>>>>>>> tests
>>>>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
>>>>> testing
>>>>>>>>>> easier,
>>>>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
>>>> even
>>>>>>> though
>>>>>>>>>> we
>>>>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
>>>>> done
>>>>>>> this
>>>>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
>>>>> Travis
>>>>>>>>>> doesn't
>>>>>>>>>>> easily support.)
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks, Gian, this is VERY helpful!
>>>>>>>>>>>>
>>>>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
>>>>>> Unsafe
>>>>>>> and
>>>>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
>>>>> this
>>>>>>> would
>>>>>>>>>>>> work
>>>>>>>>>>>> with 9+ ), and then defines some method-handles that the
>>>> other
>>>>>>> classes
>>>>>>>>>>>> use.
>>>>>>>>>>>>
>>>>>>>>>>>> Will this approach allow code to compile under JDK11 and
>>>> JDK8?
>>>>>>>>>>>>
>>>>>>>>>>>> It also seems that compiling with JDK11 is still a work in
>>>>>>> progress as
>>>>>>>>>> the
>>>>>>>>>>>> pom.xml clearly specifies JDK8.
>>>>>>>>>>>>
>>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>>>> does
>>>>>>> compile
>>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>>>> 8.
>>>>>>>>>>>>
>>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
>>>>> the
>>>>>>>>>> direct
>>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>>>> what
>>>>>> is
>>>>>>>>>>>> required.
>>>>>>>>>>>>
>>>>>>>>>>>> Is this correct?
>>>>>>>>>>>>
>>>>>>>>>>>> Lee.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
>>>> gian@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> And nobody should be using the Oracle JDK without a
>>>>> commercial
>>>>>>>>>>>> relationship
>>>>>>>>>>>>> with Oracle, its license is a nightmare.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
>>>>> gian@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
>>>>> now.
>>>>>>> The
>>>>>>>>>>>>>> differences are in terms of licensing, length of support,
>>>>> and
>>>>>>>>>> release
>>>>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
>>>>>>> third-party
>>>>>>>>>>>>> OpenJDK
>>>>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
>>>>> than
>>>>>>>>>> vanilla
>>>>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
>>>> that
>>>>>>> public
>>>>>>>>>>>> updates
>>>>>>>>>>>>>> will cease for major versions after 6 months.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
>>>> leerho@gmail.com>
>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks, Gian,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Your references are helpful and I am studying them.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
>>>>> other
>>>>>>> hidden
>>>>>>>>>>>>>>> classes)
>>>>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
>>>> far
>>>>> as
>>>>>>>>>> Druid is
>>>>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
>>>>>> OpenJDK
>>>>>>>>>> always
>>>>>>>>>>>>>>> aligned?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Lee.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
>>>>>> gian@apache.org
>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> You didn't address this message to me, but in Druid
>>>> most
>>>>>> of
>>>>>>> the
>>>>>>>>>>>> work
>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
>>>> issue,
>>>>>>> which
>>>>>>>>>> you
>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>> find helpful:
>>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
>>>>>>>>>>>>>>>> Skimming the list, these might be particularly
>>>> relevant
>>>>> to
>>>>>>>>>>>>> DataSketches:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
>>>>>>> (cleaner
>>>>>>>>>>>>>>> operations)
>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
>>>>>>>>>> (ByteBuffer
>>>>>>>>>>>>> unmap
>>>>>>>>>>>>>>>> operation)
>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
>>>>>>> (Remove
>>>>>>>>>>>> direct
>>>>>>>>>>>>>>>> references to Unsafe)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
>>>>> the
>>>>>>> list
>>>>>>>>>> at
>>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>>>> It encourages more people to participate. If Roman is
>>>> a
>>>>>>>>>> subscriber
>>>>>>>>>>>> he
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
>>>>>>> subscribe.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
>>>>> leerho@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Roman,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I hope you are doing well.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I really appreciate the contributions you made to
>>>> our
>>>>>>> Memory
>>>>>>>>>>>>>>> component!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Given that Druid has made its code base compatible
>>>>> with
>>>>>>>>>> OpenJDK
>>>>>>>>>>>> 11,
>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>> could use your help in what changes do we need to
>>>> make
>>>>>> to
>>>>>>> make
>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
>>>>> library
>>>>>>> was
>>>>>>>>>> not
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
>>>> I
>>>>>>> assume
>>>>>>>>>> you
>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>> down this path :)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Currently, I have a check in the static initializer
>>>> in
>>>>>>>>>>>>>>> Memory/UnsafeUtil:
>>>>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
>>>> returned
>>>>>> from
>>>>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
>>>>> does
>>>>>>> not
>>>>>>>>>>>> contain
>>>>>>>>>>>>>>>> "1.8"
>>>>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
>>>> not
>>>>>>> tested
>>>>>>>>>>>> with
>>>>>>>>>>>>>>> JDK 9,
>>>>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
>>>> throw
>>>>>>> rather
>>>>>>>>>> than
>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>> the users experience some weird failure later that
>>>> may
>>>>>> be
>>>>>>>>>>>> difficult
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> debug.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> With only one minor exception (which we could easily
>>>>>> fix)
>>>>>>> the
>>>>>>>>>>>> Memory
>>>>>>>>>>>>>>>>> component is the only place where Unsafe is used.
>>>>>>> However,
>>>>>>>>>> we do
>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>> reflection to gain internal access to a number of
>>>>> other
>>>>>>>>>> classes
>>>>>>>>>>>>>>> including
>>>>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Looking forward to hearing from you.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Lee.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>>>>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>> --
>>> From my cell phone.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>> For additional commands, e-mail: dev-help@datasketches.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Ah! Thank you! I will file a request with INFRA.
Lee.
On Sat, Oct 19, 2019 at 9:23 AM Dave Fisher <wa...@apache.org> wrote:
> Hi Lee,
>
> > On Oct 18, 2019, at 5:01 PM, leerho <le...@gmail.com> wrote:
> >
> > Meanwhile, back to the topic of Migration to OpenJDK11 ...
> >
> > Unfortunately, the Apache email lists do not accommodate images or files.
>
> While true by default, Infra will enable images/files if the podling
> requests.
>
> Regards,
> Dave
>
> > So I am going to move this next discussion to our asf slack channel
> > "datasketches-dev".
> >
> > I have posted my study there. You can offer comments or suggestions
> > either here or there.
> >
> > Cheers,
> >
> > Lee.
> >
> > On Fri, Oct 18, 2019 at 3:53 PM leerho <le...@gmail.com> wrote:
> >
> >> I opened a ticket (DATASKETCHES-3) with infra to see if we could add
> this
> >> capability to our slack channel.
> >>
> >> On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
> >>
> >>> Hi -
> >>>
> >>> Apache Pulsar uses this:
> https://github.com/merlimat/slack-email-digest
> >>>
> >>> They don’t have their slack channels at the-asf.slack.com but it does
> >>> work. You can always ask for infra’s help.
> >>>
> >>> Regards,
> >>> Dave
> >>>
> >>>> On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
> >>>>
> >>>> I wonder if there is a way to have slack auto-post to an email list?
> >>>>
> >>>> On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
> >>>>
> >>>>> I don't have the expertise on jdk 11 but would love know more about
> >>>>> datasketches and specifically the java lib. Would be great if you can
> >>> bring
> >>>>> back the result of what you have discussed in the slack dev channel.
> >>> This
> >>>>> helps other contributors easily to search and get to know what
> >>> happened in
> >>>>> the community and get involved :)
> >>>>>
> >>>>> Best
> >>>>> Evans
> >>>>>
> >>>>> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
> >>>>>
> >>>>>> Christopher,
> >>>>>>
> >>>>>> This would be fabulous. I have tons of questions, whenever you are
> >>>>> ready.
> >>>>>> I might be easier to interact via the ASF slack channel
> >>> "datasketches" or
> >>>>>> "datasketches-dev".
> >>>>>>
> >>>>>> Lee.
> >>>>>>
> >>>>>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>>> The memory repository is the one that needs the most conversion
> help.
> >>>>>>> Everything using Unsafe in datasketches-java runs through the
> memory
> >>>>>>> package.
> >>>>>>>
> >>>>>>> I'm not sure off-hand if the characterization repo has low level
> >>> tests
> >>>>> of
> >>>>>>> Memory (on my phone at the moment), but we'd be looking to minimize
> >>> the
> >>>>>>> speed impact in the process since it very directly impacts sketch
> >>>>> speed.
> >>>>>>>
> >>>>>>> jon
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
> >>> wrote:
> >>>>>>>
> >>>>>>>> I've converted several other projects to using JDK 11, and can
> help
> >>>>>>>> with this if there's still a need for help. I've actually been
> >>>>> looking
> >>>>>>>> at ways I can contribute to DataSketches since ApacheCon, and this
> >>>>>>>> seems to be something I can probably do.
> >>>>>>>>
> >>>>>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Gian,
> >>>>>>>>>
> >>>>>>>>> If I were to supply you with a link to a snapshot or a Release
> >>>>>>> Candidate
> >>>>>>>>> version of the datasketches jar (and Memory jar) could you test
> in
> >>>>>> your
> >>>>>>>>> environment under JDK 11 and see if it works?
> >>>>>>>>>
> >>>>>>>>> Lee
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>>> Holy Cow! I have never seen such a complex travis config!
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> I should have linked Druid's Travis config file for reference.
> >>>>>> Here
> >>>>>>>> it is:
> >>>>>>>>>>>
> >>>>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
> >>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >>>>>> does
> >>>>>>>>>>> compile
> >>>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >>>>>> 8.
> >>>>>>>>>>>>
> >>>>>>>>>>>> It depends on who you ask, I guess :)
> >>>>>>>>>>>>
> >>>>>>>>>>>> From Druid's perspective, we aren't going to be compiling your
> >>>>>>> code,
> >>>>>>>>>>> only
> >>>>>>>>>>>> running it. So the only requirement we would ask for is that
> >>>>> the
> >>>>>>>> code
> >>>>>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
> >>>>> Java
> >>>>>>> 11
> >>>>>>>> is
> >>>>>>>>>>> good
> >>>>>>>>>>>> for future-proofing your code, though. It is starting to
> >>>>> become
> >>>>>>> more
> >>>>>>>>>>>> prevalent.
> >>>>>>>>>>>>
> >>>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
> >>>>> all
> >>>>>>> the
> >>>>>>>>>>> direct
> >>>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >>>>>> what
> >>>>>>> is
> >>>>>>>>>>>> required.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is this correct?
> >>>>>>>>>>>>
> >>>>>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
> >>>>>>> that's
> >>>>>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
> >>>>>> run
> >>>>>>>> our
> >>>>>>>>>>> tests
> >>>>>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
> >>>>>> testing
> >>>>>>>>>>> easier,
> >>>>>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
> >>>>> even
> >>>>>>>> though
> >>>>>>>>>>> we
> >>>>>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
> >>>>>> done
> >>>>>>>> this
> >>>>>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
> >>>>>> Travis
> >>>>>>>>>>> doesn't
> >>>>>>>>>>>> easily support.)
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks, Gian, this is VERY helpful!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
> >>>>>>> Unsafe
> >>>>>>>> and
> >>>>>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
> >>>>>> this
> >>>>>>>> would
> >>>>>>>>>>>>> work
> >>>>>>>>>>>>> with 9+ ), and then defines some method-handles that the
> >>>>> other
> >>>>>>>> classes
> >>>>>>>>>>>>> use.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Will this approach allow code to compile under JDK11 and
> >>>>> JDK8?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It also seems that compiling with JDK11 is still a work in
> >>>>>>>> progress as
> >>>>>>>>>>> the
> >>>>>>>>>>>>> pom.xml clearly specifies JDK8.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >>>>> does
> >>>>>>>> compile
> >>>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >>>>> 8.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
> >>>>>> the
> >>>>>>>>>>> direct
> >>>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >>>>> what
> >>>>>>> is
> >>>>>>>>>>>>> required.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Is this correct?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
> >>>>> gian@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> And nobody should be using the Oracle JDK without a
> >>>>>> commercial
> >>>>>>>>>>>>> relationship
> >>>>>>>>>>>>>> with Oracle, its license is a nightmare.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
> >>>>>> gian@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
> >>>>>> now.
> >>>>>>>> The
> >>>>>>>>>>>>>>> differences are in terms of licensing, length of support,
> >>>>>> and
> >>>>>>>>>>> release
> >>>>>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
> >>>>>>>> third-party
> >>>>>>>>>>>>>> OpenJDK
> >>>>>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
> >>>>>> than
> >>>>>>>>>>> vanilla
> >>>>>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
> >>>>> that
> >>>>>>>> public
> >>>>>>>>>>>>> updates
> >>>>>>>>>>>>>>> will cease for major versions after 6 months.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
> >>>>> leerho@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks, Gian,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Your references are helpful and I am studying them.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
> >>>>>> other
> >>>>>>>> hidden
> >>>>>>>>>>>>>>>> classes)
> >>>>>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
> >>>>> far
> >>>>>> as
> >>>>>>>>>>> Druid is
> >>>>>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
> >>>>>>> OpenJDK
> >>>>>>>>>>> always
> >>>>>>>>>>>>>>>> aligned?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
> >>>>>>> gian@apache.org
> >>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> You didn't address this message to me, but in Druid
> >>>>> most
> >>>>>>> of
> >>>>>>>> the
> >>>>>>>>>>>>> work
> >>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
> >>>>> issue,
> >>>>>>>> which
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> might
> >>>>>>>>>>>>>>>>> find helpful:
> >>>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
> >>>>>>>>>>>>>>>>> Skimming the list, these might be particularly
> >>>>> relevant
> >>>>>> to
> >>>>>>>>>>>>>> DataSketches:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
> >>>>>>>> (cleaner
> >>>>>>>>>>>>>>>> operations)
> >>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
> >>>>>>>>>>> (ByteBuffer
> >>>>>>>>>>>>>> unmap
> >>>>>>>>>>>>>>>>> operation)
> >>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
> >>>>>>>> (Remove
> >>>>>>>>>>>>> direct
> >>>>>>>>>>>>>>>>> references to Unsafe)
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
> >>>>>> the
> >>>>>>>> list
> >>>>>>>>>>> at
> >>>>>>>>>>>>>>>> large.
> >>>>>>>>>>>>>>>>> It encourages more people to participate. If Roman is
> >>>>> a
> >>>>>>>>>>> subscriber
> >>>>>>>>>>>>> he
> >>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
> >>>>>>>> subscribe.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
> >>>>>> leerho@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Roman,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I hope you are doing well.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I really appreciate the contributions you made to
> >>>>> our
> >>>>>>>> Memory
> >>>>>>>>>>>>>>>> component!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Given that Druid has made its code base compatible
> >>>>>> with
> >>>>>>>>>>> OpenJDK
> >>>>>>>>>>>>> 11,
> >>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>>> could use your help in what changes do we need to
> >>>>> make
> >>>>>>> to
> >>>>>>>> make
> >>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>> happen
> >>>>>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
> >>>>>> library
> >>>>>>>> was
> >>>>>>>>>>> not
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> only
> >>>>>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
> >>>>> I
> >>>>>>>> assume
> >>>>>>>>>>> you
> >>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>>>>> down this path :)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Currently, I have a check in the static initializer
> >>>>> in
> >>>>>>>>>>>>>>>> Memory/UnsafeUtil:
> >>>>>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
> >>>>> returned
> >>>>>>> from
> >>>>>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
> >>>>>> does
> >>>>>>>> not
> >>>>>>>>>>>>> contain
> >>>>>>>>>>>>>>>>> "1.8"
> >>>>>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
> >>>>> not
> >>>>>>>> tested
> >>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>> JDK 9,
> >>>>>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
> >>>>> throw
> >>>>>>>> rather
> >>>>>>>>>>> than
> >>>>>>>>>>>>>>>> having
> >>>>>>>>>>>>>>>>>> the users experience some weird failure later that
> >>>>> may
> >>>>>>> be
> >>>>>>>>>>>>> difficult
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> debug.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> With only one minor exception (which we could easily
> >>>>>>> fix)
> >>>>>>>> the
> >>>>>>>>>>>>> Memory
> >>>>>>>>>>>>>>>>>> component is the only place where Unsafe is used.
> >>>>>>>> However,
> >>>>>>>>>>> we do
> >>>>>>>>>>>>>> use
> >>>>>>>>>>>>>>>>>> reflection to gain internal access to a number of
> >>>>>> other
> >>>>>>>>>>> classes
> >>>>>>>>>>>>>>>> including
> >>>>>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Looking forward to hearing from you.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> >>>>>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>> --
> >>>> From my cell phone.
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> >>> For additional commands, e-mail: dev-help@datasketches.apache.org
> >>>
> >>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> For additional commands, e-mail: dev-help@datasketches.apache.org
>
>
Re: Help with Migration to OpenJDK 11
Posted by Dave Fisher <wa...@apache.org>.
Hi Lee,
> On Oct 18, 2019, at 5:01 PM, leerho <le...@gmail.com> wrote:
>
> Meanwhile, back to the topic of Migration to OpenJDK11 ...
>
> Unfortunately, the Apache email lists do not accommodate images or files.
While true by default, Infra will enable images/files if the podling requests.
Regards,
Dave
> So I am going to move this next discussion to our asf slack channel
> "datasketches-dev".
>
> I have posted my study there. You can offer comments or suggestions
> either here or there.
>
> Cheers,
>
> Lee.
>
> On Fri, Oct 18, 2019 at 3:53 PM leerho <le...@gmail.com> wrote:
>
>> I opened a ticket (DATASKETCHES-3) with infra to see if we could add this
>> capability to our slack channel.
>>
>> On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
>>
>>> Hi -
>>>
>>> Apache Pulsar uses this: https://github.com/merlimat/slack-email-digest
>>>
>>> They don’t have their slack channels at the-asf.slack.com but it does
>>> work. You can always ask for infra’s help.
>>>
>>> Regards,
>>> Dave
>>>
>>>> On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
>>>>
>>>> I wonder if there is a way to have slack auto-post to an email list?
>>>>
>>>> On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
>>>>
>>>>> I don't have the expertise on jdk 11 but would love know more about
>>>>> datasketches and specifically the java lib. Would be great if you can
>>> bring
>>>>> back the result of what you have discussed in the slack dev channel.
>>> This
>>>>> helps other contributors easily to search and get to know what
>>> happened in
>>>>> the community and get involved :)
>>>>>
>>>>> Best
>>>>> Evans
>>>>>
>>>>> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
>>>>>
>>>>>> Christopher,
>>>>>>
>>>>>> This would be fabulous. I have tons of questions, whenever you are
>>>>> ready.
>>>>>> I might be easier to interact via the ASF slack channel
>>> "datasketches" or
>>>>>> "datasketches-dev".
>>>>>>
>>>>>> Lee.
>>>>>>
>>>>>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
>>> wrote:
>>>>>>
>>>>>>> The memory repository is the one that needs the most conversion help.
>>>>>>> Everything using Unsafe in datasketches-java runs through the memory
>>>>>>> package.
>>>>>>>
>>>>>>> I'm not sure off-hand if the characterization repo has low level
>>> tests
>>>>> of
>>>>>>> Memory (on my phone at the moment), but we'd be looking to minimize
>>> the
>>>>>>> speed impact in the process since it very directly impacts sketch
>>>>> speed.
>>>>>>>
>>>>>>> jon
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
>>> wrote:
>>>>>>>
>>>>>>>> I've converted several other projects to using JDK 11, and can help
>>>>>>>> with this if there's still a need for help. I've actually been
>>>>> looking
>>>>>>>> at ways I can contribute to DataSketches since ApacheCon, and this
>>>>>>>> seems to be something I can probably do.
>>>>>>>>
>>>>>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Gian,
>>>>>>>>>
>>>>>>>>> If I were to supply you with a link to a snapshot or a Release
>>>>>>> Candidate
>>>>>>>>> version of the datasketches jar (and Memory jar) could you test in
>>>>>> your
>>>>>>>>> environment under JDK 11 and see if it works?
>>>>>>>>>
>>>>>>>>> Lee
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Holy Cow! I have never seen such a complex travis config!
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> I should have linked Druid's Travis config file for reference.
>>>>>> Here
>>>>>>>> it is:
>>>>>>>>>>>
>>>>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>>>>>> does
>>>>>>>>>>> compile
>>>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>>>>>> 8.
>>>>>>>>>>>>
>>>>>>>>>>>> It depends on who you ask, I guess :)
>>>>>>>>>>>>
>>>>>>>>>>>> From Druid's perspective, we aren't going to be compiling your
>>>>>>> code,
>>>>>>>>>>> only
>>>>>>>>>>>> running it. So the only requirement we would ask for is that
>>>>> the
>>>>>>>> code
>>>>>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
>>>>> Java
>>>>>>> 11
>>>>>>>> is
>>>>>>>>>>> good
>>>>>>>>>>>> for future-proofing your code, though. It is starting to
>>>>> become
>>>>>>> more
>>>>>>>>>>>> prevalent.
>>>>>>>>>>>>
>>>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
>>>>> all
>>>>>>> the
>>>>>>>>>>> direct
>>>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>>>>>> what
>>>>>>> is
>>>>>>>>>>>> required.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this correct?
>>>>>>>>>>>>
>>>>>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
>>>>>>> that's
>>>>>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
>>>>>> run
>>>>>>>> our
>>>>>>>>>>> tests
>>>>>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
>>>>>> testing
>>>>>>>>>>> easier,
>>>>>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
>>>>> even
>>>>>>>> though
>>>>>>>>>>> we
>>>>>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
>>>>>> done
>>>>>>>> this
>>>>>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
>>>>>> Travis
>>>>>>>>>>> doesn't
>>>>>>>>>>>> easily support.)
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, Gian, this is VERY helpful!
>>>>>>>>>>>>>
>>>>>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
>>>>>>> Unsafe
>>>>>>>> and
>>>>>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
>>>>>> this
>>>>>>>> would
>>>>>>>>>>>>> work
>>>>>>>>>>>>> with 9+ ), and then defines some method-handles that the
>>>>> other
>>>>>>>> classes
>>>>>>>>>>>>> use.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Will this approach allow code to compile under JDK11 and
>>>>> JDK8?
>>>>>>>>>>>>>
>>>>>>>>>>>>> It also seems that compiling with JDK11 is still a work in
>>>>>>>> progress as
>>>>>>>>>>> the
>>>>>>>>>>>>> pom.xml clearly specifies JDK8.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>>>>> does
>>>>>>>> compile
>>>>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>>>>> 8.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
>>>>>> the
>>>>>>>>>>> direct
>>>>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>>>>> what
>>>>>>> is
>>>>>>>>>>>>> required.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is this correct?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Lee.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
>>>>> gian@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> And nobody should be using the Oracle JDK without a
>>>>>> commercial
>>>>>>>>>>>>> relationship
>>>>>>>>>>>>>> with Oracle, its license is a nightmare.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
>>>>>> gian@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
>>>>>> now.
>>>>>>>> The
>>>>>>>>>>>>>>> differences are in terms of licensing, length of support,
>>>>>> and
>>>>>>>>>>> release
>>>>>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
>>>>>>>> third-party
>>>>>>>>>>>>>> OpenJDK
>>>>>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
>>>>>> than
>>>>>>>>>>> vanilla
>>>>>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
>>>>> that
>>>>>>>> public
>>>>>>>>>>>>> updates
>>>>>>>>>>>>>>> will cease for major versions after 6 months.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
>>>>> leerho@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks, Gian,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Your references are helpful and I am studying them.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
>>>>>> other
>>>>>>>> hidden
>>>>>>>>>>>>>>>> classes)
>>>>>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
>>>>> far
>>>>>> as
>>>>>>>>>>> Druid is
>>>>>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
>>>>>>> OpenJDK
>>>>>>>>>>> always
>>>>>>>>>>>>>>>> aligned?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Lee.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
>>>>>>> gian@apache.org
>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You didn't address this message to me, but in Druid
>>>>> most
>>>>>>> of
>>>>>>>> the
>>>>>>>>>>>>> work
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
>>>>> issue,
>>>>>>>> which
>>>>>>>>>>> you
>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>> find helpful:
>>>>>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
>>>>>>>>>>>>>>>>> Skimming the list, these might be particularly
>>>>> relevant
>>>>>> to
>>>>>>>>>>>>>> DataSketches:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
>>>>>>>> (cleaner
>>>>>>>>>>>>>>>> operations)
>>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
>>>>>>>>>>> (ByteBuffer
>>>>>>>>>>>>>> unmap
>>>>>>>>>>>>>>>>> operation)
>>>>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
>>>>>>>> (Remove
>>>>>>>>>>>>> direct
>>>>>>>>>>>>>>>>> references to Unsafe)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
>>>>>> the
>>>>>>>> list
>>>>>>>>>>> at
>>>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>>>>> It encourages more people to participate. If Roman is
>>>>> a
>>>>>>>>>>> subscriber
>>>>>>>>>>>>> he
>>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
>>>>>>>> subscribe.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
>>>>>> leerho@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Roman,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I hope you are doing well.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I really appreciate the contributions you made to
>>>>> our
>>>>>>>> Memory
>>>>>>>>>>>>>>>> component!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Given that Druid has made its code base compatible
>>>>>> with
>>>>>>>>>>> OpenJDK
>>>>>>>>>>>>> 11,
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> could use your help in what changes do we need to
>>>>> make
>>>>>>> to
>>>>>>>> make
>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
>>>>>> library
>>>>>>>> was
>>>>>>>>>>> not
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
>>>>> I
>>>>>>>> assume
>>>>>>>>>>> you
>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>> down this path :)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Currently, I have a check in the static initializer
>>>>> in
>>>>>>>>>>>>>>>> Memory/UnsafeUtil:
>>>>>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
>>>>> returned
>>>>>>> from
>>>>>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
>>>>>> does
>>>>>>>> not
>>>>>>>>>>>>> contain
>>>>>>>>>>>>>>>>> "1.8"
>>>>>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
>>>>> not
>>>>>>>> tested
>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> JDK 9,
>>>>>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
>>>>> throw
>>>>>>>> rather
>>>>>>>>>>> than
>>>>>>>>>>>>>>>> having
>>>>>>>>>>>>>>>>>> the users experience some weird failure later that
>>>>> may
>>>>>>> be
>>>>>>>>>>>>> difficult
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> debug.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> With only one minor exception (which we could easily
>>>>>>> fix)
>>>>>>>> the
>>>>>>>>>>>>> Memory
>>>>>>>>>>>>>>>>>> component is the only place where Unsafe is used.
>>>>>>>> However,
>>>>>>>>>>> we do
>>>>>>>>>>>>>> use
>>>>>>>>>>>>>>>>>> reflection to gain internal access to a number of
>>>>>> other
>>>>>>>>>>> classes
>>>>>>>>>>>>>>>> including
>>>>>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Looking forward to hearing from you.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Lee.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>> --
>>>> From my cell phone.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>>> For additional commands, e-mail: dev-help@datasketches.apache.org
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Meanwhile, back to the topic of Migration to OpenJDK11 ...
Unfortunately, the Apache email lists do not accommodate images or files.
So I am going to move this next discussion to our asf slack channel
"datasketches-dev".
I have posted my study there. You can offer comments or suggestions
either here or there.
Cheers,
Lee.
On Fri, Oct 18, 2019 at 3:53 PM leerho <le...@gmail.com> wrote:
> I opened a ticket (DATASKETCHES-3) with infra to see if we could add this
> capability to our slack channel.
>
> On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
>
>> Hi -
>>
>> Apache Pulsar uses this: https://github.com/merlimat/slack-email-digest
>>
>> They don’t have their slack channels at the-asf.slack.com but it does
>> work. You can always ask for infra’s help.
>>
>> Regards,
>> Dave
>>
>> > On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
>> >
>> > I wonder if there is a way to have slack auto-post to an email list?
>> >
>> > On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
>> >
>> >> I don't have the expertise on jdk 11 but would love know more about
>> >> datasketches and specifically the java lib. Would be great if you can
>> bring
>> >> back the result of what you have discussed in the slack dev channel.
>> This
>> >> helps other contributors easily to search and get to know what
>> happened in
>> >> the community and get involved :)
>> >>
>> >> Best
>> >> Evans
>> >>
>> >> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
>> >>
>> >>> Christopher,
>> >>>
>> >>> This would be fabulous. I have tons of questions, whenever you are
>> >> ready.
>> >>> I might be easier to interact via the ASF slack channel
>> "datasketches" or
>> >>> "datasketches-dev".
>> >>>
>> >>> Lee.
>> >>>
>> >>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
>> wrote:
>> >>>
>> >>>> The memory repository is the one that needs the most conversion help.
>> >>>> Everything using Unsafe in datasketches-java runs through the memory
>> >>>> package.
>> >>>>
>> >>>> I'm not sure off-hand if the characterization repo has low level
>> tests
>> >> of
>> >>>> Memory (on my phone at the moment), but we'd be looking to minimize
>> the
>> >>>> speed impact in the process since it very directly impacts sketch
>> >> speed.
>> >>>>
>> >>>> jon
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
>> wrote:
>> >>>>
>> >>>>> I've converted several other projects to using JDK 11, and can help
>> >>>>> with this if there's still a need for help. I've actually been
>> >> looking
>> >>>>> at ways I can contribute to DataSketches since ApacheCon, and this
>> >>>>> seems to be something I can probably do.
>> >>>>>
>> >>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
>> >>>>>>
>> >>>>>> Gian,
>> >>>>>>
>> >>>>>> If I were to supply you with a link to a snapshot or a Release
>> >>>> Candidate
>> >>>>>> version of the datasketches jar (and Memory jar) could you test in
>> >>> your
>> >>>>>> environment under JDK 11 and see if it works?
>> >>>>>>
>> >>>>>> Lee
>> >>>>>>
>> >>>>>>
>> >>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
>> >>>>>>
>> >>>>>>> Holy Cow! I have never seen such a complex travis config!
>> >>>>>>>
>> >>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
>> >>>> wrote:
>> >>>>>>>
>> >>>>>>>> I should have linked Druid's Travis config file for reference.
>> >>> Here
>> >>>>> it is:
>> >>>>>>>>
>> >> https://github.com/apache/incubator-druid/blob/master/.travis.yml
>> >>>>>>>>
>> >>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
>> >>>> wrote:
>> >>>>>>>>
>> >>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>> >>> does
>> >>>>>>>> compile
>> >>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>> >>> 8.
>> >>>>>>>>>
>> >>>>>>>>> It depends on who you ask, I guess :)
>> >>>>>>>>>
>> >>>>>>>>> From Druid's perspective, we aren't going to be compiling your
>> >>>> code,
>> >>>>>>>> only
>> >>>>>>>>> running it. So the only requirement we would ask for is that
>> >> the
>> >>>>> code
>> >>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
>> >> Java
>> >>>> 11
>> >>>>> is
>> >>>>>>>> good
>> >>>>>>>>> for future-proofing your code, though. It is starting to
>> >> become
>> >>>> more
>> >>>>>>>>> prevalent.
>> >>>>>>>>>
>> >>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
>> >> all
>> >>>> the
>> >>>>>>>> direct
>> >>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>> >>> what
>> >>>> is
>> >>>>>>>>> required.
>> >>>>>>>>>>
>> >>>>>>>>>> Is this correct?
>> >>>>>>>>>
>> >>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
>> >>>> that's
>> >>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
>> >>> run
>> >>>>> our
>> >>>>>>>> tests
>> >>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
>> >>> testing
>> >>>>>>>> easier,
>> >>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
>> >> even
>> >>>>> though
>> >>>>>>>> we
>> >>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
>> >>> done
>> >>>>> this
>> >>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
>> >>> Travis
>> >>>>>>>> doesn't
>> >>>>>>>>> easily support.)
>> >>>>>>>>>
>> >>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
>> >>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Thanks, Gian, this is VERY helpful!
>> >>>>>>>>>>
>> >>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
>> >>>> Unsafe
>> >>>>> and
>> >>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
>> >>> this
>> >>>>> would
>> >>>>>>>>>> work
>> >>>>>>>>>> with 9+ ), and then defines some method-handles that the
>> >> other
>> >>>>> classes
>> >>>>>>>>>> use.
>> >>>>>>>>>>
>> >>>>>>>>>> Will this approach allow code to compile under JDK11 and
>> >> JDK8?
>> >>>>>>>>>>
>> >>>>>>>>>> It also seems that compiling with JDK11 is still a work in
>> >>>>> progress as
>> >>>>>>>> the
>> >>>>>>>>>> pom.xml clearly specifies JDK8.
>> >>>>>>>>>>
>> >>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>> >> does
>> >>>>> compile
>> >>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>> >> 8.
>> >>>>>>>>>>
>> >>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
>> >>> the
>> >>>>>>>> direct
>> >>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>> >> what
>> >>>> is
>> >>>>>>>>>> required.
>> >>>>>>>>>>
>> >>>>>>>>>> Is this correct?
>> >>>>>>>>>>
>> >>>>>>>>>> Lee.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
>> >> gian@apache.org>
>> >>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> And nobody should be using the Oracle JDK without a
>> >>> commercial
>> >>>>>>>>>> relationship
>> >>>>>>>>>>> with Oracle, its license is a nightmare.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
>> >>> gian@apache.org>
>> >>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
>> >>> now.
>> >>>>> The
>> >>>>>>>>>>>> differences are in terms of licensing, length of support,
>> >>> and
>> >>>>>>>> release
>> >>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
>> >>>>> third-party
>> >>>>>>>>>>> OpenJDK
>> >>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
>> >>> than
>> >>>>>>>> vanilla
>> >>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
>> >> that
>> >>>>> public
>> >>>>>>>>>> updates
>> >>>>>>>>>>>> will cease for major versions after 6 months.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
>> >> leerho@gmail.com>
>> >>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> Thanks, Gian,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Your references are helpful and I am studying them.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
>> >>> other
>> >>>>> hidden
>> >>>>>>>>>>>>> classes)
>> >>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
>> >> far
>> >>> as
>> >>>>>>>> Druid is
>> >>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
>> >>>> OpenJDK
>> >>>>>>>> always
>> >>>>>>>>>>>>> aligned?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Lee.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
>> >>>> gian@apache.org
>> >>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>> You didn't address this message to me, but in Druid
>> >> most
>> >>>> of
>> >>>>> the
>> >>>>>>>>>> work
>> >>>>>>>>>>> for
>> >>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
>> >> issue,
>> >>>>> which
>> >>>>>>>> you
>> >>>>>>>>>>> might
>> >>>>>>>>>>>>>> find helpful:
>> >>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
>> >>>>>>>>>>>>>> Skimming the list, these might be particularly
>> >> relevant
>> >>> to
>> >>>>>>>>>>> DataSketches:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
>> >>>>> (cleaner
>> >>>>>>>>>>>>> operations)
>> >>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
>> >>>>>>>> (ByteBuffer
>> >>>>>>>>>>> unmap
>> >>>>>>>>>>>>>> operation)
>> >>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
>> >>>>> (Remove
>> >>>>>>>>>> direct
>> >>>>>>>>>>>>>> references to Unsafe)
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
>> >>> the
>> >>>>> list
>> >>>>>>>> at
>> >>>>>>>>>>>>> large.
>> >>>>>>>>>>>>>> It encourages more people to participate. If Roman is
>> >> a
>> >>>>>>>> subscriber
>> >>>>>>>>>> he
>> >>>>>>>>>>>>> would
>> >>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
>> >>>>> subscribe.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
>> >>> leerho@gmail.com>
>> >>>>>>>> wrote:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Roman,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I hope you are doing well.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> I really appreciate the contributions you made to
>> >> our
>> >>>>> Memory
>> >>>>>>>>>>>>> component!
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Given that Druid has made its code base compatible
>> >>> with
>> >>>>>>>> OpenJDK
>> >>>>>>>>>> 11,
>> >>>>>>>>>>> we
>> >>>>>>>>>>>>>>> could use your help in what changes do we need to
>> >> make
>> >>>> to
>> >>>>> make
>> >>>>>>>>>> that
>> >>>>>>>>>>>>>> happen
>> >>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
>> >>> library
>> >>>>> was
>> >>>>>>>> not
>> >>>>>>>>>> the
>> >>>>>>>>>>>>> only
>> >>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
>> >> I
>> >>>>> assume
>> >>>>>>>> you
>> >>>>>>>>>>> have
>> >>>>>>>>>>>>>> been
>> >>>>>>>>>>>>>>> down this path :)
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Currently, I have a check in the static initializer
>> >> in
>> >>>>>>>>>>>>> Memory/UnsafeUtil:
>> >>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
>> >> returned
>> >>>> from
>> >>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
>> >>> does
>> >>>>> not
>> >>>>>>>>>> contain
>> >>>>>>>>>>>>>> "1.8"
>> >>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
>> >> not
>> >>>>> tested
>> >>>>>>>>>> with
>> >>>>>>>>>>>>> JDK 9,
>> >>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
>> >> throw
>> >>>>> rather
>> >>>>>>>> than
>> >>>>>>>>>>>>> having
>> >>>>>>>>>>>>>>> the users experience some weird failure later that
>> >> may
>> >>>> be
>> >>>>>>>>>> difficult
>> >>>>>>>>>>> to
>> >>>>>>>>>>>>>>> debug.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> With only one minor exception (which we could easily
>> >>>> fix)
>> >>>>> the
>> >>>>>>>>>> Memory
>> >>>>>>>>>>>>>>> component is the only place where Unsafe is used.
>> >>>>> However,
>> >>>>>>>> we do
>> >>>>>>>>>>> use
>> >>>>>>>>>>>>>>> reflection to gain internal access to a number of
>> >>> other
>> >>>>>>>> classes
>> >>>>>>>>>>>>> including
>> >>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Looking forward to hearing from you.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Cheers,
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>> Lee.
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>>
>> ---------------------------------------------------------------------
>> >>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>> >>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> > --
>> > From my cell phone.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>> For additional commands, e-mail: dev-help@datasketches.apache.org
>>
>>
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
I opened a ticket (DATASKETCHES-3) with infra to see if we could add this
capability to our slack channel.
On Fri, Oct 18, 2019 at 12:29 PM Dave Fisher <wa...@apache.org> wrote:
> Hi -
>
> Apache Pulsar uses this: https://github.com/merlimat/slack-email-digest
>
> They don’t have their slack channels at the-asf.slack.com but it does
> work. You can always ask for infra’s help.
>
> Regards,
> Dave
>
> > On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
> >
> > I wonder if there is a way to have slack auto-post to an email list?
> >
> > On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
> >
> >> I don't have the expertise on jdk 11 but would love know more about
> >> datasketches and specifically the java lib. Would be great if you can
> bring
> >> back the result of what you have discussed in the slack dev channel.
> This
> >> helps other contributors easily to search and get to know what happened
> in
> >> the community and get involved :)
> >>
> >> Best
> >> Evans
> >>
> >> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
> >>
> >>> Christopher,
> >>>
> >>> This would be fabulous. I have tons of questions, whenever you are
> >> ready.
> >>> I might be easier to interact via the ASF slack channel "datasketches"
> or
> >>> "datasketches-dev".
> >>>
> >>> Lee.
> >>>
> >>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com>
> wrote:
> >>>
> >>>> The memory repository is the one that needs the most conversion help.
> >>>> Everything using Unsafe in datasketches-java runs through the memory
> >>>> package.
> >>>>
> >>>> I'm not sure off-hand if the characterization repo has low level tests
> >> of
> >>>> Memory (on my phone at the moment), but we'd be looking to minimize
> the
> >>>> speed impact in the process since it very directly impacts sketch
> >> speed.
> >>>>
> >>>> jon
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org>
> wrote:
> >>>>
> >>>>> I've converted several other projects to using JDK 11, and can help
> >>>>> with this if there's still a need for help. I've actually been
> >> looking
> >>>>> at ways I can contribute to DataSketches since ApacheCon, and this
> >>>>> seems to be something I can probably do.
> >>>>>
> >>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> >>>>>>
> >>>>>> Gian,
> >>>>>>
> >>>>>> If I were to supply you with a link to a snapshot or a Release
> >>>> Candidate
> >>>>>> version of the datasketches jar (and Memory jar) could you test in
> >>> your
> >>>>>> environment under JDK 11 and see if it works?
> >>>>>>
> >>>>>> Lee
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> >>>>>>
> >>>>>>> Holy Cow! I have never seen such a complex travis config!
> >>>>>>>
> >>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> I should have linked Druid's Travis config file for reference.
> >>> Here
> >>>>> it is:
> >>>>>>>>
> >> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> >>>>>>>>
> >>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >>> does
> >>>>>>>> compile
> >>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >>> 8.
> >>>>>>>>>
> >>>>>>>>> It depends on who you ask, I guess :)
> >>>>>>>>>
> >>>>>>>>> From Druid's perspective, we aren't going to be compiling your
> >>>> code,
> >>>>>>>> only
> >>>>>>>>> running it. So the only requirement we would ask for is that
> >> the
> >>>>> code
> >>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
> >> Java
> >>>> 11
> >>>>> is
> >>>>>>>> good
> >>>>>>>>> for future-proofing your code, though. It is starting to
> >> become
> >>>> more
> >>>>>>>>> prevalent.
> >>>>>>>>>
> >>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
> >> all
> >>>> the
> >>>>>>>> direct
> >>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >>> what
> >>>> is
> >>>>>>>>> required.
> >>>>>>>>>>
> >>>>>>>>>> Is this correct?
> >>>>>>>>>
> >>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
> >>>> that's
> >>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
> >>> run
> >>>>> our
> >>>>>>>> tests
> >>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
> >>> testing
> >>>>>>>> easier,
> >>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
> >> even
> >>>>> though
> >>>>>>>> we
> >>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
> >>> done
> >>>>> this
> >>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
> >>> Travis
> >>>>>>>> doesn't
> >>>>>>>>> easily support.)
> >>>>>>>>>
> >>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
> >>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Thanks, Gian, this is VERY helpful!
> >>>>>>>>>>
> >>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
> >>>> Unsafe
> >>>>> and
> >>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
> >>> this
> >>>>> would
> >>>>>>>>>> work
> >>>>>>>>>> with 9+ ), and then defines some method-handles that the
> >> other
> >>>>> classes
> >>>>>>>>>> use.
> >>>>>>>>>>
> >>>>>>>>>> Will this approach allow code to compile under JDK11 and
> >> JDK8?
> >>>>>>>>>>
> >>>>>>>>>> It also seems that compiling with JDK11 is still a work in
> >>>>> progress as
> >>>>>>>> the
> >>>>>>>>>> pom.xml clearly specifies JDK8.
> >>>>>>>>>>
> >>>>>>>>>> So what exactly is the requirement? Our DataSketches code
> >> does
> >>>>> compile
> >>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
> >> 8.
> >>>>>>>>>>
> >>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
> >>> the
> >>>>>>>> direct
> >>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
> >> what
> >>>> is
> >>>>>>>>>> required.
> >>>>>>>>>>
> >>>>>>>>>> Is this correct?
> >>>>>>>>>>
> >>>>>>>>>> Lee.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
> >> gian@apache.org>
> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> And nobody should be using the Oracle JDK without a
> >>> commercial
> >>>>>>>>>> relationship
> >>>>>>>>>>> with Oracle, its license is a nightmare.
> >>>>>>>>>>>
> >>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
> >>> gian@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
> >>> now.
> >>>>> The
> >>>>>>>>>>>> differences are in terms of licensing, length of support,
> >>> and
> >>>>>>>> release
> >>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
> >>>>> third-party
> >>>>>>>>>>> OpenJDK
> >>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
> >>> than
> >>>>>>>> vanilla
> >>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
> >> that
> >>>>> public
> >>>>>>>>>> updates
> >>>>>>>>>>>> will cease for major versions after 6 months.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
> >> leerho@gmail.com>
> >>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks, Gian,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Your references are helpful and I am studying them.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
> >>> other
> >>>>> hidden
> >>>>>>>>>>>>> classes)
> >>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
> >> far
> >>> as
> >>>>>>>> Druid is
> >>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
> >>>> OpenJDK
> >>>>>>>> always
> >>>>>>>>>>>>> aligned?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
> >>>> gian@apache.org
> >>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> You didn't address this message to me, but in Druid
> >> most
> >>>> of
> >>>>> the
> >>>>>>>>>> work
> >>>>>>>>>>> for
> >>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
> >> issue,
> >>>>> which
> >>>>>>>> you
> >>>>>>>>>>> might
> >>>>>>>>>>>>>> find helpful:
> >>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
> >>>>>>>>>>>>>> Skimming the list, these might be particularly
> >> relevant
> >>> to
> >>>>>>>>>>> DataSketches:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
> >>>>> (cleaner
> >>>>>>>>>>>>> operations)
> >>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
> >>>>>>>> (ByteBuffer
> >>>>>>>>>>> unmap
> >>>>>>>>>>>>>> operation)
> >>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
> >>>>> (Remove
> >>>>>>>>>> direct
> >>>>>>>>>>>>>> references to Unsafe)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
> >>> the
> >>>>> list
> >>>>>>>> at
> >>>>>>>>>>>>> large.
> >>>>>>>>>>>>>> It encourages more people to participate. If Roman is
> >> a
> >>>>>>>> subscriber
> >>>>>>>>>> he
> >>>>>>>>>>>>> would
> >>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
> >>>>> subscribe.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
> >>> leerho@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Roman,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I hope you are doing well.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I really appreciate the contributions you made to
> >> our
> >>>>> Memory
> >>>>>>>>>>>>> component!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Given that Druid has made its code base compatible
> >>> with
> >>>>>>>> OpenJDK
> >>>>>>>>>> 11,
> >>>>>>>>>>> we
> >>>>>>>>>>>>>>> could use your help in what changes do we need to
> >> make
> >>>> to
> >>>>> make
> >>>>>>>>>> that
> >>>>>>>>>>>>>> happen
> >>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
> >>> library
> >>>>> was
> >>>>>>>> not
> >>>>>>>>>> the
> >>>>>>>>>>>>> only
> >>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
> >> I
> >>>>> assume
> >>>>>>>> you
> >>>>>>>>>>> have
> >>>>>>>>>>>>>> been
> >>>>>>>>>>>>>>> down this path :)
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Currently, I have a check in the static initializer
> >> in
> >>>>>>>>>>>>> Memory/UnsafeUtil:
> >>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
> >> returned
> >>>> from
> >>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
> >>> does
> >>>>> not
> >>>>>>>>>> contain
> >>>>>>>>>>>>>> "1.8"
> >>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
> >> not
> >>>>> tested
> >>>>>>>>>> with
> >>>>>>>>>>>>> JDK 9,
> >>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
> >> throw
> >>>>> rather
> >>>>>>>> than
> >>>>>>>>>>>>> having
> >>>>>>>>>>>>>>> the users experience some weird failure later that
> >> may
> >>>> be
> >>>>>>>>>> difficult
> >>>>>>>>>>> to
> >>>>>>>>>>>>>>> debug.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> With only one minor exception (which we could easily
> >>>> fix)
> >>>>> the
> >>>>>>>>>> Memory
> >>>>>>>>>>>>>>> component is the only place where Unsafe is used.
> >>>>> However,
> >>>>>>>> we do
> >>>>>>>>>>> use
> >>>>>>>>>>>>>>> reflection to gain internal access to a number of
> >>> other
> >>>>>>>> classes
> >>>>>>>>>>>>> including
> >>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Looking forward to hearing from you.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Lee.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> >>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> > --
> > From my cell phone.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> For additional commands, e-mail: dev-help@datasketches.apache.org
>
>
Re: Help with Migration to OpenJDK 11
Posted by Dave Fisher <wa...@apache.org>.
Hi -
Apache Pulsar uses this: https://github.com/merlimat/slack-email-digest
They don’t have their slack channels at the-asf.slack.com but it does work. You can always ask for infra’s help.
Regards,
Dave
> On Oct 18, 2019, at 12:26 PM, leerho <le...@gmail.com> wrote:
>
> I wonder if there is a way to have slack auto-post to an email list?
>
> On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
>
>> I don't have the expertise on jdk 11 but would love know more about
>> datasketches and specifically the java lib. Would be great if you can bring
>> back the result of what you have discussed in the slack dev channel. This
>> helps other contributors easily to search and get to know what happened in
>> the community and get involved :)
>>
>> Best
>> Evans
>>
>> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
>>
>>> Christopher,
>>>
>>> This would be fabulous. I have tons of questions, whenever you are
>> ready.
>>> I might be easier to interact via the ASF slack channel "datasketches" or
>>> "datasketches-dev".
>>>
>>> Lee.
>>>
>>> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com> wrote:
>>>
>>>> The memory repository is the one that needs the most conversion help.
>>>> Everything using Unsafe in datasketches-java runs through the memory
>>>> package.
>>>>
>>>> I'm not sure off-hand if the characterization repo has low level tests
>> of
>>>> Memory (on my phone at the moment), but we'd be looking to minimize the
>>>> speed impact in the process since it very directly impacts sketch
>> speed.
>>>>
>>>> jon
>>>>
>>>>
>>>>
>>>> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org> wrote:
>>>>
>>>>> I've converted several other projects to using JDK 11, and can help
>>>>> with this if there's still a need for help. I've actually been
>> looking
>>>>> at ways I can contribute to DataSketches since ApacheCon, and this
>>>>> seems to be something I can probably do.
>>>>>
>>>>> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
>>>>>>
>>>>>> Gian,
>>>>>>
>>>>>> If I were to supply you with a link to a snapshot or a Release
>>>> Candidate
>>>>>> version of the datasketches jar (and Memory jar) could you test in
>>> your
>>>>>> environment under JDK 11 and see if it works?
>>>>>>
>>>>>> Lee
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
>>>>>>
>>>>>>> Holy Cow! I have never seen such a complex travis config!
>>>>>>>
>>>>>>> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
>>>> wrote:
>>>>>>>
>>>>>>>> I should have linked Druid's Travis config file for reference.
>>> Here
>>>>> it is:
>>>>>>>>
>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
>>>>>>>>
>>>>>>>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
>>>> wrote:
>>>>>>>>
>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>>> does
>>>>>>>> compile
>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>>> 8.
>>>>>>>>>
>>>>>>>>> It depends on who you ask, I guess :)
>>>>>>>>>
>>>>>>>>> From Druid's perspective, we aren't going to be compiling your
>>>> code,
>>>>>>>> only
>>>>>>>>> running it. So the only requirement we would ask for is that
>> the
>>>>> code
>>>>>>>>> _runs_ under a Java 11 runtime. Being able to compile under
>> Java
>>>> 11
>>>>> is
>>>>>>>> good
>>>>>>>>> for future-proofing your code, though. It is starting to
>> become
>>>> more
>>>>>>>>> prevalent.
>>>>>>>>>
>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting
>> all
>>>> the
>>>>>>>> direct
>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>>> what
>>>> is
>>>>>>>>> required.
>>>>>>>>>>
>>>>>>>>>> Is this correct?
>>>>>>>>>
>>>>>>>>> That is certainly part of it, I'm not 100% sure if that's all
>>>> that's
>>>>>>>>> needed. Btw, CI can help verify. In Druid's Travis config, we
>>> run
>>>>> our
>>>>>>>> tests
>>>>>>>>> on both JDK 8 and JDK 11. In order to make this automated
>>> testing
>>>>>>>> easier,
>>>>>>>>> we chose to do the work needed to compile Druid on JDK 11,
>> even
>>>>> though
>>>>>>>> we
>>>>>>>>> still compile release builds using JDK 8 only. (If we hadn't
>>> done
>>>>> this
>>>>>>>>> work, then we'd have to compile on 8 and test on 11, which
>>> Travis
>>>>>>>> doesn't
>>>>>>>>> easily support.)
>>>>>>>>>
>>>>>>>>> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks, Gian, this is VERY helpful!
>>>>>>>>>>
>>>>>>>>>> From my cursory glance, it appears that UnsafeUtil acquires
>>>> Unsafe
>>>>> and
>>>>>>>>>> Unsafe-class the usual way using reflection (I didn't think
>>> this
>>>>> would
>>>>>>>>>> work
>>>>>>>>>> with 9+ ), and then defines some method-handles that the
>> other
>>>>> classes
>>>>>>>>>> use.
>>>>>>>>>>
>>>>>>>>>> Will this approach allow code to compile under JDK11 and
>> JDK8?
>>>>>>>>>>
>>>>>>>>>> It also seems that compiling with JDK11 is still a work in
>>>>> progress as
>>>>>>>> the
>>>>>>>>>> pom.xml clearly specifies JDK8.
>>>>>>>>>>
>>>>>>>>>> So what exactly is the requirement? Our DataSketches code
>> does
>>>>> compile
>>>>>>>>>> under JDK8 and we have lots of customers and users using JDK
>> 8.
>>>>>>>>>>
>>>>>>>>>> It sounds like (as far as Unsafe is concerned) converting all
>>> the
>>>>>>>> direct
>>>>>>>>>> unsafe.blah() calls to static MethodHandle.invoke calls is
>> what
>>>> is
>>>>>>>>>> required.
>>>>>>>>>>
>>>>>>>>>> Is this correct?
>>>>>>>>>>
>>>>>>>>>> Lee.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
>> gian@apache.org>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> And nobody should be using the Oracle JDK without a
>>> commercial
>>>>>>>>>> relationship
>>>>>>>>>>> with Oracle, its license is a nightmare.
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
>>> gian@apache.org>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Oracle and OpenJDK builds behave the same as each other
>>> now.
>>>>> The
>>>>>>>>>>>> differences are in terms of licensing, length of support,
>>> and
>>>>>>>> release
>>>>>>>>>>>> cadence. IMO most Java devs should be using one of the
>>>>> third-party
>>>>>>>>>>> OpenJDK
>>>>>>>>>>>> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
>>> than
>>>>>>>> vanilla
>>>>>>>>>>>> OpenJDK these days, because of the new OpenJDK policy
>> that
>>>>> public
>>>>>>>>>> updates
>>>>>>>>>>>> will cease for major versions after 6 months.
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Oct 14, 2019 at 11:07 AM leerho <
>> leerho@gmail.com>
>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks, Gian,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your references are helpful and I am studying them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I didn't see any references to OpenJDK vs Oracle's JDKs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are there any differences in the way Unsafe calls (or
>>> other
>>>>> hidden
>>>>>>>>>>>>> classes)
>>>>>>>>>>>>> are handled by OpenJDK? Or is this a "don't care" as
>> far
>>> as
>>>>>>>> Druid is
>>>>>>>>>>>>> concerned? Are the version numbers between Oracle and
>>>> OpenJDK
>>>>>>>> always
>>>>>>>>>>>>> aligned?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Lee.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
>>>> gian@apache.org
>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> You didn't address this message to me, but in Druid
>> most
>>>> of
>>>>> the
>>>>>>>>>> work
>>>>>>>>>>> for
>>>>>>>>>>>>>> Java 9+ compatibility is mentioned in this master
>> issue,
>>>>> which
>>>>>>>> you
>>>>>>>>>>> might
>>>>>>>>>>>>>> find helpful:
>>>>>>>>>> https://github.com/apache/incubator-druid/issues/5589.
>>>>>>>>>>>>>> Skimming the list, these might be particularly
>> relevant
>>> to
>>>>>>>>>>> DataSketches:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7487
>>>>> (cleaner
>>>>>>>>>>>>> operations)
>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7466
>>>>>>>> (ByteBuffer
>>>>>>>>>>> unmap
>>>>>>>>>>>>>> operation)
>>>>>>>>>>>>>> - https://github.com/apache/incubator-druid/pull/7576
>>>>> (Remove
>>>>>>>>>> direct
>>>>>>>>>>>>>> references to Unsafe)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Btw, it would be better, I think, to address emails to
>>> the
>>>>> list
>>>>>>>> at
>>>>>>>>>>>>> large.
>>>>>>>>>>>>>> It encourages more people to participate. If Roman is
>> a
>>>>>>>> subscriber
>>>>>>>>>> he
>>>>>>>>>>>>> would
>>>>>>>>>>>>>> get a copy anyway. If not, you could encourage him to
>>>>> subscribe.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Oct 11, 2019 at 3:13 PM leerho <
>>> leerho@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Roman,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I hope you are doing well.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I really appreciate the contributions you made to
>> our
>>>>> Memory
>>>>>>>>>>>>> component!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Given that Druid has made its code base compatible
>>> with
>>>>>>>> OpenJDK
>>>>>>>>>> 11,
>>>>>>>>>>> we
>>>>>>>>>>>>>>> could use your help in what changes do we need to
>> make
>>>> to
>>>>> make
>>>>>>>>>> that
>>>>>>>>>>>>>> happen
>>>>>>>>>>>>>>> for DataSketches. As I recall, our DataSketches
>>> library
>>>>> was
>>>>>>>> not
>>>>>>>>>> the
>>>>>>>>>>>>> only
>>>>>>>>>>>>>>> Druid dependency that took advantage of Unsafe. So
>> I
>>>>> assume
>>>>>>>> you
>>>>>>>>>>> have
>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>> down this path :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Currently, I have a check in the static initializer
>> in
>>>>>>>>>>>>> Memory/UnsafeUtil:
>>>>>>>>>>>>>>> parseJavaVersion(...) that checks the string
>> returned
>>>> from
>>>>>>>>>>>>>>> System.getProperty("java.version"). If that string
>>> does
>>>>> not
>>>>>>>>>> contain
>>>>>>>>>>>>>> "1.8"
>>>>>>>>>>>>>>> or "8" it will throw an error. Given that we have
>> not
>>>>> tested
>>>>>>>>>> with
>>>>>>>>>>>>> JDK 9,
>>>>>>>>>>>>>>> 10, 11 or 12, I felt it was safer to explicitly
>> throw
>>>>> rather
>>>>>>>> than
>>>>>>>>>>>>> having
>>>>>>>>>>>>>>> the users experience some weird failure later that
>> may
>>>> be
>>>>>>>>>> difficult
>>>>>>>>>>> to
>>>>>>>>>>>>>>> debug.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> With only one minor exception (which we could easily
>>>> fix)
>>>>> the
>>>>>>>>>> Memory
>>>>>>>>>>>>>>> component is the only place where Unsafe is used.
>>>>> However,
>>>>>>>> we do
>>>>>>>>>>> use
>>>>>>>>>>>>>>> reflection to gain internal access to a number of
>>> other
>>>>>>>> classes
>>>>>>>>>>>>> including
>>>>>>>>>>>>>>> ByteBuffer, Cleaner, and Bits.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Looking forward to hearing from you.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Lee.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
>>>>> For additional commands, e-mail: dev-help@datasketches.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
> --
> From my cell phone.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
I wonder if there is a way to have slack auto-post to an email list?
On Fri, Oct 18, 2019 at 5:13 AM Evans Ye <ev...@apache.org> wrote:
> I don't have the expertise on jdk 11 but would love know more about
> datasketches and specifically the java lib. Would be great if you can bring
> back the result of what you have discussed in the slack dev channel. This
> helps other contributors easily to search and get to know what happened in
> the community and get involved :)
>
> Best
> Evans
>
> leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
>
> > Christopher,
> >
> > This would be fabulous. I have tons of questions, whenever you are
> ready.
> > I might be easier to interact via the ASF slack channel "datasketches" or
> > "datasketches-dev".
> >
> > Lee.
> >
> > On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com> wrote:
> >
> > > The memory repository is the one that needs the most conversion help.
> > > Everything using Unsafe in datasketches-java runs through the memory
> > > package.
> > >
> > > I'm not sure off-hand if the characterization repo has low level tests
> of
> > > Memory (on my phone at the moment), but we'd be looking to minimize the
> > > speed impact in the process since it very directly impacts sketch
> speed.
> > >
> > > jon
> > >
> > >
> > >
> > > On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org> wrote:
> > >
> > > > I've converted several other projects to using JDK 11, and can help
> > > > with this if there's still a need for help. I've actually been
> looking
> > > > at ways I can contribute to DataSketches since ApacheCon, and this
> > > > seems to be something I can probably do.
> > > >
> > > > On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> > > > >
> > > > > Gian,
> > > > >
> > > > > If I were to supply you with a link to a snapshot or a Release
> > > Candidate
> > > > > version of the datasketches jar (and Memory jar) could you test in
> > your
> > > > > environment under JDK 11 and see if it works?
> > > > >
> > > > > Lee
> > > > >
> > > > >
> > > > > On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> > > > >
> > > > > > Holy Cow! I have never seen such a complex travis config!
> > > > > >
> > > > > > On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >
> > > > > >> I should have linked Druid's Travis config file for reference.
> > Here
> > > > it is:
> > > > > >>
> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> > > > > >>
> > > > > >> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >>
> > > > > >> > > So what exactly is the requirement? Our DataSketches code
> > does
> > > > > >> compile
> > > > > >> > > under JDK8 and we have lots of customers and users using JDK
> > 8.
> > > > > >> >
> > > > > >> > It depends on who you ask, I guess :)
> > > > > >> >
> > > > > >> > From Druid's perspective, we aren't going to be compiling your
> > > code,
> > > > > >> only
> > > > > >> > running it. So the only requirement we would ask for is that
> the
> > > > code
> > > > > >> > _runs_ under a Java 11 runtime. Being able to compile under
> Java
> > > 11
> > > > is
> > > > > >> good
> > > > > >> > for future-proofing your code, though. It is starting to
> become
> > > more
> > > > > >> > prevalent.
> > > > > >> >
> > > > > >> > > It sounds like (as far as Unsafe is concerned) converting
> all
> > > the
> > > > > >> direct
> > > > > >> > > unsafe.blah() calls to static MethodHandle.invoke calls is
> > what
> > > is
> > > > > >> > required.
> > > > > >> > >
> > > > > >> > > Is this correct?
> > > > > >> >
> > > > > >> > That is certainly part of it, I'm not 100% sure if that's all
> > > that's
> > > > > >> > needed. Btw, CI can help verify. In Druid's Travis config, we
> > run
> > > > our
> > > > > >> tests
> > > > > >> > on both JDK 8 and JDK 11. In order to make this automated
> > testing
> > > > > >> easier,
> > > > > >> > we chose to do the work needed to compile Druid on JDK 11,
> even
> > > > though
> > > > > >> we
> > > > > >> > still compile release builds using JDK 8 only. (If we hadn't
> > done
> > > > this
> > > > > >> > work, then we'd have to compile on 8 and test on 11, which
> > Travis
> > > > > >> doesn't
> > > > > >> > easily support.)
> > > > > >> >
> > > > > >> > On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
> > wrote:
> > > > > >> >
> > > > > >> >> Thanks, Gian, this is VERY helpful!
> > > > > >> >>
> > > > > >> >> From my cursory glance, it appears that UnsafeUtil acquires
> > > Unsafe
> > > > and
> > > > > >> >> Unsafe-class the usual way using reflection (I didn't think
> > this
> > > > would
> > > > > >> >> work
> > > > > >> >> with 9+ ), and then defines some method-handles that the
> other
> > > > classes
> > > > > >> >> use.
> > > > > >> >>
> > > > > >> >> Will this approach allow code to compile under JDK11 and
> JDK8?
> > > > > >> >>
> > > > > >> >> It also seems that compiling with JDK11 is still a work in
> > > > progress as
> > > > > >> the
> > > > > >> >> pom.xml clearly specifies JDK8.
> > > > > >> >>
> > > > > >> >> So what exactly is the requirement? Our DataSketches code
> does
> > > > compile
> > > > > >> >> under JDK8 and we have lots of customers and users using JDK
> 8.
> > > > > >> >>
> > > > > >> >> It sounds like (as far as Unsafe is concerned) converting all
> > the
> > > > > >> direct
> > > > > >> >> unsafe.blah() calls to static MethodHandle.invoke calls is
> what
> > > is
> > > > > >> >> required.
> > > > > >> >>
> > > > > >> >> Is this correct?
> > > > > >> >>
> > > > > >> >> Lee.
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <
> gian@apache.org>
> > > > wrote:
> > > > > >> >>
> > > > > >> >> > And nobody should be using the Oracle JDK without a
> > commercial
> > > > > >> >> relationship
> > > > > >> >> > with Oracle, its license is a nightmare.
> > > > > >> >> >
> > > > > >> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
> > gian@apache.org>
> > > > > >> wrote:
> > > > > >> >> >
> > > > > >> >> > > Oracle and OpenJDK builds behave the same as each other
> > now.
> > > > The
> > > > > >> >> > > differences are in terms of licensing, length of support,
> > and
> > > > > >> release
> > > > > >> >> > > cadence. IMO most Java devs should be using one of the
> > > > third-party
> > > > > >> >> > OpenJDK
> > > > > >> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
> > than
> > > > > >> vanilla
> > > > > >> >> > > OpenJDK these days, because of the new OpenJDK policy
> that
> > > > public
> > > > > >> >> updates
> > > > > >> >> > > will cease for major versions after 6 months.
> > > > > >> >> > >
> > > > > >> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <
> leerho@gmail.com>
> > > > wrote:
> > > > > >> >> > >
> > > > > >> >> > >> Thanks, Gian,
> > > > > >> >> > >>
> > > > > >> >> > >> Your references are helpful and I am studying them.
> > > > > >> >> > >>
> > > > > >> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> > > > > >> >> > >>
> > > > > >> >> > >> Are there any differences in the way Unsafe calls (or
> > other
> > > > hidden
> > > > > >> >> > >> classes)
> > > > > >> >> > >> are handled by OpenJDK? Or is this a "don't care" as
> far
> > as
> > > > > >> Druid is
> > > > > >> >> > >> concerned? Are the version numbers between Oracle and
> > > OpenJDK
> > > > > >> always
> > > > > >> >> > >> aligned?
> > > > > >> >> > >>
> > > > > >> >> > >>
> > > > > >> >> > >> Lee.
> > > > > >> >> > >>
> > > > > >> >> > >>
> > > > > >> >> > >>
> > > > > >> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
> > > gian@apache.org
> > > > >
> > > > > >> >> wrote:
> > > > > >> >> > >>
> > > > > >> >> > >> > You didn't address this message to me, but in Druid
> most
> > > of
> > > > the
> > > > > >> >> work
> > > > > >> >> > for
> > > > > >> >> > >> > Java 9+ compatibility is mentioned in this master
> issue,
> > > > which
> > > > > >> you
> > > > > >> >> > might
> > > > > >> >> > >> > find helpful:
> > > > > >> >> https://github.com/apache/incubator-druid/issues/5589.
> > > > > >> >> > >> > Skimming the list, these might be particularly
> relevant
> > to
> > > > > >> >> > DataSketches:
> > > > > >> >> > >> >
> > > > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7487
> > > > (cleaner
> > > > > >> >> > >> operations)
> > > > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7466
> > > > > >> (ByteBuffer
> > > > > >> >> > unmap
> > > > > >> >> > >> > operation)
> > > > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7576
> > > > (Remove
> > > > > >> >> direct
> > > > > >> >> > >> > references to Unsafe)
> > > > > >> >> > >> >
> > > > > >> >> > >> > Btw, it would be better, I think, to address emails to
> > the
> > > > list
> > > > > >> at
> > > > > >> >> > >> large.
> > > > > >> >> > >> > It encourages more people to participate. If Roman is
> a
> > > > > >> subscriber
> > > > > >> >> he
> > > > > >> >> > >> would
> > > > > >> >> > >> > get a copy anyway. If not, you could encourage him to
> > > > subscribe.
> > > > > >> >> > >> >
> > > > > >> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <
> > leerho@gmail.com>
> > > > > >> wrote:
> > > > > >> >> > >> >
> > > > > >> >> > >> > > Roman,
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > I hope you are doing well.
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > I really appreciate the contributions you made to
> our
> > > > Memory
> > > > > >> >> > >> component!
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > Given that Druid has made its code base compatible
> > with
> > > > > >> OpenJDK
> > > > > >> >> 11,
> > > > > >> >> > we
> > > > > >> >> > >> > > could use your help in what changes do we need to
> make
> > > to
> > > > make
> > > > > >> >> that
> > > > > >> >> > >> > happen
> > > > > >> >> > >> > > for DataSketches. As I recall, our DataSketches
> > library
> > > > was
> > > > > >> not
> > > > > >> >> the
> > > > > >> >> > >> only
> > > > > >> >> > >> > > Druid dependency that took advantage of Unsafe. So
> I
> > > > assume
> > > > > >> you
> > > > > >> >> > have
> > > > > >> >> > >> > been
> > > > > >> >> > >> > > down this path :)
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > Currently, I have a check in the static initializer
> in
> > > > > >> >> > >> Memory/UnsafeUtil:
> > > > > >> >> > >> > > parseJavaVersion(...) that checks the string
> returned
> > > from
> > > > > >> >> > >> > > System.getProperty("java.version"). If that string
> > does
> > > > not
> > > > > >> >> contain
> > > > > >> >> > >> > "1.8"
> > > > > >> >> > >> > > or "8" it will throw an error. Given that we have
> not
> > > > tested
> > > > > >> >> with
> > > > > >> >> > >> JDK 9,
> > > > > >> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly
> throw
> > > > rather
> > > > > >> than
> > > > > >> >> > >> having
> > > > > >> >> > >> > > the users experience some weird failure later that
> may
> > > be
> > > > > >> >> difficult
> > > > > >> >> > to
> > > > > >> >> > >> > > debug.
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > With only one minor exception (which we could easily
> > > fix)
> > > > the
> > > > > >> >> Memory
> > > > > >> >> > >> > > component is the only place where Unsafe is used.
> > > > However,
> > > > > >> we do
> > > > > >> >> > use
> > > > > >> >> > >> > > reflection to gain internal access to a number of
> > other
> > > > > >> classes
> > > > > >> >> > >> including
> > > > > >> >> > >> > > ByteBuffer, Cleaner, and Bits.
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > Looking forward to hearing from you.
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > Cheers,
> > > > > >> >> > >> > >
> > > > > >> >> > >> > > Lee.
> > > > > >> >> > >> > >
> > > > > >> >> > >> >
> > > > > >> >> > >>
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >> >
> > > > > >>
> > > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> > > > For additional commands, e-mail: dev-help@datasketches.apache.org
> > > >
> > > >
> > >
> >
>
--
From my cell phone.
Re: Help with Migration to OpenJDK 11
Posted by Evans Ye <ev...@apache.org>.
I don't have the expertise on jdk 11 but would love know more about
datasketches and specifically the java lib. Would be great if you can bring
back the result of what you have discussed in the slack dev channel. This
helps other contributors easily to search and get to know what happened in
the community and get involved :)
Best
Evans
leerho <le...@gmail.com> 於 2019年10月18日 週五 11:03 寫道：
> Christopher,
>
> This would be fabulous. I have tons of questions, whenever you are ready.
> I might be easier to interact via the ASF slack channel "datasketches" or
> "datasketches-dev".
>
> Lee.
>
> On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com> wrote:
>
> > The memory repository is the one that needs the most conversion help.
> > Everything using Unsafe in datasketches-java runs through the memory
> > package.
> >
> > I'm not sure off-hand if the characterization repo has low level tests of
> > Memory (on my phone at the moment), but we'd be looking to minimize the
> > speed impact in the process since it very directly impacts sketch speed.
> >
> > jon
> >
> >
> >
> > On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org> wrote:
> >
> > > I've converted several other projects to using JDK 11, and can help
> > > with this if there's still a need for help. I've actually been looking
> > > at ways I can contribute to DataSketches since ApacheCon, and this
> > > seems to be something I can probably do.
> > >
> > > On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> > > >
> > > > Gian,
> > > >
> > > > If I were to supply you with a link to a snapshot or a Release
> > Candidate
> > > > version of the datasketches jar (and Memory jar) could you test in
> your
> > > > environment under JDK 11 and see if it works?
> > > >
> > > > Lee
> > > >
> > > >
> > > > On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> > > >
> > > > > Holy Cow! I have never seen such a complex travis config!
> > > > >
> > > > > On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >
> > > > >> I should have linked Druid's Travis config file for reference.
> Here
> > > it is:
> > > > >> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> > > > >>
> > > > >> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >>
> > > > >> > > So what exactly is the requirement? Our DataSketches code
> does
> > > > >> compile
> > > > >> > > under JDK8 and we have lots of customers and users using JDK
> 8.
> > > > >> >
> > > > >> > It depends on who you ask, I guess :)
> > > > >> >
> > > > >> > From Druid's perspective, we aren't going to be compiling your
> > code,
> > > > >> only
> > > > >> > running it. So the only requirement we would ask for is that the
> > > code
> > > > >> > _runs_ under a Java 11 runtime. Being able to compile under Java
> > 11
> > > is
> > > > >> good
> > > > >> > for future-proofing your code, though. It is starting to become
> > more
> > > > >> > prevalent.
> > > > >> >
> > > > >> > > It sounds like (as far as Unsafe is concerned) converting all
> > the
> > > > >> direct
> > > > >> > > unsafe.blah() calls to static MethodHandle.invoke calls is
> what
> > is
> > > > >> > required.
> > > > >> > >
> > > > >> > > Is this correct?
> > > > >> >
> > > > >> > That is certainly part of it, I'm not 100% sure if that's all
> > that's
> > > > >> > needed. Btw, CI can help verify. In Druid's Travis config, we
> run
> > > our
> > > > >> tests
> > > > >> > on both JDK 8 and JDK 11. In order to make this automated
> testing
> > > > >> easier,
> > > > >> > we chose to do the work needed to compile Druid on JDK 11, even
> > > though
> > > > >> we
> > > > >> > still compile release builds using JDK 8 only. (If we hadn't
> done
> > > this
> > > > >> > work, then we'd have to compile on 8 and test on 11, which
> Travis
> > > > >> doesn't
> > > > >> > easily support.)
> > > > >> >
> > > > >> > On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com>
> wrote:
> > > > >> >
> > > > >> >> Thanks, Gian, this is VERY helpful!
> > > > >> >>
> > > > >> >> From my cursory glance, it appears that UnsafeUtil acquires
> > Unsafe
> > > and
> > > > >> >> Unsafe-class the usual way using reflection (I didn't think
> this
> > > would
> > > > >> >> work
> > > > >> >> with 9+ ), and then defines some method-handles that the other
> > > classes
> > > > >> >> use.
> > > > >> >>
> > > > >> >> Will this approach allow code to compile under JDK11 and JDK8?
> > > > >> >>
> > > > >> >> It also seems that compiling with JDK11 is still a work in
> > > progress as
> > > > >> the
> > > > >> >> pom.xml clearly specifies JDK8.
> > > > >> >>
> > > > >> >> So what exactly is the requirement? Our DataSketches code does
> > > compile
> > > > >> >> under JDK8 and we have lots of customers and users using JDK 8.
> > > > >> >>
> > > > >> >> It sounds like (as far as Unsafe is concerned) converting all
> the
> > > > >> direct
> > > > >> >> unsafe.blah() calls to static MethodHandle.invoke calls is what
> > is
> > > > >> >> required.
> > > > >> >>
> > > > >> >> Is this correct?
> > > > >> >>
> > > > >> >> Lee.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > >> >>
> > > > >> >> > And nobody should be using the Oracle JDK without a
> commercial
> > > > >> >> relationship
> > > > >> >> > with Oracle, its license is a nightmare.
> > > > >> >> >
> > > > >> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <
> gian@apache.org>
> > > > >> wrote:
> > > > >> >> >
> > > > >> >> > > Oracle and OpenJDK builds behave the same as each other
> now.
> > > The
> > > > >> >> > > differences are in terms of licensing, length of support,
> and
> > > > >> release
> > > > >> >> > > cadence. IMO most Java devs should be using one of the
> > > third-party
> > > > >> >> > OpenJDK
> > > > >> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather
> than
> > > > >> vanilla
> > > > >> >> > > OpenJDK these days, because of the new OpenJDK policy that
> > > public
> > > > >> >> updates
> > > > >> >> > > will cease for major versions after 6 months.
> > > > >> >> > >
> > > > >> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com>
> > > wrote:
> > > > >> >> > >
> > > > >> >> > >> Thanks, Gian,
> > > > >> >> > >>
> > > > >> >> > >> Your references are helpful and I am studying them.
> > > > >> >> > >>
> > > > >> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> > > > >> >> > >>
> > > > >> >> > >> Are there any differences in the way Unsafe calls (or
> other
> > > hidden
> > > > >> >> > >> classes)
> > > > >> >> > >> are handled by OpenJDK? Or is this a "don't care" as far
> as
> > > > >> Druid is
> > > > >> >> > >> concerned? Are the version numbers between Oracle and
> > OpenJDK
> > > > >> always
> > > > >> >> > >> aligned?
> > > > >> >> > >>
> > > > >> >> > >>
> > > > >> >> > >> Lee.
> > > > >> >> > >>
> > > > >> >> > >>
> > > > >> >> > >>
> > > > >> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
> > gian@apache.org
> > > >
> > > > >> >> wrote:
> > > > >> >> > >>
> > > > >> >> > >> > You didn't address this message to me, but in Druid most
> > of
> > > the
> > > > >> >> work
> > > > >> >> > for
> > > > >> >> > >> > Java 9+ compatibility is mentioned in this master issue,
> > > which
> > > > >> you
> > > > >> >> > might
> > > > >> >> > >> > find helpful:
> > > > >> >> https://github.com/apache/incubator-druid/issues/5589.
> > > > >> >> > >> > Skimming the list, these might be particularly relevant
> to
> > > > >> >> > DataSketches:
> > > > >> >> > >> >
> > > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7487
> > > (cleaner
> > > > >> >> > >> operations)
> > > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7466
> > > > >> (ByteBuffer
> > > > >> >> > unmap
> > > > >> >> > >> > operation)
> > > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7576
> > > (Remove
> > > > >> >> direct
> > > > >> >> > >> > references to Unsafe)
> > > > >> >> > >> >
> > > > >> >> > >> > Btw, it would be better, I think, to address emails to
> the
> > > list
> > > > >> at
> > > > >> >> > >> large.
> > > > >> >> > >> > It encourages more people to participate. If Roman is a
> > > > >> subscriber
> > > > >> >> he
> > > > >> >> > >> would
> > > > >> >> > >> > get a copy anyway. If not, you could encourage him to
> > > subscribe.
> > > > >> >> > >> >
> > > > >> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <
> leerho@gmail.com>
> > > > >> wrote:
> > > > >> >> > >> >
> > > > >> >> > >> > > Roman,
> > > > >> >> > >> > >
> > > > >> >> > >> > > I hope you are doing well.
> > > > >> >> > >> > >
> > > > >> >> > >> > > I really appreciate the contributions you made to our
> > > Memory
> > > > >> >> > >> component!
> > > > >> >> > >> > >
> > > > >> >> > >> > > Given that Druid has made its code base compatible
> with
> > > > >> OpenJDK
> > > > >> >> 11,
> > > > >> >> > we
> > > > >> >> > >> > > could use your help in what changes do we need to make
> > to
> > > make
> > > > >> >> that
> > > > >> >> > >> > happen
> > > > >> >> > >> > > for DataSketches. As I recall, our DataSketches
> library
> > > was
> > > > >> not
> > > > >> >> the
> > > > >> >> > >> only
> > > > >> >> > >> > > Druid dependency that took advantage of Unsafe. So I
> > > assume
> > > > >> you
> > > > >> >> > have
> > > > >> >> > >> > been
> > > > >> >> > >> > > down this path :)
> > > > >> >> > >> > >
> > > > >> >> > >> > > Currently, I have a check in the static initializer in
> > > > >> >> > >> Memory/UnsafeUtil:
> > > > >> >> > >> > > parseJavaVersion(...) that checks the string returned
> > from
> > > > >> >> > >> > > System.getProperty("java.version"). If that string
> does
> > > not
> > > > >> >> contain
> > > > >> >> > >> > "1.8"
> > > > >> >> > >> > > or "8" it will throw an error. Given that we have not
> > > tested
> > > > >> >> with
> > > > >> >> > >> JDK 9,
> > > > >> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw
> > > rather
> > > > >> than
> > > > >> >> > >> having
> > > > >> >> > >> > > the users experience some weird failure later that may
> > be
> > > > >> >> difficult
> > > > >> >> > to
> > > > >> >> > >> > > debug.
> > > > >> >> > >> > >
> > > > >> >> > >> > > With only one minor exception (which we could easily
> > fix)
> > > the
> > > > >> >> Memory
> > > > >> >> > >> > > component is the only place where Unsafe is used.
> > > However,
> > > > >> we do
> > > > >> >> > use
> > > > >> >> > >> > > reflection to gain internal access to a number of
> other
> > > > >> classes
> > > > >> >> > >> including
> > > > >> >> > >> > > ByteBuffer, Cleaner, and Bits.
> > > > >> >> > >> > >
> > > > >> >> > >> > > Looking forward to hearing from you.
> > > > >> >> > >> > >
> > > > >> >> > >> > > Cheers,
> > > > >> >> > >> > >
> > > > >> >> > >> > > Lee.
> > > > >> >> > >> > >
> > > > >> >> > >> >
> > > > >> >> > >>
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> > > For additional commands, e-mail: dev-help@datasketches.apache.org
> > >
> > >
> >
>
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Christopher,
This would be fabulous. I have tons of questions, whenever you are ready.
I might be easier to interact via the ASF slack channel "datasketches" or
"datasketches-dev".
Lee.
On Thu, Oct 17, 2019 at 5:07 PM Jon Malkin <jo...@gmail.com> wrote:
> The memory repository is the one that needs the most conversion help.
> Everything using Unsafe in datasketches-java runs through the memory
> package.
>
> I'm not sure off-hand if the characterization repo has low level tests of
> Memory (on my phone at the moment), but we'd be looking to minimize the
> speed impact in the process since it very directly impacts sketch speed.
>
> jon
>
>
>
> On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org> wrote:
>
> > I've converted several other projects to using JDK 11, and can help
> > with this if there's still a need for help. I've actually been looking
> > at ways I can contribute to DataSketches since ApacheCon, and this
> > seems to be something I can probably do.
> >
> > On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> > >
> > > Gian,
> > >
> > > If I were to supply you with a link to a snapshot or a Release
> Candidate
> > > version of the datasketches jar (and Memory jar) could you test in your
> > > environment under JDK 11 and see if it works?
> > >
> > > Lee
> > >
> > >
> > > On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> > >
> > > > Holy Cow! I have never seen such a complex travis config!
> > > >
> > > > On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > >> I should have linked Druid's Travis config file for reference. Here
> > it is:
> > > >> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> > > >>
> > > >> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >>
> > > >> > > So what exactly is the requirement? Our DataSketches code does
> > > >> compile
> > > >> > > under JDK8 and we have lots of customers and users using JDK 8.
> > > >> >
> > > >> > It depends on who you ask, I guess :)
> > > >> >
> > > >> > From Druid's perspective, we aren't going to be compiling your
> code,
> > > >> only
> > > >> > running it. So the only requirement we would ask for is that the
> > code
> > > >> > _runs_ under a Java 11 runtime. Being able to compile under Java
> 11
> > is
> > > >> good
> > > >> > for future-proofing your code, though. It is starting to become
> more
> > > >> > prevalent.
> > > >> >
> > > >> > > It sounds like (as far as Unsafe is concerned) converting all
> the
> > > >> direct
> > > >> > > unsafe.blah() calls to static MethodHandle.invoke calls is what
> is
> > > >> > required.
> > > >> > >
> > > >> > > Is this correct?
> > > >> >
> > > >> > That is certainly part of it, I'm not 100% sure if that's all
> that's
> > > >> > needed. Btw, CI can help verify. In Druid's Travis config, we run
> > our
> > > >> tests
> > > >> > on both JDK 8 and JDK 11. In order to make this automated testing
> > > >> easier,
> > > >> > we chose to do the work needed to compile Druid on JDK 11, even
> > though
> > > >> we
> > > >> > still compile release builds using JDK 8 only. (If we hadn't done
> > this
> > > >> > work, then we'd have to compile on 8 and test on 11, which Travis
> > > >> doesn't
> > > >> > easily support.)
> > > >> >
> > > >> > On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com> wrote:
> > > >> >
> > > >> >> Thanks, Gian, this is VERY helpful!
> > > >> >>
> > > >> >> From my cursory glance, it appears that UnsafeUtil acquires
> Unsafe
> > and
> > > >> >> Unsafe-class the usual way using reflection (I didn't think this
> > would
> > > >> >> work
> > > >> >> with 9+ ), and then defines some method-handles that the other
> > classes
> > > >> >> use.
> > > >> >>
> > > >> >> Will this approach allow code to compile under JDK11 and JDK8?
> > > >> >>
> > > >> >> It also seems that compiling with JDK11 is still a work in
> > progress as
> > > >> the
> > > >> >> pom.xml clearly specifies JDK8.
> > > >> >>
> > > >> >> So what exactly is the requirement? Our DataSketches code does
> > compile
> > > >> >> under JDK8 and we have lots of customers and users using JDK 8.
> > > >> >>
> > > >> >> It sounds like (as far as Unsafe is concerned) converting all the
> > > >> direct
> > > >> >> unsafe.blah() calls to static MethodHandle.invoke calls is what
> is
> > > >> >> required.
> > > >> >>
> > > >> >> Is this correct?
> > > >> >>
> > > >> >> Lee.
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > >> >>
> > > >> >> > And nobody should be using the Oracle JDK without a commercial
> > > >> >> relationship
> > > >> >> > with Oracle, its license is a nightmare.
> > > >> >> >
> > > >> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org>
> > > >> wrote:
> > > >> >> >
> > > >> >> > > Oracle and OpenJDK builds behave the same as each other now.
> > The
> > > >> >> > > differences are in terms of licensing, length of support, and
> > > >> release
> > > >> >> > > cadence. IMO most Java devs should be using one of the
> > third-party
> > > >> >> > OpenJDK
> > > >> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than
> > > >> vanilla
> > > >> >> > > OpenJDK these days, because of the new OpenJDK policy that
> > public
> > > >> >> updates
> > > >> >> > > will cease for major versions after 6 months.
> > > >> >> > >
> > > >> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com>
> > wrote:
> > > >> >> > >
> > > >> >> > >> Thanks, Gian,
> > > >> >> > >>
> > > >> >> > >> Your references are helpful and I am studying them.
> > > >> >> > >>
> > > >> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> > > >> >> > >>
> > > >> >> > >> Are there any differences in the way Unsafe calls (or other
> > hidden
> > > >> >> > >> classes)
> > > >> >> > >> are handled by OpenJDK? Or is this a "don't care" as far as
> > > >> Druid is
> > > >> >> > >> concerned? Are the version numbers between Oracle and
> OpenJDK
> > > >> always
> > > >> >> > >> aligned?
> > > >> >> > >>
> > > >> >> > >>
> > > >> >> > >> Lee.
> > > >> >> > >>
> > > >> >> > >>
> > > >> >> > >>
> > > >> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <
> gian@apache.org
> > >
> > > >> >> wrote:
> > > >> >> > >>
> > > >> >> > >> > You didn't address this message to me, but in Druid most
> of
> > the
> > > >> >> work
> > > >> >> > for
> > > >> >> > >> > Java 9+ compatibility is mentioned in this master issue,
> > which
> > > >> you
> > > >> >> > might
> > > >> >> > >> > find helpful:
> > > >> >> https://github.com/apache/incubator-druid/issues/5589.
> > > >> >> > >> > Skimming the list, these might be particularly relevant to
> > > >> >> > DataSketches:
> > > >> >> > >> >
> > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7487
> > (cleaner
> > > >> >> > >> operations)
> > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7466
> > > >> (ByteBuffer
> > > >> >> > unmap
> > > >> >> > >> > operation)
> > > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7576
> > (Remove
> > > >> >> direct
> > > >> >> > >> > references to Unsafe)
> > > >> >> > >> >
> > > >> >> > >> > Btw, it would be better, I think, to address emails to the
> > list
> > > >> at
> > > >> >> > >> large.
> > > >> >> > >> > It encourages more people to participate. If Roman is a
> > > >> subscriber
> > > >> >> he
> > > >> >> > >> would
> > > >> >> > >> > get a copy anyway. If not, you could encourage him to
> > subscribe.
> > > >> >> > >> >
> > > >> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com>
> > > >> wrote:
> > > >> >> > >> >
> > > >> >> > >> > > Roman,
> > > >> >> > >> > >
> > > >> >> > >> > > I hope you are doing well.
> > > >> >> > >> > >
> > > >> >> > >> > > I really appreciate the contributions you made to our
> > Memory
> > > >> >> > >> component!
> > > >> >> > >> > >
> > > >> >> > >> > > Given that Druid has made its code base compatible with
> > > >> OpenJDK
> > > >> >> 11,
> > > >> >> > we
> > > >> >> > >> > > could use your help in what changes do we need to make
> to
> > make
> > > >> >> that
> > > >> >> > >> > happen
> > > >> >> > >> > > for DataSketches. As I recall, our DataSketches library
> > was
> > > >> not
> > > >> >> the
> > > >> >> > >> only
> > > >> >> > >> > > Druid dependency that took advantage of Unsafe. So I
> > assume
> > > >> you
> > > >> >> > have
> > > >> >> > >> > been
> > > >> >> > >> > > down this path :)
> > > >> >> > >> > >
> > > >> >> > >> > > Currently, I have a check in the static initializer in
> > > >> >> > >> Memory/UnsafeUtil:
> > > >> >> > >> > > parseJavaVersion(...) that checks the string returned
> from
> > > >> >> > >> > > System.getProperty("java.version"). If that string does
> > not
> > > >> >> contain
> > > >> >> > >> > "1.8"
> > > >> >> > >> > > or "8" it will throw an error. Given that we have not
> > tested
> > > >> >> with
> > > >> >> > >> JDK 9,
> > > >> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw
> > rather
> > > >> than
> > > >> >> > >> having
> > > >> >> > >> > > the users experience some weird failure later that may
> be
> > > >> >> difficult
> > > >> >> > to
> > > >> >> > >> > > debug.
> > > >> >> > >> > >
> > > >> >> > >> > > With only one minor exception (which we could easily
> fix)
> > the
> > > >> >> Memory
> > > >> >> > >> > > component is the only place where Unsafe is used.
> > However,
> > > >> we do
> > > >> >> > use
> > > >> >> > >> > > reflection to gain internal access to a number of other
> > > >> classes
> > > >> >> > >> including
> > > >> >> > >> > > ByteBuffer, Cleaner, and Bits.
> > > >> >> > >> > >
> > > >> >> > >> > > Looking forward to hearing from you.
> > > >> >> > >> > >
> > > >> >> > >> > > Cheers,
> > > >> >> > >> > >
> > > >> >> > >> > > Lee.
> > > >> >> > >> > >
> > > >> >> > >> >
> > > >> >> > >>
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> > For additional commands, e-mail: dev-help@datasketches.apache.org
> >
> >
>
Re: Help with Migration to OpenJDK 11
Posted by Jon Malkin <jo...@gmail.com>.
The memory repository is the one that needs the most conversion help.
Everything using Unsafe in datasketches-java runs through the memory
package.
I'm not sure off-hand if the characterization repo has low level tests of
Memory (on my phone at the moment), but we'd be looking to minimize the
speed impact in the process since it very directly impacts sketch speed.
jon
On Thu, Oct 17, 2019, 4:43 PM Christopher <ct...@apache.org> wrote:
> I've converted several other projects to using JDK 11, and can help
> with this if there's still a need for help. I've actually been looking
> at ways I can contribute to DataSketches since ApacheCon, and this
> seems to be something I can probably do.
>
> On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
> >
> > Gian,
> >
> > If I were to supply you with a link to a snapshot or a Release Candidate
> > version of the datasketches jar (and Memory jar) could you test in your
> > environment under JDK 11 and see if it works?
> >
> > Lee
> >
> >
> > On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> >
> > > Holy Cow! I have never seen such a complex travis config!
> > >
> > > On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > >> I should have linked Druid's Travis config file for reference. Here
> it is:
> > >> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> > >>
> > >> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > > So what exactly is the requirement? Our DataSketches code does
> > >> compile
> > >> > > under JDK8 and we have lots of customers and users using JDK 8.
> > >> >
> > >> > It depends on who you ask, I guess :)
> > >> >
> > >> > From Druid's perspective, we aren't going to be compiling your code,
> > >> only
> > >> > running it. So the only requirement we would ask for is that the
> code
> > >> > _runs_ under a Java 11 runtime. Being able to compile under Java 11
> is
> > >> good
> > >> > for future-proofing your code, though. It is starting to become more
> > >> > prevalent.
> > >> >
> > >> > > It sounds like (as far as Unsafe is concerned) converting all the
> > >> direct
> > >> > > unsafe.blah() calls to static MethodHandle.invoke calls is what is
> > >> > required.
> > >> > >
> > >> > > Is this correct?
> > >> >
> > >> > That is certainly part of it, I'm not 100% sure if that's all that's
> > >> > needed. Btw, CI can help verify. In Druid's Travis config, we run
> our
> > >> tests
> > >> > on both JDK 8 and JDK 11. In order to make this automated testing
> > >> easier,
> > >> > we chose to do the work needed to compile Druid on JDK 11, even
> though
> > >> we
> > >> > still compile release builds using JDK 8 only. (If we hadn't done
> this
> > >> > work, then we'd have to compile on 8 and test on 11, which Travis
> > >> doesn't
> > >> > easily support.)
> > >> >
> > >> > On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com> wrote:
> > >> >
> > >> >> Thanks, Gian, this is VERY helpful!
> > >> >>
> > >> >> From my cursory glance, it appears that UnsafeUtil acquires Unsafe
> and
> > >> >> Unsafe-class the usual way using reflection (I didn't think this
> would
> > >> >> work
> > >> >> with 9+ ), and then defines some method-handles that the other
> classes
> > >> >> use.
> > >> >>
> > >> >> Will this approach allow code to compile under JDK11 and JDK8?
> > >> >>
> > >> >> It also seems that compiling with JDK11 is still a work in
> progress as
> > >> the
> > >> >> pom.xml clearly specifies JDK8.
> > >> >>
> > >> >> So what exactly is the requirement? Our DataSketches code does
> compile
> > >> >> under JDK8 and we have lots of customers and users using JDK 8.
> > >> >>
> > >> >> It sounds like (as far as Unsafe is concerned) converting all the
> > >> direct
> > >> >> unsafe.blah() calls to static MethodHandle.invoke calls is what is
> > >> >> required.
> > >> >>
> > >> >> Is this correct?
> > >> >>
> > >> >> Lee.
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org>
> wrote:
> > >> >>
> > >> >> > And nobody should be using the Oracle JDK without a commercial
> > >> >> relationship
> > >> >> > with Oracle, its license is a nightmare.
> > >> >> >
> > >> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org>
> > >> wrote:
> > >> >> >
> > >> >> > > Oracle and OpenJDK builds behave the same as each other now.
> The
> > >> >> > > differences are in terms of licensing, length of support, and
> > >> release
> > >> >> > > cadence. IMO most Java devs should be using one of the
> third-party
> > >> >> > OpenJDK
> > >> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than
> > >> vanilla
> > >> >> > > OpenJDK these days, because of the new OpenJDK policy that
> public
> > >> >> updates
> > >> >> > > will cease for major versions after 6 months.
> > >> >> > >
> > >> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com>
> wrote:
> > >> >> > >
> > >> >> > >> Thanks, Gian,
> > >> >> > >>
> > >> >> > >> Your references are helpful and I am studying them.
> > >> >> > >>
> > >> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> > >> >> > >>
> > >> >> > >> Are there any differences in the way Unsafe calls (or other
> hidden
> > >> >> > >> classes)
> > >> >> > >> are handled by OpenJDK? Or is this a "don't care" as far as
> > >> Druid is
> > >> >> > >> concerned? Are the version numbers between Oracle and OpenJDK
> > >> always
> > >> >> > >> aligned?
> > >> >> > >>
> > >> >> > >>
> > >> >> > >> Lee.
> > >> >> > >>
> > >> >> > >>
> > >> >> > >>
> > >> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gian@apache.org
> >
> > >> >> wrote:
> > >> >> > >>
> > >> >> > >> > You didn't address this message to me, but in Druid most of
> the
> > >> >> work
> > >> >> > for
> > >> >> > >> > Java 9+ compatibility is mentioned in this master issue,
> which
> > >> you
> > >> >> > might
> > >> >> > >> > find helpful:
> > >> >> https://github.com/apache/incubator-druid/issues/5589.
> > >> >> > >> > Skimming the list, these might be particularly relevant to
> > >> >> > DataSketches:
> > >> >> > >> >
> > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7487
> (cleaner
> > >> >> > >> operations)
> > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7466
> > >> (ByteBuffer
> > >> >> > unmap
> > >> >> > >> > operation)
> > >> >> > >> > - https://github.com/apache/incubator-druid/pull/7576
> (Remove
> > >> >> direct
> > >> >> > >> > references to Unsafe)
> > >> >> > >> >
> > >> >> > >> > Btw, it would be better, I think, to address emails to the
> list
> > >> at
> > >> >> > >> large.
> > >> >> > >> > It encourages more people to participate. If Roman is a
> > >> subscriber
> > >> >> he
> > >> >> > >> would
> > >> >> > >> > get a copy anyway. If not, you could encourage him to
> subscribe.
> > >> >> > >> >
> > >> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com>
> > >> wrote:
> > >> >> > >> >
> > >> >> > >> > > Roman,
> > >> >> > >> > >
> > >> >> > >> > > I hope you are doing well.
> > >> >> > >> > >
> > >> >> > >> > > I really appreciate the contributions you made to our
> Memory
> > >> >> > >> component!
> > >> >> > >> > >
> > >> >> > >> > > Given that Druid has made its code base compatible with
> > >> OpenJDK
> > >> >> 11,
> > >> >> > we
> > >> >> > >> > > could use your help in what changes do we need to make to
> make
> > >> >> that
> > >> >> > >> > happen
> > >> >> > >> > > for DataSketches. As I recall, our DataSketches library
> was
> > >> not
> > >> >> the
> > >> >> > >> only
> > >> >> > >> > > Druid dependency that took advantage of Unsafe. So I
> assume
> > >> you
> > >> >> > have
> > >> >> > >> > been
> > >> >> > >> > > down this path :)
> > >> >> > >> > >
> > >> >> > >> > > Currently, I have a check in the static initializer in
> > >> >> > >> Memory/UnsafeUtil:
> > >> >> > >> > > parseJavaVersion(...) that checks the string returned from
> > >> >> > >> > > System.getProperty("java.version"). If that string does
> not
> > >> >> contain
> > >> >> > >> > "1.8"
> > >> >> > >> > > or "8" it will throw an error. Given that we have not
> tested
> > >> >> with
> > >> >> > >> JDK 9,
> > >> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw
> rather
> > >> than
> > >> >> > >> having
> > >> >> > >> > > the users experience some weird failure later that may be
> > >> >> difficult
> > >> >> > to
> > >> >> > >> > > debug.
> > >> >> > >> > >
> > >> >> > >> > > With only one minor exception (which we could easily fix)
> the
> > >> >> Memory
> > >> >> > >> > > component is the only place where Unsafe is used.
> However,
> > >> we do
> > >> >> > use
> > >> >> > >> > > reflection to gain internal access to a number of other
> > >> classes
> > >> >> > >> including
> > >> >> > >> > > ByteBuffer, Cleaner, and Bits.
> > >> >> > >> > >
> > >> >> > >> > > Looking forward to hearing from you.
> > >> >> > >> > >
> > >> >> > >> > > Cheers,
> > >> >> > >> > >
> > >> >> > >> > > Lee.
> > >> >> > >> > >
> > >> >> > >> >
> > >> >> > >>
> > >> >> > >
> > >> >> >
> > >> >>
> > >> >
> > >>
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
> For additional commands, e-mail: dev-help@datasketches.apache.org
>
>
Re: Help with Migration to OpenJDK 11
Posted by Christopher <ct...@apache.org>.
I've converted several other projects to using JDK 11, and can help
with this if there's still a need for help. I've actually been looking
at ways I can contribute to DataSketches since ApacheCon, and this
seems to be something I can probably do.
On Wed, Oct 16, 2019 at 9:47 PM leerho <le...@gmail.com> wrote:
>
> Gian,
>
> If I were to supply you with a link to a snapshot or a Release Candidate
> version of the datasketches jar (and Memory jar) could you test in your
> environment under JDK 11 and see if it works?
>
> Lee
>
>
> On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
>
> > Holy Cow! I have never seen such a complex travis config!
> >
> > On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org> wrote:
> >
> >> I should have linked Druid's Travis config file for reference. Here it is:
> >> https://github.com/apache/incubator-druid/blob/master/.travis.yml
> >>
> >> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > > So what exactly is the requirement? Our DataSketches code does
> >> compile
> >> > > under JDK8 and we have lots of customers and users using JDK 8.
> >> >
> >> > It depends on who you ask, I guess :)
> >> >
> >> > From Druid's perspective, we aren't going to be compiling your code,
> >> only
> >> > running it. So the only requirement we would ask for is that the code
> >> > _runs_ under a Java 11 runtime. Being able to compile under Java 11 is
> >> good
> >> > for future-proofing your code, though. It is starting to become more
> >> > prevalent.
> >> >
> >> > > It sounds like (as far as Unsafe is concerned) converting all the
> >> direct
> >> > > unsafe.blah() calls to static MethodHandle.invoke calls is what is
> >> > required.
> >> > >
> >> > > Is this correct?
> >> >
> >> > That is certainly part of it, I'm not 100% sure if that's all that's
> >> > needed. Btw, CI can help verify. In Druid's Travis config, we run our
> >> tests
> >> > on both JDK 8 and JDK 11. In order to make this automated testing
> >> easier,
> >> > we chose to do the work needed to compile Druid on JDK 11, even though
> >> we
> >> > still compile release builds using JDK 8 only. (If we hadn't done this
> >> > work, then we'd have to compile on 8 and test on 11, which Travis
> >> doesn't
> >> > easily support.)
> >> >
> >> > On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com> wrote:
> >> >
> >> >> Thanks, Gian, this is VERY helpful!
> >> >>
> >> >> From my cursory glance, it appears that UnsafeUtil acquires Unsafe and
> >> >> Unsafe-class the usual way using reflection (I didn't think this would
> >> >> work
> >> >> with 9+ ), and then defines some method-handles that the other classes
> >> >> use.
> >> >>
> >> >> Will this approach allow code to compile under JDK11 and JDK8?
> >> >>
> >> >> It also seems that compiling with JDK11 is still a work in progress as
> >> the
> >> >> pom.xml clearly specifies JDK8.
> >> >>
> >> >> So what exactly is the requirement? Our DataSketches code does compile
> >> >> under JDK8 and we have lots of customers and users using JDK 8.
> >> >>
> >> >> It sounds like (as far as Unsafe is concerned) converting all the
> >> direct
> >> >> unsafe.blah() calls to static MethodHandle.invoke calls is what is
> >> >> required.
> >> >>
> >> >> Is this correct?
> >> >>
> >> >> Lee.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org> wrote:
> >> >>
> >> >> > And nobody should be using the Oracle JDK without a commercial
> >> >> relationship
> >> >> > with Oracle, its license is a nightmare.
> >> >> >
> >> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org>
> >> wrote:
> >> >> >
> >> >> > > Oracle and OpenJDK builds behave the same as each other now. The
> >> >> > > differences are in terms of licensing, length of support, and
> >> release
> >> >> > > cadence. IMO most Java devs should be using one of the third-party
> >> >> > OpenJDK
> >> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than
> >> vanilla
> >> >> > > OpenJDK these days, because of the new OpenJDK policy that public
> >> >> updates
> >> >> > > will cease for major versions after 6 months.
> >> >> > >
> >> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
> >> >> > >
> >> >> > >> Thanks, Gian,
> >> >> > >>
> >> >> > >> Your references are helpful and I am studying them.
> >> >> > >>
> >> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> >> >> > >>
> >> >> > >> Are there any differences in the way Unsafe calls (or other hidden
> >> >> > >> classes)
> >> >> > >> are handled by OpenJDK? Or is this a "don't care" as far as
> >> Druid is
> >> >> > >> concerned? Are the version numbers between Oracle and OpenJDK
> >> always
> >> >> > >> aligned?
> >> >> > >>
> >> >> > >>
> >> >> > >> Lee.
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org>
> >> >> wrote:
> >> >> > >>
> >> >> > >> > You didn't address this message to me, but in Druid most of the
> >> >> work
> >> >> > for
> >> >> > >> > Java 9+ compatibility is mentioned in this master issue, which
> >> you
> >> >> > might
> >> >> > >> > find helpful:
> >> >> https://github.com/apache/incubator-druid/issues/5589.
> >> >> > >> > Skimming the list, these might be particularly relevant to
> >> >> > DataSketches:
> >> >> > >> >
> >> >> > >> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
> >> >> > >> operations)
> >> >> > >> > - https://github.com/apache/incubator-druid/pull/7466
> >> (ByteBuffer
> >> >> > unmap
> >> >> > >> > operation)
> >> >> > >> > - https://github.com/apache/incubator-druid/pull/7576 (Remove
> >> >> direct
> >> >> > >> > references to Unsafe)
> >> >> > >> >
> >> >> > >> > Btw, it would be better, I think, to address emails to the list
> >> at
> >> >> > >> large.
> >> >> > >> > It encourages more people to participate. If Roman is a
> >> subscriber
> >> >> he
> >> >> > >> would
> >> >> > >> > get a copy anyway. If not, you could encourage him to subscribe.
> >> >> > >> >
> >> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com>
> >> wrote:
> >> >> > >> >
> >> >> > >> > > Roman,
> >> >> > >> > >
> >> >> > >> > > I hope you are doing well.
> >> >> > >> > >
> >> >> > >> > > I really appreciate the contributions you made to our Memory
> >> >> > >> component!
> >> >> > >> > >
> >> >> > >> > > Given that Druid has made its code base compatible with
> >> OpenJDK
> >> >> 11,
> >> >> > we
> >> >> > >> > > could use your help in what changes do we need to make to make
> >> >> that
> >> >> > >> > happen
> >> >> > >> > > for DataSketches. As I recall, our DataSketches library was
> >> not
> >> >> the
> >> >> > >> only
> >> >> > >> > > Druid dependency that took advantage of Unsafe. So I assume
> >> you
> >> >> > have
> >> >> > >> > been
> >> >> > >> > > down this path :)
> >> >> > >> > >
> >> >> > >> > > Currently, I have a check in the static initializer in
> >> >> > >> Memory/UnsafeUtil:
> >> >> > >> > > parseJavaVersion(...) that checks the string returned from
> >> >> > >> > > System.getProperty("java.version"). If that string does not
> >> >> contain
> >> >> > >> > "1.8"
> >> >> > >> > > or "8" it will throw an error. Given that we have not tested
> >> >> with
> >> >> > >> JDK 9,
> >> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw rather
> >> than
> >> >> > >> having
> >> >> > >> > > the users experience some weird failure later that may be
> >> >> difficult
> >> >> > to
> >> >> > >> > > debug.
> >> >> > >> > >
> >> >> > >> > > With only one minor exception (which we could easily fix) the
> >> >> Memory
> >> >> > >> > > component is the only place where Unsafe is used. However,
> >> we do
> >> >> > use
> >> >> > >> > > reflection to gain internal access to a number of other
> >> classes
> >> >> > >> including
> >> >> > >> > > ByteBuffer, Cleaner, and Bits.
> >> >> > >> > >
> >> >> > >> > > Looking forward to hearing from you.
> >> >> > >> > >
> >> >> > >> > > Cheers,
> >> >> > >> > >
> >> >> > >> > > Lee.
> >> >> > >> > >
> >> >> > >> >
> >> >> > >>
> >> >> > >
> >> >> >
> >> >>
> >> >
> >>
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@datasketches.apache.org
For additional commands, e-mail: dev-help@datasketches.apache.org
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Gian,
If I were to supply you with a link to a snapshot or a Release Candidate
version of the datasketches jar (and Memory jar) could you test in your
environment under JDK 11 and see if it works?
Lee
On Mon, Oct 14, 2019 at 9:19 PM leerho <le...@gmail.com> wrote:
> Holy Cow! I have never seen such a complex travis config!
>
> On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org> wrote:
>
>> I should have linked Druid's Travis config file for reference. Here it is:
>> https://github.com/apache/incubator-druid/blob/master/.travis.yml
>>
>> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org> wrote:
>>
>> > > So what exactly is the requirement? Our DataSketches code does
>> compile
>> > > under JDK8 and we have lots of customers and users using JDK 8.
>> >
>> > It depends on who you ask, I guess :)
>> >
>> > From Druid's perspective, we aren't going to be compiling your code,
>> only
>> > running it. So the only requirement we would ask for is that the code
>> > _runs_ under a Java 11 runtime. Being able to compile under Java 11 is
>> good
>> > for future-proofing your code, though. It is starting to become more
>> > prevalent.
>> >
>> > > It sounds like (as far as Unsafe is concerned) converting all the
>> direct
>> > > unsafe.blah() calls to static MethodHandle.invoke calls is what is
>> > required.
>> > >
>> > > Is this correct?
>> >
>> > That is certainly part of it, I'm not 100% sure if that's all that's
>> > needed. Btw, CI can help verify. In Druid's Travis config, we run our
>> tests
>> > on both JDK 8 and JDK 11. In order to make this automated testing
>> easier,
>> > we chose to do the work needed to compile Druid on JDK 11, even though
>> we
>> > still compile release builds using JDK 8 only. (If we hadn't done this
>> > work, then we'd have to compile on 8 and test on 11, which Travis
>> doesn't
>> > easily support.)
>> >
>> > On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com> wrote:
>> >
>> >> Thanks, Gian, this is VERY helpful!
>> >>
>> >> From my cursory glance, it appears that UnsafeUtil acquires Unsafe and
>> >> Unsafe-class the usual way using reflection (I didn't think this would
>> >> work
>> >> with 9+ ), and then defines some method-handles that the other classes
>> >> use.
>> >>
>> >> Will this approach allow code to compile under JDK11 and JDK8?
>> >>
>> >> It also seems that compiling with JDK11 is still a work in progress as
>> the
>> >> pom.xml clearly specifies JDK8.
>> >>
>> >> So what exactly is the requirement? Our DataSketches code does compile
>> >> under JDK8 and we have lots of customers and users using JDK 8.
>> >>
>> >> It sounds like (as far as Unsafe is concerned) converting all the
>> direct
>> >> unsafe.blah() calls to static MethodHandle.invoke calls is what is
>> >> required.
>> >>
>> >> Is this correct?
>> >>
>> >> Lee.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org> wrote:
>> >>
>> >> > And nobody should be using the Oracle JDK without a commercial
>> >> relationship
>> >> > with Oracle, its license is a nightmare.
>> >> >
>> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org>
>> wrote:
>> >> >
>> >> > > Oracle and OpenJDK builds behave the same as each other now. The
>> >> > > differences are in terms of licensing, length of support, and
>> release
>> >> > > cadence. IMO most Java devs should be using one of the third-party
>> >> > OpenJDK
>> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than
>> vanilla
>> >> > > OpenJDK these days, because of the new OpenJDK policy that public
>> >> updates
>> >> > > will cease for major versions after 6 months.
>> >> > >
>> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
>> >> > >
>> >> > >> Thanks, Gian,
>> >> > >>
>> >> > >> Your references are helpful and I am studying them.
>> >> > >>
>> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
>> >> > >>
>> >> > >> Are there any differences in the way Unsafe calls (or other hidden
>> >> > >> classes)
>> >> > >> are handled by OpenJDK? Or is this a "don't care" as far as
>> Druid is
>> >> > >> concerned? Are the version numbers between Oracle and OpenJDK
>> always
>> >> > >> aligned?
>> >> > >>
>> >> > >>
>> >> > >> Lee.
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org>
>> >> wrote:
>> >> > >>
>> >> > >> > You didn't address this message to me, but in Druid most of the
>> >> work
>> >> > for
>> >> > >> > Java 9+ compatibility is mentioned in this master issue, which
>> you
>> >> > might
>> >> > >> > find helpful:
>> >> https://github.com/apache/incubator-druid/issues/5589.
>> >> > >> > Skimming the list, these might be particularly relevant to
>> >> > DataSketches:
>> >> > >> >
>> >> > >> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
>> >> > >> operations)
>> >> > >> > - https://github.com/apache/incubator-druid/pull/7466
>> (ByteBuffer
>> >> > unmap
>> >> > >> > operation)
>> >> > >> > - https://github.com/apache/incubator-druid/pull/7576 (Remove
>> >> direct
>> >> > >> > references to Unsafe)
>> >> > >> >
>> >> > >> > Btw, it would be better, I think, to address emails to the list
>> at
>> >> > >> large.
>> >> > >> > It encourages more people to participate. If Roman is a
>> subscriber
>> >> he
>> >> > >> would
>> >> > >> > get a copy anyway. If not, you could encourage him to subscribe.
>> >> > >> >
>> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com>
>> wrote:
>> >> > >> >
>> >> > >> > > Roman,
>> >> > >> > >
>> >> > >> > > I hope you are doing well.
>> >> > >> > >
>> >> > >> > > I really appreciate the contributions you made to our Memory
>> >> > >> component!
>> >> > >> > >
>> >> > >> > > Given that Druid has made its code base compatible with
>> OpenJDK
>> >> 11,
>> >> > we
>> >> > >> > > could use your help in what changes do we need to make to make
>> >> that
>> >> > >> > happen
>> >> > >> > > for DataSketches. As I recall, our DataSketches library was
>> not
>> >> the
>> >> > >> only
>> >> > >> > > Druid dependency that took advantage of Unsafe. So I assume
>> you
>> >> > have
>> >> > >> > been
>> >> > >> > > down this path :)
>> >> > >> > >
>> >> > >> > > Currently, I have a check in the static initializer in
>> >> > >> Memory/UnsafeUtil:
>> >> > >> > > parseJavaVersion(...) that checks the string returned from
>> >> > >> > > System.getProperty("java.version"). If that string does not
>> >> contain
>> >> > >> > "1.8"
>> >> > >> > > or "8" it will throw an error. Given that we have not tested
>> >> with
>> >> > >> JDK 9,
>> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw rather
>> than
>> >> > >> having
>> >> > >> > > the users experience some weird failure later that may be
>> >> difficult
>> >> > to
>> >> > >> > > debug.
>> >> > >> > >
>> >> > >> > > With only one minor exception (which we could easily fix) the
>> >> Memory
>> >> > >> > > component is the only place where Unsafe is used. However,
>> we do
>> >> > use
>> >> > >> > > reflection to gain internal access to a number of other
>> classes
>> >> > >> including
>> >> > >> > > ByteBuffer, Cleaner, and Bits.
>> >> > >> > >
>> >> > >> > > Looking forward to hearing from you.
>> >> > >> > >
>> >> > >> > > Cheers,
>> >> > >> > >
>> >> > >> > > Lee.
>> >> > >> > >
>> >> > >> >
>> >> > >>
>> >> > >
>> >> >
>> >>
>> >
>>
>
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Holy Cow! I have never seen such a complex travis config!
On Mon, Oct 14, 2019 at 6:54 PM Gian Merlino <gi...@apache.org> wrote:
> I should have linked Druid's Travis config file for reference. Here it is:
> https://github.com/apache/incubator-druid/blob/master/.travis.yml
>
> On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org> wrote:
>
> > > So what exactly is the requirement? Our DataSketches code does compile
> > > under JDK8 and we have lots of customers and users using JDK 8.
> >
> > It depends on who you ask, I guess :)
> >
> > From Druid's perspective, we aren't going to be compiling your code, only
> > running it. So the only requirement we would ask for is that the code
> > _runs_ under a Java 11 runtime. Being able to compile under Java 11 is
> good
> > for future-proofing your code, though. It is starting to become more
> > prevalent.
> >
> > > It sounds like (as far as Unsafe is concerned) converting all the
> direct
> > > unsafe.blah() calls to static MethodHandle.invoke calls is what is
> > required.
> > >
> > > Is this correct?
> >
> > That is certainly part of it, I'm not 100% sure if that's all that's
> > needed. Btw, CI can help verify. In Druid's Travis config, we run our
> tests
> > on both JDK 8 and JDK 11. In order to make this automated testing easier,
> > we chose to do the work needed to compile Druid on JDK 11, even though we
> > still compile release builds using JDK 8 only. (If we hadn't done this
> > work, then we'd have to compile on 8 and test on 11, which Travis doesn't
> > easily support.)
> >
> > On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com> wrote:
> >
> >> Thanks, Gian, this is VERY helpful!
> >>
> >> From my cursory glance, it appears that UnsafeUtil acquires Unsafe and
> >> Unsafe-class the usual way using reflection (I didn't think this would
> >> work
> >> with 9+ ), and then defines some method-handles that the other classes
> >> use.
> >>
> >> Will this approach allow code to compile under JDK11 and JDK8?
> >>
> >> It also seems that compiling with JDK11 is still a work in progress as
> the
> >> pom.xml clearly specifies JDK8.
> >>
> >> So what exactly is the requirement? Our DataSketches code does compile
> >> under JDK8 and we have lots of customers and users using JDK 8.
> >>
> >> It sounds like (as far as Unsafe is concerned) converting all the direct
> >> unsafe.blah() calls to static MethodHandle.invoke calls is what is
> >> required.
> >>
> >> Is this correct?
> >>
> >> Lee.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > And nobody should be using the Oracle JDK without a commercial
> >> relationship
> >> > with Oracle, its license is a nightmare.
> >> >
> >> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org> wrote:
> >> >
> >> > > Oracle and OpenJDK builds behave the same as each other now. The
> >> > > differences are in terms of licensing, length of support, and
> release
> >> > > cadence. IMO most Java devs should be using one of the third-party
> >> > OpenJDK
> >> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than
> vanilla
> >> > > OpenJDK these days, because of the new OpenJDK policy that public
> >> updates
> >> > > will cease for major versions after 6 months.
> >> > >
> >> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
> >> > >
> >> > >> Thanks, Gian,
> >> > >>
> >> > >> Your references are helpful and I am studying them.
> >> > >>
> >> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> >> > >>
> >> > >> Are there any differences in the way Unsafe calls (or other hidden
> >> > >> classes)
> >> > >> are handled by OpenJDK? Or is this a "don't care" as far as Druid
> is
> >> > >> concerned? Are the version numbers between Oracle and OpenJDK
> always
> >> > >> aligned?
> >> > >>
> >> > >>
> >> > >> Lee.
> >> > >>
> >> > >>
> >> > >>
> >> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org>
> >> wrote:
> >> > >>
> >> > >> > You didn't address this message to me, but in Druid most of the
> >> work
> >> > for
> >> > >> > Java 9+ compatibility is mentioned in this master issue, which
> you
> >> > might
> >> > >> > find helpful:
> >> https://github.com/apache/incubator-druid/issues/5589.
> >> > >> > Skimming the list, these might be particularly relevant to
> >> > DataSketches:
> >> > >> >
> >> > >> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
> >> > >> operations)
> >> > >> > - https://github.com/apache/incubator-druid/pull/7466
> (ByteBuffer
> >> > unmap
> >> > >> > operation)
> >> > >> > - https://github.com/apache/incubator-druid/pull/7576 (Remove
> >> direct
> >> > >> > references to Unsafe)
> >> > >> >
> >> > >> > Btw, it would be better, I think, to address emails to the list
> at
> >> > >> large.
> >> > >> > It encourages more people to participate. If Roman is a
> subscriber
> >> he
> >> > >> would
> >> > >> > get a copy anyway. If not, you could encourage him to subscribe.
> >> > >> >
> >> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
> >> > >> >
> >> > >> > > Roman,
> >> > >> > >
> >> > >> > > I hope you are doing well.
> >> > >> > >
> >> > >> > > I really appreciate the contributions you made to our Memory
> >> > >> component!
> >> > >> > >
> >> > >> > > Given that Druid has made its code base compatible with OpenJDK
> >> 11,
> >> > we
> >> > >> > > could use your help in what changes do we need to make to make
> >> that
> >> > >> > happen
> >> > >> > > for DataSketches. As I recall, our DataSketches library was
> not
> >> the
> >> > >> only
> >> > >> > > Druid dependency that took advantage of Unsafe. So I assume
> you
> >> > have
> >> > >> > been
> >> > >> > > down this path :)
> >> > >> > >
> >> > >> > > Currently, I have a check in the static initializer in
> >> > >> Memory/UnsafeUtil:
> >> > >> > > parseJavaVersion(...) that checks the string returned from
> >> > >> > > System.getProperty("java.version"). If that string does not
> >> contain
> >> > >> > "1.8"
> >> > >> > > or "8" it will throw an error. Given that we have not tested
> >> with
> >> > >> JDK 9,
> >> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw rather
> than
> >> > >> having
> >> > >> > > the users experience some weird failure later that may be
> >> difficult
> >> > to
> >> > >> > > debug.
> >> > >> > >
> >> > >> > > With only one minor exception (which we could easily fix) the
> >> Memory
> >> > >> > > component is the only place where Unsafe is used. However, we
> do
> >> > use
> >> > >> > > reflection to gain internal access to a number of other classes
> >> > >> including
> >> > >> > > ByteBuffer, Cleaner, and Bits.
> >> > >> > >
> >> > >> > > Looking forward to hearing from you.
> >> > >> > >
> >> > >> > > Cheers,
> >> > >> > >
> >> > >> > > Lee.
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >
> >> >
> >>
> >
>
Re: Help with Migration to OpenJDK 11
Posted by Gian Merlino <gi...@apache.org>.
I should have linked Druid's Travis config file for reference. Here it is:
https://github.com/apache/incubator-druid/blob/master/.travis.yml
On Mon, Oct 14, 2019 at 6:53 PM Gian Merlino <gi...@apache.org> wrote:
> > So what exactly is the requirement? Our DataSketches code does compile
> > under JDK8 and we have lots of customers and users using JDK 8.
>
> It depends on who you ask, I guess :)
>
> From Druid's perspective, we aren't going to be compiling your code, only
> running it. So the only requirement we would ask for is that the code
> _runs_ under a Java 11 runtime. Being able to compile under Java 11 is good
> for future-proofing your code, though. It is starting to become more
> prevalent.
>
> > It sounds like (as far as Unsafe is concerned) converting all the direct
> > unsafe.blah() calls to static MethodHandle.invoke calls is what is
> required.
> >
> > Is this correct?
>
> That is certainly part of it, I'm not 100% sure if that's all that's
> needed. Btw, CI can help verify. In Druid's Travis config, we run our tests
> on both JDK 8 and JDK 11. In order to make this automated testing easier,
> we chose to do the work needed to compile Druid on JDK 11, even though we
> still compile release builds using JDK 8 only. (If we hadn't done this
> work, then we'd have to compile on 8 and test on 11, which Travis doesn't
> easily support.)
>
> On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com> wrote:
>
>> Thanks, Gian, this is VERY helpful!
>>
>> From my cursory glance, it appears that UnsafeUtil acquires Unsafe and
>> Unsafe-class the usual way using reflection (I didn't think this would
>> work
>> with 9+ ), and then defines some method-handles that the other classes
>> use.
>>
>> Will this approach allow code to compile under JDK11 and JDK8?
>>
>> It also seems that compiling with JDK11 is still a work in progress as the
>> pom.xml clearly specifies JDK8.
>>
>> So what exactly is the requirement? Our DataSketches code does compile
>> under JDK8 and we have lots of customers and users using JDK 8.
>>
>> It sounds like (as far as Unsafe is concerned) converting all the direct
>> unsafe.blah() calls to static MethodHandle.invoke calls is what is
>> required.
>>
>> Is this correct?
>>
>> Lee.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org> wrote:
>>
>> > And nobody should be using the Oracle JDK without a commercial
>> relationship
>> > with Oracle, its license is a nightmare.
>> >
>> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org> wrote:
>> >
>> > > Oracle and OpenJDK builds behave the same as each other now. The
>> > > differences are in terms of licensing, length of support, and release
>> > > cadence. IMO most Java devs should be using one of the third-party
>> > OpenJDK
>> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than vanilla
>> > > OpenJDK these days, because of the new OpenJDK policy that public
>> updates
>> > > will cease for major versions after 6 months.
>> > >
>> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
>> > >
>> > >> Thanks, Gian,
>> > >>
>> > >> Your references are helpful and I am studying them.
>> > >>
>> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
>> > >>
>> > >> Are there any differences in the way Unsafe calls (or other hidden
>> > >> classes)
>> > >> are handled by OpenJDK? Or is this a "don't care" as far as Druid is
>> > >> concerned? Are the version numbers between Oracle and OpenJDK always
>> > >> aligned?
>> > >>
>> > >>
>> > >> Lee.
>> > >>
>> > >>
>> > >>
>> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org>
>> wrote:
>> > >>
>> > >> > You didn't address this message to me, but in Druid most of the
>> work
>> > for
>> > >> > Java 9+ compatibility is mentioned in this master issue, which you
>> > might
>> > >> > find helpful:
>> https://github.com/apache/incubator-druid/issues/5589.
>> > >> > Skimming the list, these might be particularly relevant to
>> > DataSketches:
>> > >> >
>> > >> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
>> > >> operations)
>> > >> > - https://github.com/apache/incubator-druid/pull/7466 (ByteBuffer
>> > unmap
>> > >> > operation)
>> > >> > - https://github.com/apache/incubator-druid/pull/7576 (Remove
>> direct
>> > >> > references to Unsafe)
>> > >> >
>> > >> > Btw, it would be better, I think, to address emails to the list at
>> > >> large.
>> > >> > It encourages more people to participate. If Roman is a subscriber
>> he
>> > >> would
>> > >> > get a copy anyway. If not, you could encourage him to subscribe.
>> > >> >
>> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
>> > >> >
>> > >> > > Roman,
>> > >> > >
>> > >> > > I hope you are doing well.
>> > >> > >
>> > >> > > I really appreciate the contributions you made to our Memory
>> > >> component!
>> > >> > >
>> > >> > > Given that Druid has made its code base compatible with OpenJDK
>> 11,
>> > we
>> > >> > > could use your help in what changes do we need to make to make
>> that
>> > >> > happen
>> > >> > > for DataSketches. As I recall, our DataSketches library was not
>> the
>> > >> only
>> > >> > > Druid dependency that took advantage of Unsafe. So I assume you
>> > have
>> > >> > been
>> > >> > > down this path :)
>> > >> > >
>> > >> > > Currently, I have a check in the static initializer in
>> > >> Memory/UnsafeUtil:
>> > >> > > parseJavaVersion(...) that checks the string returned from
>> > >> > > System.getProperty("java.version"). If that string does not
>> contain
>> > >> > "1.8"
>> > >> > > or "8" it will throw an error. Given that we have not tested
>> with
>> > >> JDK 9,
>> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw rather than
>> > >> having
>> > >> > > the users experience some weird failure later that may be
>> difficult
>> > to
>> > >> > > debug.
>> > >> > >
>> > >> > > With only one minor exception (which we could easily fix) the
>> Memory
>> > >> > > component is the only place where Unsafe is used. However, we do
>> > use
>> > >> > > reflection to gain internal access to a number of other classes
>> > >> including
>> > >> > > ByteBuffer, Cleaner, and Bits.
>> > >> > >
>> > >> > > Looking forward to hearing from you.
>> > >> > >
>> > >> > > Cheers,
>> > >> > >
>> > >> > > Lee.
>> > >> > >
>> > >> >
>> > >>
>> > >
>> >
>>
>
Re: Help with Migration to OpenJDK 11
Posted by Gian Merlino <gi...@apache.org>.
> So what exactly is the requirement? Our DataSketches code does compile
> under JDK8 and we have lots of customers and users using JDK 8.
It depends on who you ask, I guess :)
From Druid's perspective, we aren't going to be compiling your code, only
running it. So the only requirement we would ask for is that the code
_runs_ under a Java 11 runtime. Being able to compile under Java 11 is good
for future-proofing your code, though. It is starting to become more
prevalent.
> It sounds like (as far as Unsafe is concerned) converting all the direct
> unsafe.blah() calls to static MethodHandle.invoke calls is what is
required.
>
> Is this correct?
That is certainly part of it, I'm not 100% sure if that's all that's
needed. Btw, CI can help verify. In Druid's Travis config, we run our tests
on both JDK 8 and JDK 11. In order to make this automated testing easier,
we chose to do the work needed to compile Druid on JDK 11, even though we
still compile release builds using JDK 8 only. (If we hadn't done this
work, then we'd have to compile on 8 and test on 11, which Travis doesn't
easily support.)
On Mon, Oct 14, 2019 at 5:51 PM leerho <le...@gmail.com> wrote:
> Thanks, Gian, this is VERY helpful!
>
> From my cursory glance, it appears that UnsafeUtil acquires Unsafe and
> Unsafe-class the usual way using reflection (I didn't think this would work
> with 9+ ), and then defines some method-handles that the other classes
> use.
>
> Will this approach allow code to compile under JDK11 and JDK8?
>
> It also seems that compiling with JDK11 is still a work in progress as the
> pom.xml clearly specifies JDK8.
>
> So what exactly is the requirement? Our DataSketches code does compile
> under JDK8 and we have lots of customers and users using JDK 8.
>
> It sounds like (as far as Unsafe is concerned) converting all the direct
> unsafe.blah() calls to static MethodHandle.invoke calls is what is
> required.
>
> Is this correct?
>
> Lee.
>
>
>
>
>
>
>
>
>
>
> On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org> wrote:
>
> > And nobody should be using the Oracle JDK without a commercial
> relationship
> > with Oracle, its license is a nightmare.
> >
> > On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > Oracle and OpenJDK builds behave the same as each other now. The
> > > differences are in terms of licensing, length of support, and release
> > > cadence. IMO most Java devs should be using one of the third-party
> > OpenJDK
> > > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than vanilla
> > > OpenJDK these days, because of the new OpenJDK policy that public
> updates
> > > will cease for major versions after 6 months.
> > >
> > > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
> > >
> > >> Thanks, Gian,
> > >>
> > >> Your references are helpful and I am studying them.
> > >>
> > >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> > >>
> > >> Are there any differences in the way Unsafe calls (or other hidden
> > >> classes)
> > >> are handled by OpenJDK? Or is this a "don't care" as far as Druid is
> > >> concerned? Are the version numbers between Oracle and OpenJDK always
> > >> aligned?
> > >>
> > >>
> > >> Lee.
> > >>
> > >>
> > >>
> > >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org> wrote:
> > >>
> > >> > You didn't address this message to me, but in Druid most of the work
> > for
> > >> > Java 9+ compatibility is mentioned in this master issue, which you
> > might
> > >> > find helpful: https://github.com/apache/incubator-druid/issues/5589
> .
> > >> > Skimming the list, these might be particularly relevant to
> > DataSketches:
> > >> >
> > >> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
> > >> operations)
> > >> > - https://github.com/apache/incubator-druid/pull/7466 (ByteBuffer
> > unmap
> > >> > operation)
> > >> > - https://github.com/apache/incubator-druid/pull/7576 (Remove
> direct
> > >> > references to Unsafe)
> > >> >
> > >> > Btw, it would be better, I think, to address emails to the list at
> > >> large.
> > >> > It encourages more people to participate. If Roman is a subscriber
> he
> > >> would
> > >> > get a copy anyway. If not, you could encourage him to subscribe.
> > >> >
> > >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
> > >> >
> > >> > > Roman,
> > >> > >
> > >> > > I hope you are doing well.
> > >> > >
> > >> > > I really appreciate the contributions you made to our Memory
> > >> component!
> > >> > >
> > >> > > Given that Druid has made its code base compatible with OpenJDK
> 11,
> > we
> > >> > > could use your help in what changes do we need to make to make
> that
> > >> > happen
> > >> > > for DataSketches. As I recall, our DataSketches library was not
> the
> > >> only
> > >> > > Druid dependency that took advantage of Unsafe. So I assume you
> > have
> > >> > been
> > >> > > down this path :)
> > >> > >
> > >> > > Currently, I have a check in the static initializer in
> > >> Memory/UnsafeUtil:
> > >> > > parseJavaVersion(...) that checks the string returned from
> > >> > > System.getProperty("java.version"). If that string does not
> contain
> > >> > "1.8"
> > >> > > or "8" it will throw an error. Given that we have not tested with
> > >> JDK 9,
> > >> > > 10, 11 or 12, I felt it was safer to explicitly throw rather than
> > >> having
> > >> > > the users experience some weird failure later that may be
> difficult
> > to
> > >> > > debug.
> > >> > >
> > >> > > With only one minor exception (which we could easily fix) the
> Memory
> > >> > > component is the only place where Unsafe is used. However, we do
> > use
> > >> > > reflection to gain internal access to a number of other classes
> > >> including
> > >> > > ByteBuffer, Cleaner, and Bits.
> > >> > >
> > >> > > Looking forward to hearing from you.
> > >> > >
> > >> > > Cheers,
> > >> > >
> > >> > > Lee.
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Thanks, Gian, this is VERY helpful!
From my cursory glance, it appears that UnsafeUtil acquires Unsafe and
Unsafe-class the usual way using reflection (I didn't think this would work
with 9+ ), and then defines some method-handles that the other classes
use.
Will this approach allow code to compile under JDK11 and JDK8?
It also seems that compiling with JDK11 is still a work in progress as the
pom.xml clearly specifies JDK8.
So what exactly is the requirement? Our DataSketches code does compile
under JDK8 and we have lots of customers and users using JDK 8.
It sounds like (as far as Unsafe is concerned) converting all the direct
unsafe.blah() calls to static MethodHandle.invoke calls is what is required.
Is this correct?
Lee.
On Mon, Oct 14, 2019 at 2:07 PM Gian Merlino <gi...@apache.org> wrote:
> And nobody should be using the Oracle JDK without a commercial relationship
> with Oracle, its license is a nightmare.
>
> On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org> wrote:
>
> > Oracle and OpenJDK builds behave the same as each other now. The
> > differences are in terms of licensing, length of support, and release
> > cadence. IMO most Java devs should be using one of the third-party
> OpenJDK
> > distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than vanilla
> > OpenJDK these days, because of the new OpenJDK policy that public updates
> > will cease for major versions after 6 months.
> >
> > On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
> >
> >> Thanks, Gian,
> >>
> >> Your references are helpful and I am studying them.
> >>
> >> I didn't see any references to OpenJDK vs Oracle's JDKs.
> >>
> >> Are there any differences in the way Unsafe calls (or other hidden
> >> classes)
> >> are handled by OpenJDK? Or is this a "don't care" as far as Druid is
> >> concerned? Are the version numbers between Oracle and OpenJDK always
> >> aligned?
> >>
> >>
> >> Lee.
> >>
> >>
> >>
> >> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org> wrote:
> >>
> >> > You didn't address this message to me, but in Druid most of the work
> for
> >> > Java 9+ compatibility is mentioned in this master issue, which you
> might
> >> > find helpful: https://github.com/apache/incubator-druid/issues/5589.
> >> > Skimming the list, these might be particularly relevant to
> DataSketches:
> >> >
> >> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
> >> operations)
> >> > - https://github.com/apache/incubator-druid/pull/7466 (ByteBuffer
> unmap
> >> > operation)
> >> > - https://github.com/apache/incubator-druid/pull/7576 (Remove direct
> >> > references to Unsafe)
> >> >
> >> > Btw, it would be better, I think, to address emails to the list at
> >> large.
> >> > It encourages more people to participate. If Roman is a subscriber he
> >> would
> >> > get a copy anyway. If not, you could encourage him to subscribe.
> >> >
> >> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
> >> >
> >> > > Roman,
> >> > >
> >> > > I hope you are doing well.
> >> > >
> >> > > I really appreciate the contributions you made to our Memory
> >> component!
> >> > >
> >> > > Given that Druid has made its code base compatible with OpenJDK 11,
> we
> >> > > could use your help in what changes do we need to make to make that
> >> > happen
> >> > > for DataSketches. As I recall, our DataSketches library was not the
> >> only
> >> > > Druid dependency that took advantage of Unsafe. So I assume you
> have
> >> > been
> >> > > down this path :)
> >> > >
> >> > > Currently, I have a check in the static initializer in
> >> Memory/UnsafeUtil:
> >> > > parseJavaVersion(...) that checks the string returned from
> >> > > System.getProperty("java.version"). If that string does not contain
> >> > "1.8"
> >> > > or "8" it will throw an error. Given that we have not tested with
> >> JDK 9,
> >> > > 10, 11 or 12, I felt it was safer to explicitly throw rather than
> >> having
> >> > > the users experience some weird failure later that may be difficult
> to
> >> > > debug.
> >> > >
> >> > > With only one minor exception (which we could easily fix) the Memory
> >> > > component is the only place where Unsafe is used. However, we do
> use
> >> > > reflection to gain internal access to a number of other classes
> >> including
> >> > > ByteBuffer, Cleaner, and Bits.
> >> > >
> >> > > Looking forward to hearing from you.
> >> > >
> >> > > Cheers,
> >> > >
> >> > > Lee.
> >> > >
> >> >
> >>
> >
>
Re: Help with Migration to OpenJDK 11
Posted by Gian Merlino <gi...@apache.org>.
And nobody should be using the Oracle JDK without a commercial relationship
with Oracle, its license is a nightmare.
On Mon, Oct 14, 2019 at 2:04 PM Gian Merlino <gi...@apache.org> wrote:
> Oracle and OpenJDK builds behave the same as each other now. The
> differences are in terms of licensing, length of support, and release
> cadence. IMO most Java devs should be using one of the third-party OpenJDK
> distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than vanilla
> OpenJDK these days, because of the new OpenJDK policy that public updates
> will cease for major versions after 6 months.
>
> On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
>
>> Thanks, Gian,
>>
>> Your references are helpful and I am studying them.
>>
>> I didn't see any references to OpenJDK vs Oracle's JDKs.
>>
>> Are there any differences in the way Unsafe calls (or other hidden
>> classes)
>> are handled by OpenJDK? Or is this a "don't care" as far as Druid is
>> concerned? Are the version numbers between Oracle and OpenJDK always
>> aligned?
>>
>>
>> Lee.
>>
>>
>>
>> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org> wrote:
>>
>> > You didn't address this message to me, but in Druid most of the work for
>> > Java 9+ compatibility is mentioned in this master issue, which you might
>> > find helpful: https://github.com/apache/incubator-druid/issues/5589.
>> > Skimming the list, these might be particularly relevant to DataSketches:
>> >
>> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
>> operations)
>> > - https://github.com/apache/incubator-druid/pull/7466 (ByteBuffer unmap
>> > operation)
>> > - https://github.com/apache/incubator-druid/pull/7576 (Remove direct
>> > references to Unsafe)
>> >
>> > Btw, it would be better, I think, to address emails to the list at
>> large.
>> > It encourages more people to participate. If Roman is a subscriber he
>> would
>> > get a copy anyway. If not, you could encourage him to subscribe.
>> >
>> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
>> >
>> > > Roman,
>> > >
>> > > I hope you are doing well.
>> > >
>> > > I really appreciate the contributions you made to our Memory
>> component!
>> > >
>> > > Given that Druid has made its code base compatible with OpenJDK 11, we
>> > > could use your help in what changes do we need to make to make that
>> > happen
>> > > for DataSketches. As I recall, our DataSketches library was not the
>> only
>> > > Druid dependency that took advantage of Unsafe. So I assume you have
>> > been
>> > > down this path :)
>> > >
>> > > Currently, I have a check in the static initializer in
>> Memory/UnsafeUtil:
>> > > parseJavaVersion(...) that checks the string returned from
>> > > System.getProperty("java.version"). If that string does not contain
>> > "1.8"
>> > > or "8" it will throw an error. Given that we have not tested with
>> JDK 9,
>> > > 10, 11 or 12, I felt it was safer to explicitly throw rather than
>> having
>> > > the users experience some weird failure later that may be difficult to
>> > > debug.
>> > >
>> > > With only one minor exception (which we could easily fix) the Memory
>> > > component is the only place where Unsafe is used. However, we do use
>> > > reflection to gain internal access to a number of other classes
>> including
>> > > ByteBuffer, Cleaner, and Bits.
>> > >
>> > > Looking forward to hearing from you.
>> > >
>> > > Cheers,
>> > >
>> > > Lee.
>> > >
>> >
>>
>
Re: Help with Migration to OpenJDK 11
Posted by Gian Merlino <gi...@apache.org>.
Oracle and OpenJDK builds behave the same as each other now. The
differences are in terms of licensing, length of support, and release
cadence. IMO most Java devs should be using one of the third-party OpenJDK
distributions (e.g. Corretto, Zulu, AdoptOpenJDK) rather than vanilla
OpenJDK these days, because of the new OpenJDK policy that public updates
will cease for major versions after 6 months.
On Mon, Oct 14, 2019 at 11:07 AM leerho <le...@gmail.com> wrote:
> Thanks, Gian,
>
> Your references are helpful and I am studying them.
>
> I didn't see any references to OpenJDK vs Oracle's JDKs.
>
> Are there any differences in the way Unsafe calls (or other hidden classes)
> are handled by OpenJDK? Or is this a "don't care" as far as Druid is
> concerned? Are the version numbers between Oracle and OpenJDK always
> aligned?
>
>
> Lee.
>
>
>
> On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org> wrote:
>
> > You didn't address this message to me, but in Druid most of the work for
> > Java 9+ compatibility is mentioned in this master issue, which you might
> > find helpful: https://github.com/apache/incubator-druid/issues/5589.
> > Skimming the list, these might be particularly relevant to DataSketches:
> >
> > - https://github.com/apache/incubator-druid/pull/7487 (cleaner
> operations)
> > - https://github.com/apache/incubator-druid/pull/7466 (ByteBuffer unmap
> > operation)
> > - https://github.com/apache/incubator-druid/pull/7576 (Remove direct
> > references to Unsafe)
> >
> > Btw, it would be better, I think, to address emails to the list at large.
> > It encourages more people to participate. If Roman is a subscriber he
> would
> > get a copy anyway. If not, you could encourage him to subscribe.
> >
> > On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
> >
> > > Roman,
> > >
> > > I hope you are doing well.
> > >
> > > I really appreciate the contributions you made to our Memory component!
> > >
> > > Given that Druid has made its code base compatible with OpenJDK 11, we
> > > could use your help in what changes do we need to make to make that
> > happen
> > > for DataSketches. As I recall, our DataSketches library was not the
> only
> > > Druid dependency that took advantage of Unsafe. So I assume you have
> > been
> > > down this path :)
> > >
> > > Currently, I have a check in the static initializer in
> Memory/UnsafeUtil:
> > > parseJavaVersion(...) that checks the string returned from
> > > System.getProperty("java.version"). If that string does not contain
> > "1.8"
> > > or "8" it will throw an error. Given that we have not tested with JDK
> 9,
> > > 10, 11 or 12, I felt it was safer to explicitly throw rather than
> having
> > > the users experience some weird failure later that may be difficult to
> > > debug.
> > >
> > > With only one minor exception (which we could easily fix) the Memory
> > > component is the only place where Unsafe is used. However, we do use
> > > reflection to gain internal access to a number of other classes
> including
> > > ByteBuffer, Cleaner, and Bits.
> > >
> > > Looking forward to hearing from you.
> > >
> > > Cheers,
> > >
> > > Lee.
> > >
> >
>
Re: Help with Migration to OpenJDK 11
Posted by leerho <le...@gmail.com>.
Thanks, Gian,
Your references are helpful and I am studying them.
I didn't see any references to OpenJDK vs Oracle's JDKs.
Are there any differences in the way Unsafe calls (or other hidden classes)
are handled by OpenJDK? Or is this a "don't care" as far as Druid is
concerned? Are the version numbers between Oracle and OpenJDK always
aligned?
Lee.
On Fri, Oct 11, 2019 at 5:45 PM Gian Merlino <gi...@apache.org> wrote:
> You didn't address this message to me, but in Druid most of the work for
> Java 9+ compatibility is mentioned in this master issue, which you might
> find helpful: https://github.com/apache/incubator-druid/issues/5589.
> Skimming the list, these might be particularly relevant to DataSketches:
>
> - https://github.com/apache/incubator-druid/pull/7487 (cleaner operations)
> - https://github.com/apache/incubator-druid/pull/7466 (ByteBuffer unmap
> operation)
> - https://github.com/apache/incubator-druid/pull/7576 (Remove direct
> references to Unsafe)
>
> Btw, it would be better, I think, to address emails to the list at large.
> It encourages more people to participate. If Roman is a subscriber he would
> get a copy anyway. If not, you could encourage him to subscribe.
>
> On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
>
> > Roman,
> >
> > I hope you are doing well.
> >
> > I really appreciate the contributions you made to our Memory component!
> >
> > Given that Druid has made its code base compatible with OpenJDK 11, we
> > could use your help in what changes do we need to make to make that
> happen
> > for DataSketches. As I recall, our DataSketches library was not the only
> > Druid dependency that took advantage of Unsafe. So I assume you have
> been
> > down this path :)
> >
> > Currently, I have a check in the static initializer in Memory/UnsafeUtil:
> > parseJavaVersion(...) that checks the string returned from
> > System.getProperty("java.version"). If that string does not contain
> "1.8"
> > or "8" it will throw an error. Given that we have not tested with JDK 9,
> > 10, 11 or 12, I felt it was safer to explicitly throw rather than having
> > the users experience some weird failure later that may be difficult to
> > debug.
> >
> > With only one minor exception (which we could easily fix) the Memory
> > component is the only place where Unsafe is used. However, we do use
> > reflection to gain internal access to a number of other classes including
> > ByteBuffer, Cleaner, and Bits.
> >
> > Looking forward to hearing from you.
> >
> > Cheers,
> >
> > Lee.
> >
>
Re: Help with Migration to OpenJDK 11
Posted by Gian Merlino <gi...@apache.org>.
You didn't address this message to me, but in Druid most of the work for
Java 9+ compatibility is mentioned in this master issue, which you might
find helpful: https://github.com/apache/incubator-druid/issues/5589.
Skimming the list, these might be particularly relevant to DataSketches:
- https://github.com/apache/incubator-druid/pull/7487 (cleaner operations)
- https://github.com/apache/incubator-druid/pull/7466 (ByteBuffer unmap
operation)
- https://github.com/apache/incubator-druid/pull/7576 (Remove direct
references to Unsafe)
Btw, it would be better, I think, to address emails to the list at large.
It encourages more people to participate. If Roman is a subscriber he would
get a copy anyway. If not, you could encourage him to subscribe.
On Fri, Oct 11, 2019 at 3:13 PM leerho <le...@gmail.com> wrote:
> Roman,
>
> I hope you are doing well.
>
> I really appreciate the contributions you made to our Memory component!
>
> Given that Druid has made its code base compatible with OpenJDK 11, we
> could use your help in what changes do we need to make to make that happen
> for DataSketches. As I recall, our DataSketches library was not the only
> Druid dependency that took advantage of Unsafe. So I assume you have been
> down this path :)
>
> Currently, I have a check in the static initializer in Memory/UnsafeUtil:
> parseJavaVersion(...) that checks the string returned from
> System.getProperty("java.version"). If that string does not contain "1.8"
> or "8" it will throw an error. Given that we have not tested with JDK 9,
> 10, 11 or 12, I felt it was safer to explicitly throw rather than having
> the users experience some weird failure later that may be difficult to
> debug.
>
> With only one minor exception (which we could easily fix) the Memory
> component is the only place where Unsafe is used. However, we do use
> reflection to gain internal access to a number of other classes including
> ByteBuffer, Cleaner, and Bits.
>
> Looking forward to hearing from you.
>
> Cheers,
>
> Lee.
>