You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Suneet Saldanha <su...@imply.io> on 2020/11/09 22:40:55 UTC
Removing Druid support for JDK 8 and adding support for JDK 11
Hi Druids!
I've been thinking about what it would take for us to remove support for
JDK 8 in the 0.21 release, and officially add support for JDK 11.
I see that unit tests for jdk11 were added about 1+ year ago in Aug 2019 -
https://github.com/apache/druid/pull/8400
And integration tests were added ~ 8 months ago in Feb 2020 -
https://github.com/apache/druid/pull/9249
Since the jdk11 integration tests were added, I can't remember seeing any
tests fail for a jdk11 test, but pass for jdk 8 or vice versa. Given that
it's been about 1 year that these tests have been running, is there any
opposition to officially removing support for JDK 8 and adding support for
JDK 11?
There are a few big advantages I can think of for this:
1. Java 8 is end of life, so officially supporting java 11 will allow
users to run Druid with at least java 11 which is slotted for support till
September 2023 according to
https://www.oracle.com/java/technologies/java-se-support-roadmap.html
2. We can reduce our Travis usage by almost half, and lighten the load
on ASF's Travis account - today we are the second highest consumer of
travis resources by project https://infra-reports.apache.org/cistats/
[image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
The only limiting factor I can think of are full scale performance tests
that prove there is no major regression in moving from jdk 8 to jdk 11. Are
there any other issues that should be considered before adding support for
java 11?
Re: [E] Re: Removing Druid support for JDK 8 and adding support for
JDK 11
Posted by Gian Merlino <gi...@apache.org>.
Seconding (thirding?) the idea that keeping JDK 8 for integration with
Hadoop is important. Druid's Hadoop integration is built against Hadoop 2.x
and that version only supports JDK 8:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions. We
shouldn't drop JDK 8 support until we move to Hadoop 3.3 or later (or drop
the Hadoop integration completely).
The costs of keeping things stable on JDK 8 don't seem that high. The ones
Suneet raised were:
1) Java 8 is end of life.
2) We can reduce our Travis usage by almost half.
As Xavier pointed out, though, (1) isn't wholly fair since there are still
plenty of supported enterprise distributions for JDK 8. And ASF Infra
hasn't complained to us about (2).
To me, the benefits of dropping JDK 8 support don't outweigh the downsides.
I do think we should start telling people to deploy on JDK 11 by default,
though, and we should focus any future expansion of testing efforts (like
perf tests, etc) on JDK 11. I would whole-heartedly support this for the
next Druid release.
I also agree with Julian that supporting newer JDKs, if feasible, would be
good too.
On Thu, Nov 12, 2020 at 5:25 PM Xavier Léauté <xa...@confluent.io> wrote:
> I agree with Himanshu we should keep JDK 8 support for integrations if we
> feel those are important.
>
> As far as lifecycle goes, JDK 8 support will continue as part of several
> linux distributions.
> For instance, RedHat has taken on the role of publishing clean upstream
> builds for OpenJDK JDK8
> <https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries> and JDK11
> <https://github.com/AdoptOpenJDK/openjdk11-upstream-binaries> updates,
> with
> support until 2026 <https://access.redhat.com/articles/1299013>.
> This means JDK8 will probably not go away for some time with enterprise
> users.
>
> Before we can even deprecate 8 we should at least make 11 (or later) the
> default / preferred JDK version.
> What would help is to have some first hand accounts of people running
> production deployments with 11.
> I don't think we have a good sense of whether there are any significant
> performance differences between 8 and 11.
> We should rule out any regressions before we make it the default.
>
> I also agree with Julian that we should keep the ball rolling. The biggest
> hurdle was (hopefully) to get from 8 to 11,
> and the changes for 15 should hopefully be smaller, but it's important we
> keep the momentum going.
>
> Generally I would recommend we support the LTS JDK versions + the latest
> Other projects (e.g. Apache Kafka) does the same and builds 8+11+15.
> If we are concerned about build times, we can consider running integration
> tests for preferred version
> and reserve the full suite of JDKs for release candidates.
>
> Xavier
>
Re: [E] Re: Removing Druid support for JDK 8 and adding support for
JDK 11
Posted by Xavier Léauté <xa...@confluent.io>.
I agree with Himanshu we should keep JDK 8 support for integrations if we
feel those are important.
As far as lifecycle goes, JDK 8 support will continue as part of several
linux distributions.
For instance, RedHat has taken on the role of publishing clean upstream
builds for OpenJDK JDK8
<https://github.com/AdoptOpenJDK/openjdk8-upstream-binaries> and JDK11
<https://github.com/AdoptOpenJDK/openjdk11-upstream-binaries> updates, with
support until 2026 <https://access.redhat.com/articles/1299013>.
This means JDK8 will probably not go away for some time with enterprise
users.
Before we can even deprecate 8 we should at least make 11 (or later) the
default / preferred JDK version.
What would help is to have some first hand accounts of people running
production deployments with 11.
I don't think we have a good sense of whether there are any significant
performance differences between 8 and 11.
We should rule out any regressions before we make it the default.
I also agree with Julian that we should keep the ball rolling. The biggest
hurdle was (hopefully) to get from 8 to 11,
and the changes for 15 should hopefully be smaller, but it's important we
keep the momentum going.
Generally I would recommend we support the LTS JDK versions + the latest
Other projects (e.g. Apache Kafka) does the same and builds 8+11+15.
If we are concerned about build times, we can consider running integration
tests for preferred version
and reserve the full suite of JDKs for release candidates.
Xavier
Re: [E] Re: Removing Druid support for JDK 8 and adding support for
JDK 11
Posted by Himanshu <g....@gmail.com>.
I think data-sketches related integrations are fine as of now, as they get
tested in Druid's integration test builds which are running with JDK11 as
well as JDK8.
On Tue, Nov 10, 2020 at 8:58 AM leerho <le...@gmail.com> wrote:
> They are not using Memory to create their off-heap spaces, they are using
> ByteBuffer to create them. The issue with JDK11 is it has removed access
> to some classes like Cleaner that Memory used when creating off-heap
> memory. Even if they create their off-heap spaces with ByteBuffer, we can
> still use Memory to access them.
>
> On Tue, Nov 10, 2020 at 7:24 AM Will Lauer <wlauer@verizonmedia.com
> .invalid>
> wrote:
>
> > If Druid isn't creating off-heap memory, what are BufferAggregators for?
> > And what about the aggregation buffers that are preallocated? I thought
> > that was all off-heap memory.
> >
> > Will
> >
> > <http://www.verizonmedia.com>
> >
> > Will Lauer
> >
> > Senior Principal Architect, Audience & Advertising Reporting
> > Data Platforms & Systems Engineering
> >
> > M 508 561 6427
> > 1908 S. First St
> > Champaign, IL 61822
> >
> > <http://www.facebook.com/verizonmedia> <
> http://twitter.com/verizonmedia>
> > <https://www.linkedin.com/company/verizon-media/>
> > <http://www.instagram.com/verizonmedia>
> >
> >
> >
> > On Mon, Nov 9, 2020 at 8:27 PM leerho <le...@gmail.com> wrote:
> >
> > > Well, partly. As long as Druid doesn't actually create off-heap
> Memory,
> > it
> > > works OK with JDK11. Currently compiling with JDK11 will show errors,
> > but
> > > those errors can be ignored as long as they don't create off-heap
> Memory,
> > > which they do not do.
> > >
> > > This is an issue we will begin addressing soon, we just haven't had the
> > > resources to devote to it.
> > >
> > > Lee.
> > >
> > > On Mon, Nov 9, 2020 at 5:53 PM Will Lauer <wlauer@verizonmedia.com
> > > .invalid>
> > > wrote:
> > >
> > > > My understanding was that the Apache Datasketch project was still
> > having
> > > > issues with Java 11 and direct memory support in their sketch
> > > > implementation. That would impact Druid's use of datasketches, and
> > > possibly
> > > > Druid's use of direct memory as well. Does anyone know whether the
> > > > Datasketch issues have been resolved yet?
> > > >
> > > > Will
> > > >
> > > > <http://www.verizonmedia.com>
> > > >
> > > > Will Lauer
> > > >
> > > > Senior Principal Architect, Audience & Advertising Reporting
> > > > Data Platforms & Systems Engineering
> > > >
> > > > M 508 561 6427
> > > > 1908 S. First St
> > > > Champaign, IL 61822
> > > >
> > > > <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=-tGMOH_OrEiWdOQ86LBnuU2qMSfXIS5hP-9VpQieV6s&e=
> > > > <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=_IGT6oamCmA-1OIb2AoR35JkHzKVLlU2VB11khMrM3A&e=
> > > >
> > > > <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_verizon-2Dmedia_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=rT_p44rQm49vbvQbe1LVN7bNzawOkiGC2q19B0SI12k&e=
> > > >
> > > > <
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instagram.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=6UhWWfy4257r45sCNSepsm1KfzAA1vZI4U9O0A1tLUM&e=
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Nov 9, 2020 at 5:39 PM Himanshu <g....@gmail.com>
> wrote:
> > > >
> > > > > There are many projects that have Druid integration and use jars
> > > created
> > > > by
> > > > > Druid releases. Druid itself has hadoop based batch ingestion
> feature
> > > > where
> > > > > it ships its own jars over to hadoop and the code is executed in
> > hadoop
> > > > > cluster. For these integrations to continue working, it is
> essential
> > > that
> > > > > Druid jars continue to work with Java8 JVMs or that all other
> related
> > > > > projects also provide support for JDK-11 . I know this could be a
> > > > > chicken-or-the-egg problem, so just giving another justification to
> > why
> > > > it
> > > > > makes sense to continue supporting JDK8 .
> > > > >
> > > > > -- Himanshu
> > > > >
> > > > > On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org>
> wrote:
> > > > >
> > > > > > I have no objections to dropping support for JDK 8, but why only
> > > > support
> > > > > > JDK 11? What would it take to support JDK 11 and all newer JDKs?
> > > > > >
> > > > > > There's an analogy with starting and stopping a train. Once you
> > have
> > > > > > stopped a train (settled on a particular JDK release for a number
> > of
> > > > > years)
> > > > > > starting it is hard, because of static friction. So it's easier
> to
> > > just
> > > > > > keep the train rolling.
> > > > > >
> > > > > > In Calcite we have a policy to support all JDK versions, and it
> > means
> > > > > that
> > > > > > we have never become dug into one version. Supporting a new
> version
> > > > > becomes
> > > > > > a small incremental effort. Sometimes we have had to build shims
> to
> > > > > protect
> > > > > > us against changes, but it was never too onerous in terms of
> > > > maintenance
> > > > > or
> > > > > > performance. Obviously Druid is a very different project to
> > Calcite,
> > > > but
> > > > > > "keeping the train rolling" has worked well for us.
> > > > > >
> > > > > > Also, remember that Druid is a project, not a product. It's fine
> if
> > > > Druid
> > > > > > compiles, and builds, and passes the basic test suite, on JDK 12
> > even
> > > > > > though you declare that is not 'supported' in production.
> > > > > >
> > > > > > Julian
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha <
> > > > suneet.saldanha@imply.io
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Druids!
> > > > > > >
> > > > > > > I've been thinking about what it would take for us to remove
> > > support
> > > > > for
> > > > > > > JDK 8 in the 0.21 release, and officially add support for JDK
> 11.
> > > > > > >
> > > > > > > I see that unit tests for jdk11 were added about 1+ year ago in
> > Aug
> > > > > 2019
> > > > > > -
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_8400&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=P8_diPbQbYLZ-mpgzW0bj2b4np3XzqtxJNaAKPbJip0&e=
> > > > > > > And integration tests were added ~ 8 months ago in Feb 2020 -
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_9249&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=EQAynkpyeUQFocV4cb39geHJ87UXU8SZcDHXlhNL8uA&e=
> > > > > > >
> > > > > > > Since the jdk11 integration tests were added, I can't remember
> > > seeing
> > > > > any
> > > > > > > tests fail for a jdk11 test, but pass for jdk 8 or vice versa.
> > > Given
> > > > > that
> > > > > > > it's been about 1 year that these tests have been running, is
> > there
> > > > any
> > > > > > > opposition to officially removing support for JDK 8 and adding
> > > > support
> > > > > > for
> > > > > > > JDK 11?
> > > > > > >
> > > > > > > There are a few big advantages I can think of for this:
> > > > > > >
> > > > > > > 1. Java 8 is end of life, so officially supporting java 11
> > will
> > > > > allow
> > > > > > > users to run Druid with at least java 11 which is slotted
> for
> > > > > support
> > > > > > till
> > > > > > > September 2023 according to
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_java_technologies_java-2Dse-2Dsupport-2Droadmap.html&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=WsB-nZyjMgEFbJE0pz2LYHAEOen0781bgDwEbqkCaro&e=
> > > > > > > 2. We can reduce our Travis usage by almost half, and
> lighten
> > > the
> > > > > load
> > > > > > > on ASF's Travis account - today we are the second highest
> > > consumer
> > > > > of
> > > > > > > travis resources by project
> > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__infra-2Dreports.apache.org_cistats_&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=RSK7N5Mti_x-eG7WQxNIM5zMgfulpJey2eh8dLkQJJs&e=
> > > > > > > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
> > > > > > >
> > > > > > > The only limiting factor I can think of are full scale
> > performance
> > > > > tests
> > > > > > > that prove there is no major regression in moving from jdk 8 to
> > jdk
> > > > 11.
> > > > > > Are
> > > > > > > there any other issues that should be considered before adding
> > > > support
> > > > > > for
> > > > > > > java 11?
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [E] Re: Removing Druid support for JDK 8 and adding support for
JDK 11
Posted by leerho <le...@gmail.com>.
They are not using Memory to create their off-heap spaces, they are using
ByteBuffer to create them. The issue with JDK11 is it has removed access
to some classes like Cleaner that Memory used when creating off-heap
memory. Even if they create their off-heap spaces with ByteBuffer, we can
still use Memory to access them.
On Tue, Nov 10, 2020 at 7:24 AM Will Lauer <wl...@verizonmedia.com.invalid>
wrote:
> If Druid isn't creating off-heap memory, what are BufferAggregators for?
> And what about the aggregation buffers that are preallocated? I thought
> that was all off-heap memory.
>
> Will
>
> <http://www.verizonmedia.com>
>
> Will Lauer
>
> Senior Principal Architect, Audience & Advertising Reporting
> Data Platforms & Systems Engineering
>
> M 508 561 6427
> 1908 S. First St
> Champaign, IL 61822
>
> <http://www.facebook.com/verizonmedia> <http://twitter.com/verizonmedia>
> <https://www.linkedin.com/company/verizon-media/>
> <http://www.instagram.com/verizonmedia>
>
>
>
> On Mon, Nov 9, 2020 at 8:27 PM leerho <le...@gmail.com> wrote:
>
> > Well, partly. As long as Druid doesn't actually create off-heap Memory,
> it
> > works OK with JDK11. Currently compiling with JDK11 will show errors,
> but
> > those errors can be ignored as long as they don't create off-heap Memory,
> > which they do not do.
> >
> > This is an issue we will begin addressing soon, we just haven't had the
> > resources to devote to it.
> >
> > Lee.
> >
> > On Mon, Nov 9, 2020 at 5:53 PM Will Lauer <wlauer@verizonmedia.com
> > .invalid>
> > wrote:
> >
> > > My understanding was that the Apache Datasketch project was still
> having
> > > issues with Java 11 and direct memory support in their sketch
> > > implementation. That would impact Druid's use of datasketches, and
> > possibly
> > > Druid's use of direct memory as well. Does anyone know whether the
> > > Datasketch issues have been resolved yet?
> > >
> > > Will
> > >
> > > <http://www.verizonmedia.com>
> > >
> > > Will Lauer
> > >
> > > Senior Principal Architect, Audience & Advertising Reporting
> > > Data Platforms & Systems Engineering
> > >
> > > M 508 561 6427
> > > 1908 S. First St
> > > Champaign, IL 61822
> > >
> > > <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=-tGMOH_OrEiWdOQ86LBnuU2qMSfXIS5hP-9VpQieV6s&e=
> > > <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=_IGT6oamCmA-1OIb2AoR35JkHzKVLlU2VB11khMrM3A&e=
> > >
> > > <
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_verizon-2Dmedia_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=rT_p44rQm49vbvQbe1LVN7bNzawOkiGC2q19B0SI12k&e=
> > >
> > > <
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instagram.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=6UhWWfy4257r45sCNSepsm1KfzAA1vZI4U9O0A1tLUM&e=
> > >
> > >
> > >
> > >
> > > On Mon, Nov 9, 2020 at 5:39 PM Himanshu <g....@gmail.com> wrote:
> > >
> > > > There are many projects that have Druid integration and use jars
> > created
> > > by
> > > > Druid releases. Druid itself has hadoop based batch ingestion feature
> > > where
> > > > it ships its own jars over to hadoop and the code is executed in
> hadoop
> > > > cluster. For these integrations to continue working, it is essential
> > that
> > > > Druid jars continue to work with Java8 JVMs or that all other related
> > > > projects also provide support for JDK-11 . I know this could be a
> > > > chicken-or-the-egg problem, so just giving another justification to
> why
> > > it
> > > > makes sense to continue supporting JDK8 .
> > > >
> > > > -- Himanshu
> > > >
> > > > On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org> wrote:
> > > >
> > > > > I have no objections to dropping support for JDK 8, but why only
> > > support
> > > > > JDK 11? What would it take to support JDK 11 and all newer JDKs?
> > > > >
> > > > > There's an analogy with starting and stopping a train. Once you
> have
> > > > > stopped a train (settled on a particular JDK release for a number
> of
> > > > years)
> > > > > starting it is hard, because of static friction. So it's easier to
> > just
> > > > > keep the train rolling.
> > > > >
> > > > > In Calcite we have a policy to support all JDK versions, and it
> means
> > > > that
> > > > > we have never become dug into one version. Supporting a new version
> > > > becomes
> > > > > a small incremental effort. Sometimes we have had to build shims to
> > > > protect
> > > > > us against changes, but it was never too onerous in terms of
> > > maintenance
> > > > or
> > > > > performance. Obviously Druid is a very different project to
> Calcite,
> > > but
> > > > > "keeping the train rolling" has worked well for us.
> > > > >
> > > > > Also, remember that Druid is a project, not a product. It's fine if
> > > Druid
> > > > > compiles, and builds, and passes the basic test suite, on JDK 12
> even
> > > > > though you declare that is not 'supported' in production.
> > > > >
> > > > > Julian
> > > > >
> > > > >
> > > > > On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha <
> > > suneet.saldanha@imply.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Druids!
> > > > > >
> > > > > > I've been thinking about what it would take for us to remove
> > support
> > > > for
> > > > > > JDK 8 in the 0.21 release, and officially add support for JDK 11.
> > > > > >
> > > > > > I see that unit tests for jdk11 were added about 1+ year ago in
> Aug
> > > > 2019
> > > > > -
> > > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_8400&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=P8_diPbQbYLZ-mpgzW0bj2b4np3XzqtxJNaAKPbJip0&e=
> > > > > > And integration tests were added ~ 8 months ago in Feb 2020 -
> > > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_9249&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=EQAynkpyeUQFocV4cb39geHJ87UXU8SZcDHXlhNL8uA&e=
> > > > > >
> > > > > > Since the jdk11 integration tests were added, I can't remember
> > seeing
> > > > any
> > > > > > tests fail for a jdk11 test, but pass for jdk 8 or vice versa.
> > Given
> > > > that
> > > > > > it's been about 1 year that these tests have been running, is
> there
> > > any
> > > > > > opposition to officially removing support for JDK 8 and adding
> > > support
> > > > > for
> > > > > > JDK 11?
> > > > > >
> > > > > > There are a few big advantages I can think of for this:
> > > > > >
> > > > > > 1. Java 8 is end of life, so officially supporting java 11
> will
> > > > allow
> > > > > > users to run Druid with at least java 11 which is slotted for
> > > > support
> > > > > till
> > > > > > September 2023 according to
> > > > > >
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_java_technologies_java-2Dse-2Dsupport-2Droadmap.html&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=WsB-nZyjMgEFbJE0pz2LYHAEOen0781bgDwEbqkCaro&e=
> > > > > > 2. We can reduce our Travis usage by almost half, and lighten
> > the
> > > > load
> > > > > > on ASF's Travis account - today we are the second highest
> > consumer
> > > > of
> > > > > > travis resources by project
> > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__infra-2Dreports.apache.org_cistats_&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=RSK7N5Mti_x-eG7WQxNIM5zMgfulpJey2eh8dLkQJJs&e=
> > > > > > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
> > > > > >
> > > > > > The only limiting factor I can think of are full scale
> performance
> > > > tests
> > > > > > that prove there is no major regression in moving from jdk 8 to
> jdk
> > > 11.
> > > > > Are
> > > > > > there any other issues that should be considered before adding
> > > support
> > > > > for
> > > > > > java 11?
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [E] Re: Removing Druid support for JDK 8 and adding support for
JDK 11
Posted by Will Lauer <wl...@verizonmedia.com.INVALID>.
If Druid isn't creating off-heap memory, what are BufferAggregators for?
And what about the aggregation buffers that are preallocated? I thought
that was all off-heap memory.
Will
<http://www.verizonmedia.com>
Will Lauer
Senior Principal Architect, Audience & Advertising Reporting
Data Platforms & Systems Engineering
M 508 561 6427
1908 S. First St
Champaign, IL 61822
<http://www.facebook.com/verizonmedia> <http://twitter.com/verizonmedia>
<https://www.linkedin.com/company/verizon-media/>
<http://www.instagram.com/verizonmedia>
On Mon, Nov 9, 2020 at 8:27 PM leerho <le...@gmail.com> wrote:
> Well, partly. As long as Druid doesn't actually create off-heap Memory, it
> works OK with JDK11. Currently compiling with JDK11 will show errors, but
> those errors can be ignored as long as they don't create off-heap Memory,
> which they do not do.
>
> This is an issue we will begin addressing soon, we just haven't had the
> resources to devote to it.
>
> Lee.
>
> On Mon, Nov 9, 2020 at 5:53 PM Will Lauer <wlauer@verizonmedia.com
> .invalid>
> wrote:
>
> > My understanding was that the Apache Datasketch project was still having
> > issues with Java 11 and direct memory support in their sketch
> > implementation. That would impact Druid's use of datasketches, and
> possibly
> > Druid's use of direct memory as well. Does anyone know whether the
> > Datasketch issues have been resolved yet?
> >
> > Will
> >
> > <http://www.verizonmedia.com>
> >
> > Will Lauer
> >
> > Senior Principal Architect, Audience & Advertising Reporting
> > Data Platforms & Systems Engineering
> >
> > M 508 561 6427
> > 1908 S. First St
> > Champaign, IL 61822
> >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.facebook.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=-tGMOH_OrEiWdOQ86LBnuU2qMSfXIS5hP-9VpQieV6s&e=
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=_IGT6oamCmA-1OIb2AoR35JkHzKVLlU2VB11khMrM3A&e=
> >
> > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.linkedin.com_company_verizon-2Dmedia_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=rT_p44rQm49vbvQbe1LVN7bNzawOkiGC2q19B0SI12k&e=
> >
> > <
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.instagram.com_verizonmedia&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=30cwQjI_tqpOsvBcWhnY-ZftONuafrczjKp_ncVpW0M&s=6UhWWfy4257r45sCNSepsm1KfzAA1vZI4U9O0A1tLUM&e=
> >
> >
> >
> >
> > On Mon, Nov 9, 2020 at 5:39 PM Himanshu <g....@gmail.com> wrote:
> >
> > > There are many projects that have Druid integration and use jars
> created
> > by
> > > Druid releases. Druid itself has hadoop based batch ingestion feature
> > where
> > > it ships its own jars over to hadoop and the code is executed in hadoop
> > > cluster. For these integrations to continue working, it is essential
> that
> > > Druid jars continue to work with Java8 JVMs or that all other related
> > > projects also provide support for JDK-11 . I know this could be a
> > > chicken-or-the-egg problem, so just giving another justification to why
> > it
> > > makes sense to continue supporting JDK8 .
> > >
> > > -- Himanshu
> > >
> > > On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org> wrote:
> > >
> > > > I have no objections to dropping support for JDK 8, but why only
> > support
> > > > JDK 11? What would it take to support JDK 11 and all newer JDKs?
> > > >
> > > > There's an analogy with starting and stopping a train. Once you have
> > > > stopped a train (settled on a particular JDK release for a number of
> > > years)
> > > > starting it is hard, because of static friction. So it's easier to
> just
> > > > keep the train rolling.
> > > >
> > > > In Calcite we have a policy to support all JDK versions, and it means
> > > that
> > > > we have never become dug into one version. Supporting a new version
> > > becomes
> > > > a small incremental effort. Sometimes we have had to build shims to
> > > protect
> > > > us against changes, but it was never too onerous in terms of
> > maintenance
> > > or
> > > > performance. Obviously Druid is a very different project to Calcite,
> > but
> > > > "keeping the train rolling" has worked well for us.
> > > >
> > > > Also, remember that Druid is a project, not a product. It's fine if
> > Druid
> > > > compiles, and builds, and passes the basic test suite, on JDK 12 even
> > > > though you declare that is not 'supported' in production.
> > > >
> > > > Julian
> > > >
> > > >
> > > > On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha <
> > suneet.saldanha@imply.io
> > > >
> > > > wrote:
> > > >
> > > > > Hi Druids!
> > > > >
> > > > > I've been thinking about what it would take for us to remove
> support
> > > for
> > > > > JDK 8 in the 0.21 release, and officially add support for JDK 11.
> > > > >
> > > > > I see that unit tests for jdk11 were added about 1+ year ago in Aug
> > > 2019
> > > > -
> > > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_8400&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=P8_diPbQbYLZ-mpgzW0bj2b4np3XzqtxJNaAKPbJip0&e=
> > > > > And integration tests were added ~ 8 months ago in Feb 2020 -
> > > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_9249&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=EQAynkpyeUQFocV4cb39geHJ87UXU8SZcDHXlhNL8uA&e=
> > > > >
> > > > > Since the jdk11 integration tests were added, I can't remember
> seeing
> > > any
> > > > > tests fail for a jdk11 test, but pass for jdk 8 or vice versa.
> Given
> > > that
> > > > > it's been about 1 year that these tests have been running, is there
> > any
> > > > > opposition to officially removing support for JDK 8 and adding
> > support
> > > > for
> > > > > JDK 11?
> > > > >
> > > > > There are a few big advantages I can think of for this:
> > > > >
> > > > > 1. Java 8 is end of life, so officially supporting java 11 will
> > > allow
> > > > > users to run Druid with at least java 11 which is slotted for
> > > support
> > > > till
> > > > > September 2023 according to
> > > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_java_technologies_java-2Dse-2Dsupport-2Droadmap.html&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=WsB-nZyjMgEFbJE0pz2LYHAEOen0781bgDwEbqkCaro&e=
> > > > > 2. We can reduce our Travis usage by almost half, and lighten
> the
> > > load
> > > > > on ASF's Travis account - today we are the second highest
> consumer
> > > of
> > > > > travis resources by project
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__infra-2Dreports.apache.org_cistats_&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=RSK7N5Mti_x-eG7WQxNIM5zMgfulpJey2eh8dLkQJJs&e=
> > > > > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
> > > > >
> > > > > The only limiting factor I can think of are full scale performance
> > > tests
> > > > > that prove there is no major regression in moving from jdk 8 to jdk
> > 11.
> > > > Are
> > > > > there any other issues that should be considered before adding
> > support
> > > > for
> > > > > java 11?
> > > > >
> > > >
> > >
> >
>
Re: [E] Re: Removing Druid support for JDK 8 and adding support for
JDK 11
Posted by leerho <le...@gmail.com>.
Well, partly. As long as Druid doesn't actually create off-heap Memory, it
works OK with JDK11. Currently compiling with JDK11 will show errors, but
those errors can be ignored as long as they don't create off-heap Memory,
which they do not do.
This is an issue we will begin addressing soon, we just haven't had the
resources to devote to it.
Lee.
On Mon, Nov 9, 2020 at 5:53 PM Will Lauer <wl...@verizonmedia.com.invalid>
wrote:
> My understanding was that the Apache Datasketch project was still having
> issues with Java 11 and direct memory support in their sketch
> implementation. That would impact Druid's use of datasketches, and possibly
> Druid's use of direct memory as well. Does anyone know whether the
> Datasketch issues have been resolved yet?
>
> Will
>
> <http://www.verizonmedia.com>
>
> Will Lauer
>
> Senior Principal Architect, Audience & Advertising Reporting
> Data Platforms & Systems Engineering
>
> M 508 561 6427
> 1908 S. First St
> Champaign, IL 61822
>
> <http://www.facebook.com/verizonmedia> <http://twitter.com/verizonmedia>
> <https://www.linkedin.com/company/verizon-media/>
> <http://www.instagram.com/verizonmedia>
>
>
>
> On Mon, Nov 9, 2020 at 5:39 PM Himanshu <g....@gmail.com> wrote:
>
> > There are many projects that have Druid integration and use jars created
> by
> > Druid releases. Druid itself has hadoop based batch ingestion feature
> where
> > it ships its own jars over to hadoop and the code is executed in hadoop
> > cluster. For these integrations to continue working, it is essential that
> > Druid jars continue to work with Java8 JVMs or that all other related
> > projects also provide support for JDK-11 . I know this could be a
> > chicken-or-the-egg problem, so just giving another justification to why
> it
> > makes sense to continue supporting JDK8 .
> >
> > -- Himanshu
> >
> > On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org> wrote:
> >
> > > I have no objections to dropping support for JDK 8, but why only
> support
> > > JDK 11? What would it take to support JDK 11 and all newer JDKs?
> > >
> > > There's an analogy with starting and stopping a train. Once you have
> > > stopped a train (settled on a particular JDK release for a number of
> > years)
> > > starting it is hard, because of static friction. So it's easier to just
> > > keep the train rolling.
> > >
> > > In Calcite we have a policy to support all JDK versions, and it means
> > that
> > > we have never become dug into one version. Supporting a new version
> > becomes
> > > a small incremental effort. Sometimes we have had to build shims to
> > protect
> > > us against changes, but it was never too onerous in terms of
> maintenance
> > or
> > > performance. Obviously Druid is a very different project to Calcite,
> but
> > > "keeping the train rolling" has worked well for us.
> > >
> > > Also, remember that Druid is a project, not a product. It's fine if
> Druid
> > > compiles, and builds, and passes the basic test suite, on JDK 12 even
> > > though you declare that is not 'supported' in production.
> > >
> > > Julian
> > >
> > >
> > > On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha <
> suneet.saldanha@imply.io
> > >
> > > wrote:
> > >
> > > > Hi Druids!
> > > >
> > > > I've been thinking about what it would take for us to remove support
> > for
> > > > JDK 8 in the 0.21 release, and officially add support for JDK 11.
> > > >
> > > > I see that unit tests for jdk11 were added about 1+ year ago in Aug
> > 2019
> > > -
> > > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_8400&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=P8_diPbQbYLZ-mpgzW0bj2b4np3XzqtxJNaAKPbJip0&e=
> > > > And integration tests were added ~ 8 months ago in Feb 2020 -
> > > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_9249&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=EQAynkpyeUQFocV4cb39geHJ87UXU8SZcDHXlhNL8uA&e=
> > > >
> > > > Since the jdk11 integration tests were added, I can't remember seeing
> > any
> > > > tests fail for a jdk11 test, but pass for jdk 8 or vice versa. Given
> > that
> > > > it's been about 1 year that these tests have been running, is there
> any
> > > > opposition to officially removing support for JDK 8 and adding
> support
> > > for
> > > > JDK 11?
> > > >
> > > > There are a few big advantages I can think of for this:
> > > >
> > > > 1. Java 8 is end of life, so officially supporting java 11 will
> > allow
> > > > users to run Druid with at least java 11 which is slotted for
> > support
> > > till
> > > > September 2023 according to
> > > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_java_technologies_java-2Dse-2Dsupport-2Droadmap.html&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=WsB-nZyjMgEFbJE0pz2LYHAEOen0781bgDwEbqkCaro&e=
> > > > 2. We can reduce our Travis usage by almost half, and lighten the
> > load
> > > > on ASF's Travis account - today we are the second highest consumer
> > of
> > > > travis resources by project
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__infra-2Dreports.apache.org_cistats_&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=RSK7N5Mti_x-eG7WQxNIM5zMgfulpJey2eh8dLkQJJs&e=
> > > > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
> > > >
> > > > The only limiting factor I can think of are full scale performance
> > tests
> > > > that prove there is no major regression in moving from jdk 8 to jdk
> 11.
> > > Are
> > > > there any other issues that should be considered before adding
> support
> > > for
> > > > java 11?
> > > >
> > >
> >
>
Re: [E] Re: Removing Druid support for JDK 8 and adding support for
JDK 11
Posted by Will Lauer <wl...@verizonmedia.com.INVALID>.
My understanding was that the Apache Datasketch project was still having
issues with Java 11 and direct memory support in their sketch
implementation. That would impact Druid's use of datasketches, and possibly
Druid's use of direct memory as well. Does anyone know whether the
Datasketch issues have been resolved yet?
Will
<http://www.verizonmedia.com>
Will Lauer
Senior Principal Architect, Audience & Advertising Reporting
Data Platforms & Systems Engineering
M 508 561 6427
1908 S. First St
Champaign, IL 61822
<http://www.facebook.com/verizonmedia> <http://twitter.com/verizonmedia>
<https://www.linkedin.com/company/verizon-media/>
<http://www.instagram.com/verizonmedia>
On Mon, Nov 9, 2020 at 5:39 PM Himanshu <g....@gmail.com> wrote:
> There are many projects that have Druid integration and use jars created by
> Druid releases. Druid itself has hadoop based batch ingestion feature where
> it ships its own jars over to hadoop and the code is executed in hadoop
> cluster. For these integrations to continue working, it is essential that
> Druid jars continue to work with Java8 JVMs or that all other related
> projects also provide support for JDK-11 . I know this could be a
> chicken-or-the-egg problem, so just giving another justification to why it
> makes sense to continue supporting JDK8 .
>
> -- Himanshu
>
> On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org> wrote:
>
> > I have no objections to dropping support for JDK 8, but why only support
> > JDK 11? What would it take to support JDK 11 and all newer JDKs?
> >
> > There's an analogy with starting and stopping a train. Once you have
> > stopped a train (settled on a particular JDK release for a number of
> years)
> > starting it is hard, because of static friction. So it's easier to just
> > keep the train rolling.
> >
> > In Calcite we have a policy to support all JDK versions, and it means
> that
> > we have never become dug into one version. Supporting a new version
> becomes
> > a small incremental effort. Sometimes we have had to build shims to
> protect
> > us against changes, but it was never too onerous in terms of maintenance
> or
> > performance. Obviously Druid is a very different project to Calcite, but
> > "keeping the train rolling" has worked well for us.
> >
> > Also, remember that Druid is a project, not a product. It's fine if Druid
> > compiles, and builds, and passes the basic test suite, on JDK 12 even
> > though you declare that is not 'supported' in production.
> >
> > Julian
> >
> >
> > On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha <suneet.saldanha@imply.io
> >
> > wrote:
> >
> > > Hi Druids!
> > >
> > > I've been thinking about what it would take for us to remove support
> for
> > > JDK 8 in the 0.21 release, and officially add support for JDK 11.
> > >
> > > I see that unit tests for jdk11 were added about 1+ year ago in Aug
> 2019
> > -
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_8400&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=P8_diPbQbYLZ-mpgzW0bj2b4np3XzqtxJNaAKPbJip0&e=
> > > And integration tests were added ~ 8 months ago in Feb 2020 -
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_9249&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=EQAynkpyeUQFocV4cb39geHJ87UXU8SZcDHXlhNL8uA&e=
> > >
> > > Since the jdk11 integration tests were added, I can't remember seeing
> any
> > > tests fail for a jdk11 test, but pass for jdk 8 or vice versa. Given
> that
> > > it's been about 1 year that these tests have been running, is there any
> > > opposition to officially removing support for JDK 8 and adding support
> > for
> > > JDK 11?
> > >
> > > There are a few big advantages I can think of for this:
> > >
> > > 1. Java 8 is end of life, so officially supporting java 11 will
> allow
> > > users to run Druid with at least java 11 which is slotted for
> support
> > till
> > > September 2023 according to
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.oracle.com_java_technologies_java-2Dse-2Dsupport-2Droadmap.html&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=WsB-nZyjMgEFbJE0pz2LYHAEOen0781bgDwEbqkCaro&e=
> > > 2. We can reduce our Travis usage by almost half, and lighten the
> load
> > > on ASF's Travis account - today we are the second highest consumer
> of
> > > travis resources by project
> https://urldefense.proofpoint.com/v2/url?u=https-3A__infra-2Dreports.apache.org_cistats_&d=DwIBaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=ULseRJUsY5gTBgFA9-BUxg&m=oTTYxCRAYdLG9vX5nRES2_H05PLf_g6-6O07b4RijaI&s=RSK7N5Mti_x-eG7WQxNIM5zMgfulpJey2eh8dLkQJJs&e=
> > > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
> > >
> > > The only limiting factor I can think of are full scale performance
> tests
> > > that prove there is no major regression in moving from jdk 8 to jdk 11.
> > Are
> > > there any other issues that should be considered before adding support
> > for
> > > java 11?
> > >
> >
>
Re: Removing Druid support for JDK 8 and adding support for JDK 11
Posted by Himanshu <g....@gmail.com>.
There are many projects that have Druid integration and use jars created by
Druid releases. Druid itself has hadoop based batch ingestion feature where
it ships its own jars over to hadoop and the code is executed in hadoop
cluster. For these integrations to continue working, it is essential that
Druid jars continue to work with Java8 JVMs or that all other related
projects also provide support for JDK-11 . I know this could be a
chicken-or-the-egg problem, so just giving another justification to why it
makes sense to continue supporting JDK8 .
-- Himanshu
On Mon, Nov 9, 2020 at 3:25 PM Julian Hyde <jh...@apache.org> wrote:
> I have no objections to dropping support for JDK 8, but why only support
> JDK 11? What would it take to support JDK 11 and all newer JDKs?
>
> There's an analogy with starting and stopping a train. Once you have
> stopped a train (settled on a particular JDK release for a number of years)
> starting it is hard, because of static friction. So it's easier to just
> keep the train rolling.
>
> In Calcite we have a policy to support all JDK versions, and it means that
> we have never become dug into one version. Supporting a new version becomes
> a small incremental effort. Sometimes we have had to build shims to protect
> us against changes, but it was never too onerous in terms of maintenance or
> performance. Obviously Druid is a very different project to Calcite, but
> "keeping the train rolling" has worked well for us.
>
> Also, remember that Druid is a project, not a product. It's fine if Druid
> compiles, and builds, and passes the basic test suite, on JDK 12 even
> though you declare that is not 'supported' in production.
>
> Julian
>
>
> On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha <su...@imply.io>
> wrote:
>
> > Hi Druids!
> >
> > I've been thinking about what it would take for us to remove support for
> > JDK 8 in the 0.21 release, and officially add support for JDK 11.
> >
> > I see that unit tests for jdk11 were added about 1+ year ago in Aug 2019
> -
> > https://github.com/apache/druid/pull/8400
> > And integration tests were added ~ 8 months ago in Feb 2020 -
> > https://github.com/apache/druid/pull/9249
> >
> > Since the jdk11 integration tests were added, I can't remember seeing any
> > tests fail for a jdk11 test, but pass for jdk 8 or vice versa. Given that
> > it's been about 1 year that these tests have been running, is there any
> > opposition to officially removing support for JDK 8 and adding support
> for
> > JDK 11?
> >
> > There are a few big advantages I can think of for this:
> >
> > 1. Java 8 is end of life, so officially supporting java 11 will allow
> > users to run Druid with at least java 11 which is slotted for support
> till
> > September 2023 according to
> > https://www.oracle.com/java/technologies/java-se-support-roadmap.html
> > 2. We can reduce our Travis usage by almost half, and lighten the load
> > on ASF's Travis account - today we are the second highest consumer of
> > travis resources by project https://infra-reports.apache.org/cistats/
> > [image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
> >
> > The only limiting factor I can think of are full scale performance tests
> > that prove there is no major regression in moving from jdk 8 to jdk 11.
> Are
> > there any other issues that should be considered before adding support
> for
> > java 11?
> >
>
Re: Removing Druid support for JDK 8 and adding support for JDK 11
Posted by Julian Hyde <jh...@apache.org>.
I have no objections to dropping support for JDK 8, but why only support
JDK 11? What would it take to support JDK 11 and all newer JDKs?
There's an analogy with starting and stopping a train. Once you have
stopped a train (settled on a particular JDK release for a number of years)
starting it is hard, because of static friction. So it's easier to just
keep the train rolling.
In Calcite we have a policy to support all JDK versions, and it means that
we have never become dug into one version. Supporting a new version becomes
a small incremental effort. Sometimes we have had to build shims to protect
us against changes, but it was never too onerous in terms of maintenance or
performance. Obviously Druid is a very different project to Calcite, but
"keeping the train rolling" has worked well for us.
Also, remember that Druid is a project, not a product. It's fine if Druid
compiles, and builds, and passes the basic test suite, on JDK 12 even
though you declare that is not 'supported' in production.
Julian
On Mon, Nov 9, 2020 at 2:41 PM Suneet Saldanha <su...@imply.io>
wrote:
> Hi Druids!
>
> I've been thinking about what it would take for us to remove support for
> JDK 8 in the 0.21 release, and officially add support for JDK 11.
>
> I see that unit tests for jdk11 were added about 1+ year ago in Aug 2019 -
> https://github.com/apache/druid/pull/8400
> And integration tests were added ~ 8 months ago in Feb 2020 -
> https://github.com/apache/druid/pull/9249
>
> Since the jdk11 integration tests were added, I can't remember seeing any
> tests fail for a jdk11 test, but pass for jdk 8 or vice versa. Given that
> it's been about 1 year that these tests have been running, is there any
> opposition to officially removing support for JDK 8 and adding support for
> JDK 11?
>
> There are a few big advantages I can think of for this:
>
> 1. Java 8 is end of life, so officially supporting java 11 will allow
> users to run Druid with at least java 11 which is slotted for support till
> September 2023 according to
> https://www.oracle.com/java/technologies/java-se-support-roadmap.html
> 2. We can reduce our Travis usage by almost half, and lighten the load
> on ASF's Travis account - today we are the second highest consumer of
> travis resources by project https://infra-reports.apache.org/cistats/
> [image: Screen Shot 2020-11-09 at 2.34.45 PM.png]
>
> The only limiting factor I can think of are full scale performance tests
> that prove there is no major regression in moving from jdk 8 to jdk 11. Are
> there any other issues that should be considered before adding support for
> java 11?
>