You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Ayush Saxena <ay...@gmail.com> on 2023/05/31 10:38:50 UTC
Move to JDK-11
Hi Everyone,
Want to pull in the attention of folks towards moving to JDK-11 compile
time support in Hive. There was a ticket in the past [1] which talks about
it and If I could decode it right, it was blocked because the Hadoop
version used by Hive didn't had JDK-11 runtime support, But with [2] in we
have upgraded the Hadoop version, so that problem is sorted out. I couldn't
even see any unresolved tickets in the blocked state either.
I quickly tried* a mvn clean install -DskipTests -Piceberg -Pitests
-Dmaven.javadoc.skip=true
And no surprises it failed with some weird exceptions towards the end. But
I think that should be solvable.
So, Questions?
- What do folks think about this? Should we put in some effort towards
JDK-11
- Should we support both JDK-11 & JDK-8?
- Ditch JDK-11 and directly shoot for JDK-17?
Let me know your thoughts, In case anyone has some experience in this area
and have tried something in the context, feel free to share or may be if
someone has any potential action plan or so
-Ayush
[1] https://issues.apache.org/jira/browse/HIVE-22415
[2] https://issues.apache.org/jira/browse/HIVE-24484
* changed the maven.compiler.source & maven.compiler.target to 11
Re: Move to JDK-11
Posted by Attila Turoczy <at...@cloudera.com.INVALID>.
Returning to this topic, I kindly request those who would like to advocate
for the continued support of JDK8 to please share their reasoning and
insights with us. Your input and perspective are greatly appreciated!
Thank you.
-Attila
On Fri, Jun 2, 2023 at 12:43 PM Attila Turoczy <at...@cloudera.com>
wrote:
> Hi All,
>
> I know my opinion might not be the most popular, but I advocate for using*
> JDK 17*. Here's why:
>
> Let's consider a scenario where a customer wants to use the latest version
> of Apache Hive. They would typically install it locally or on a small
> cluster. In 2023, is it realistic to assume that this customer won't be
> able to install JDK 17 on their cluster? Even in large enterprises, it
> should be feasible to install an LTS JDK, especially considering the
> widespread adoption of cloud computing. Sungwoo Park's measurements also
> support this recommendation to go with JDK 17. It outperforms JDK 11 by 8%
> in terms of runtime speed, and JDK 11 itself is 10+% faster than JDK 8.
> This is a significant value proposition. Who would be the customer that
> says, "I don't want faster query execution! I'd rather use JDK 8 and pay
> more for cloud or data center resources instead of using JDK 17!" It
> doesn't make sense to me.
>
> The tech industry has been evolving at an incredible pace, with
> improvements in serialization, IPC mechanisms, and parallelized frameworks
> since the release of JDK 8 ten years ago. We should leverage these
> advancements! Couple years ago, we invested a lot to improve 1-2% of the
> execution. We prayed for 3 gods, sacrificed 2 ships and traveled around the
> world to make it happen. :-) Now, the JDK itself provides a substantial
> amount of improvement. So, why would we resist progress just because there
> are a few lazy or conservative admins who don't want to spend two minutes
> installing a JDK?
>
> A platform needs to be modern and incorporate the latest technologies to
> attract developers and users. I understand that some may prefer to stay
> with JDK 8 as it seems like the safest position, but I believe in taking
> bold bets to achieve big wins. Even if we decide to stick with JDK 8, I
> would still be happy since we are moving forward and not dwelling on a JDK
> that is a decade old. Personally, I think focusing on one thing that brings
> more value to us and our users is the idealistic path forward.
>
> -Attila
>
>
>
>
>
>
>
>
>
> On Thu, Jun 1, 2023 at 11:23 AM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
>> Hey everyone,
>>
>> If we claim that Hive supports a certain JDK then we should compile and
>> run
>> tests with it.
>>
>> The more JDKs we can support the better for everyone but this comes at a
>> cost (resources mostly). We should have a precommit run for every
>> supported
>> JDK (frequency to be determined once per day/week) that compiles and run
>> all tests.
>>
>> From my perspective, I would be pretty happy if we could cover the two
>> edge
>> LTS releases at every point in time.
>>
>> Then we have to decide also which JDK shall we use for the pull requests
>> and local dev environment. I think it makes sense to use the latest.
>> People
>> like working on modern stuff and also it makes sense that newer releases
>> will also use newer versions. It would be pretty awkward if someone wants
>> to use the latest Hive version and it turns out that it can only run on
>> JDK8.
>>
>> Best,
>> Stamatis
>>
>> On Thu, Jun 1, 2023, 3:42 AM Sungwoo Park <gl...@gmail.com> wrote:
>>
>> > Hi, everyone.
>> >
>> > I have not tested the master branch with Java 11/17 yet, but I would
>> like
>> > to share my experience with testing a fork of branch-3.1 with Java 11/17
>> > (as part of developing Hive-MR3), in case that it can be useful for the
>> > discussion. I merged the patches listed in [1] HIVE-22415 and updated
>> the
>> > Maven configuration for Java 11.
>> >
>> > 1. Building Hive was fine and I was able to run it with Java 11 as well
>> as
>> > Java 17. So, it seems that the work reported in [1] is indeed complete
>> for
>> > upgrading to Java 11 (and Java 17) and getting Hive to work.
>> >
>> > 2. However, there was a problem with running tests, so this can be
>> > additional work for upgrading to Java 11.
>> >
>> > 3. For performance, Java 17 gives about 8 percent of (free) performance
>> > improvement. When tested with 10TB TPC-DS, Java 8 takes 8074 seconds,
>> > whereas Java 17 takes 7415 seconds. Considering the maturity of Hive, I
>> > think this is not a small improvement because almost every query gets
>> some
>> > speedup.
>> >
>> > Thanks,
>> >
>> > --- Sungwoo
>> >
>> > [1] https://issues.apache.org/jira/browse/HIVE-22415
>> >
>> >
>> > On Thu, Jun 1, 2023 at 3:53 AM Sai Hemanth Gantasala
>> > <sa...@cloudera.com.invalid> wrote:
>> >
>> > > Hi All,
>> > >
>> > > I would strongly advocate keeping support for JDK8.
>> > > Between JDK11 and JDK17, Depending on the amount of effort on the
>> upgrade
>> > > I'm inclined towards JDK17 (JDK21 LTS will be released in Sep 2023).
>> > >
>> > > Thanks,
>> > > Sai.
>> > >
>> > > On Wed, May 31, 2023 at 5:39 AM László Bodor <
>> bodorlaszlo0202@gmail.com>
>> > > wrote:
>> > >
>> > > > *Hi!*
>> > > >
>> > > >
>> > > > *Should we support both JDK-11 & JDK-8?*
>> > > > IMO absolutely yes, let's not break up with JDK-8: according to its
>> > > > lifecycle, it's going to stay with us for a long time.
>> > > >
>> > > > I believe
>> > > > a) we should be able to compile on JDK8, JDK11, and JDK17 (github
>> > actions
>> > > > can cover this conveniently in precommit time, like tez
>> > > > <
>> https://github.com/apache/tez/blob/master/.github/workflows/build.yml
>> > >)
>> > > > b) the release artifacts should be compatible with JDK8 as long as
>> it
>> > is
>> > > > with us.
>> > > >
>> > > > Regards,
>> > > > Laszlo Bodor
>> > > >
>> > > >
>> > > > Butao Zhang <bu...@163.com> ezt írta (időpont: 2023. máj.
>> 31.,
>> > > Sze,
>> > > > 14:33):
>> > > >
>> > > > > Thanks Ayush for driving this! Good to know that Hive is getting
>> > ready
>> > > > for
>> > > > > newer JDK.
>> > > > > From my opinon, if we have more community energy to put into it,
>> we
>> > can
>> > > > > support both JDK-11 and JDK-17 like Spark[1]. If we have to make
>> a
>> > > > choice
>> > > > > between a JDK-11 and JDK-17, i would like to choose the relatively
>> > new
>> > > > > version JDK-17, meanwhile, we should maintain compatibility with
>> > jdk8,
>> > > as
>> > > > > JDK-8 is still widely used in most big data platforms.
>> > > > >
>> > > > >
>> > > > > Thanks,
>> > > > > Butao Zhang
>> > > > >
>> > > > >
>> > > > > [1]https://issues.apache.org/jira/browse/SPARK-33772
>> > > > > ---- Replied Message ----
>> > > > > | From | Ayush Saxena<ay...@gmail.com> |
>> > > > > | Date | 5/31/2023 18:39 |
>> > > > > | To | dev<de...@hive.apache.org> |
>> > > > > | Subject | Move to JDK-11 |
>> > > > > Hi Everyone,
>> > > > > Want to pull in the attention of folks towards moving to JDK-11
>> > compile
>> > > > > time support in Hive. There was a ticket in the past [1] which
>> talks
>> > > > about
>> > > > > it and If I could decode it right, it was blocked because the
>> Hadoop
>> > > > > version used by Hive didn't had JDK-11 runtime support, But with
>> [2]
>> > in
>> > > > we
>> > > > > have upgraded the Hadoop version, so that problem is sorted out. I
>> > > > couldn't
>> > > > > even see any unresolved tickets in the blocked state either.
>> > > > >
>> > > > > I quickly tried* a mvn clean install -DskipTests -Piceberg
>> -Pitests
>> > > > > -Dmaven.javadoc.skip=true
>> > > > >
>> > > > > And no surprises it failed with some weird exceptions towards the
>> > end.
>> > > > But
>> > > > > I think that should be solvable.
>> > > > >
>> > > > > So, Questions?
>> > > > >
>> > > > > - What do folks think about this? Should we put in some effort
>> > towards
>> > > > > JDK-11
>> > > > > - Should we support both JDK-11 & JDK-8?
>> > > > > - Ditch JDK-11 and directly shoot for JDK-17?
>> > > > >
>> > > > > Let me know your thoughts, In case anyone has some experience in
>> this
>> > > > area
>> > > > > and have tried something in the context, feel free to share or
>> may be
>> > > if
>> > > > > someone has any potential action plan or so
>> > > > >
>> > > > > -Ayush
>> > > > >
>> > > > > [1] https://issues.apache.org/jira/browse/HIVE-22415
>> > > > > [2] https://issues.apache.org/jira/browse/HIVE-24484
>> > > > >
>> > > > > * changed the maven.compiler.source & maven.compiler.target to 11
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Move to JDK-11
Posted by Attila Turoczy <at...@cloudera.com.INVALID>.
Hi All,
I know my opinion might not be the most popular, but I advocate for using*
JDK 17*. Here's why:
Let's consider a scenario where a customer wants to use the latest version
of Apache Hive. They would typically install it locally or on a small
cluster. In 2023, is it realistic to assume that this customer won't be
able to install JDK 17 on their cluster? Even in large enterprises, it
should be feasible to install an LTS JDK, especially considering the
widespread adoption of cloud computing. Sungwoo Park's measurements also
support this recommendation to go with JDK 17. It outperforms JDK 11 by 8%
in terms of runtime speed, and JDK 11 itself is 10+% faster than JDK 8.
This is a significant value proposition. Who would be the customer that
says, "I don't want faster query execution! I'd rather use JDK 8 and pay
more for cloud or data center resources instead of using JDK 17!" It
doesn't make sense to me.
The tech industry has been evolving at an incredible pace, with
improvements in serialization, IPC mechanisms, and parallelized frameworks
since the release of JDK 8 ten years ago. We should leverage these
advancements! Couple years ago, we invested a lot to improve 1-2% of the
execution. We prayed for 3 gods, sacrificed 2 ships and traveled around the
world to make it happen. :-) Now, the JDK itself provides a substantial
amount of improvement. So, why would we resist progress just because there
are a few lazy or conservative admins who don't want to spend two minutes
installing a JDK?
A platform needs to be modern and incorporate the latest technologies to
attract developers and users. I understand that some may prefer to stay
with JDK 8 as it seems like the safest position, but I believe in taking
bold bets to achieve big wins. Even if we decide to stick with JDK 8, I
would still be happy since we are moving forward and not dwelling on a JDK
that is a decade old. Personally, I think focusing on one thing that brings
more value to us and our users is the idealistic path forward.
-Attila
On Thu, Jun 1, 2023 at 11:23 AM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Hey everyone,
>
> If we claim that Hive supports a certain JDK then we should compile and run
> tests with it.
>
> The more JDKs we can support the better for everyone but this comes at a
> cost (resources mostly). We should have a precommit run for every supported
> JDK (frequency to be determined once per day/week) that compiles and run
> all tests.
>
> From my perspective, I would be pretty happy if we could cover the two edge
> LTS releases at every point in time.
>
> Then we have to decide also which JDK shall we use for the pull requests
> and local dev environment. I think it makes sense to use the latest. People
> like working on modern stuff and also it makes sense that newer releases
> will also use newer versions. It would be pretty awkward if someone wants
> to use the latest Hive version and it turns out that it can only run on
> JDK8.
>
> Best,
> Stamatis
>
> On Thu, Jun 1, 2023, 3:42 AM Sungwoo Park <gl...@gmail.com> wrote:
>
> > Hi, everyone.
> >
> > I have not tested the master branch with Java 11/17 yet, but I would like
> > to share my experience with testing a fork of branch-3.1 with Java 11/17
> > (as part of developing Hive-MR3), in case that it can be useful for the
> > discussion. I merged the patches listed in [1] HIVE-22415 and updated the
> > Maven configuration for Java 11.
> >
> > 1. Building Hive was fine and I was able to run it with Java 11 as well
> as
> > Java 17. So, it seems that the work reported in [1] is indeed complete
> for
> > upgrading to Java 11 (and Java 17) and getting Hive to work.
> >
> > 2. However, there was a problem with running tests, so this can be
> > additional work for upgrading to Java 11.
> >
> > 3. For performance, Java 17 gives about 8 percent of (free) performance
> > improvement. When tested with 10TB TPC-DS, Java 8 takes 8074 seconds,
> > whereas Java 17 takes 7415 seconds. Considering the maturity of Hive, I
> > think this is not a small improvement because almost every query gets
> some
> > speedup.
> >
> > Thanks,
> >
> > --- Sungwoo
> >
> > [1] https://issues.apache.org/jira/browse/HIVE-22415
> >
> >
> > On Thu, Jun 1, 2023 at 3:53 AM Sai Hemanth Gantasala
> > <sa...@cloudera.com.invalid> wrote:
> >
> > > Hi All,
> > >
> > > I would strongly advocate keeping support for JDK8.
> > > Between JDK11 and JDK17, Depending on the amount of effort on the
> upgrade
> > > I'm inclined towards JDK17 (JDK21 LTS will be released in Sep 2023).
> > >
> > > Thanks,
> > > Sai.
> > >
> > > On Wed, May 31, 2023 at 5:39 AM László Bodor <
> bodorlaszlo0202@gmail.com>
> > > wrote:
> > >
> > > > *Hi!*
> > > >
> > > >
> > > > *Should we support both JDK-11 & JDK-8?*
> > > > IMO absolutely yes, let's not break up with JDK-8: according to its
> > > > lifecycle, it's going to stay with us for a long time.
> > > >
> > > > I believe
> > > > a) we should be able to compile on JDK8, JDK11, and JDK17 (github
> > actions
> > > > can cover this conveniently in precommit time, like tez
> > > > <
> https://github.com/apache/tez/blob/master/.github/workflows/build.yml
> > >)
> > > > b) the release artifacts should be compatible with JDK8 as long as it
> > is
> > > > with us.
> > > >
> > > > Regards,
> > > > Laszlo Bodor
> > > >
> > > >
> > > > Butao Zhang <bu...@163.com> ezt írta (időpont: 2023. máj. 31.,
> > > Sze,
> > > > 14:33):
> > > >
> > > > > Thanks Ayush for driving this! Good to know that Hive is getting
> > ready
> > > > for
> > > > > newer JDK.
> > > > > From my opinon, if we have more community energy to put into it, we
> > can
> > > > > support both JDK-11 and JDK-17 like Spark[1]. If we have to make a
> > > > choice
> > > > > between a JDK-11 and JDK-17, i would like to choose the relatively
> > new
> > > > > version JDK-17, meanwhile, we should maintain compatibility with
> > jdk8,
> > > as
> > > > > JDK-8 is still widely used in most big data platforms.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Butao Zhang
> > > > >
> > > > >
> > > > > [1]https://issues.apache.org/jira/browse/SPARK-33772
> > > > > ---- Replied Message ----
> > > > > | From | Ayush Saxena<ay...@gmail.com> |
> > > > > | Date | 5/31/2023 18:39 |
> > > > > | To | dev<de...@hive.apache.org> |
> > > > > | Subject | Move to JDK-11 |
> > > > > Hi Everyone,
> > > > > Want to pull in the attention of folks towards moving to JDK-11
> > compile
> > > > > time support in Hive. There was a ticket in the past [1] which
> talks
> > > > about
> > > > > it and If I could decode it right, it was blocked because the
> Hadoop
> > > > > version used by Hive didn't had JDK-11 runtime support, But with
> [2]
> > in
> > > > we
> > > > > have upgraded the Hadoop version, so that problem is sorted out. I
> > > > couldn't
> > > > > even see any unresolved tickets in the blocked state either.
> > > > >
> > > > > I quickly tried* a mvn clean install -DskipTests -Piceberg
> -Pitests
> > > > > -Dmaven.javadoc.skip=true
> > > > >
> > > > > And no surprises it failed with some weird exceptions towards the
> > end.
> > > > But
> > > > > I think that should be solvable.
> > > > >
> > > > > So, Questions?
> > > > >
> > > > > - What do folks think about this? Should we put in some effort
> > towards
> > > > > JDK-11
> > > > > - Should we support both JDK-11 & JDK-8?
> > > > > - Ditch JDK-11 and directly shoot for JDK-17?
> > > > >
> > > > > Let me know your thoughts, In case anyone has some experience in
> this
> > > > area
> > > > > and have tried something in the context, feel free to share or may
> be
> > > if
> > > > > someone has any potential action plan or so
> > > > >
> > > > > -Ayush
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/HIVE-22415
> > > > > [2] https://issues.apache.org/jira/browse/HIVE-24484
> > > > >
> > > > > * changed the maven.compiler.source & maven.compiler.target to 11
> > > > >
> > > >
> > >
> >
>
Re: Move to JDK-11
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hey everyone,
If we claim that Hive supports a certain JDK then we should compile and run
tests with it.
The more JDKs we can support the better for everyone but this comes at a
cost (resources mostly). We should have a precommit run for every supported
JDK (frequency to be determined once per day/week) that compiles and run
all tests.
From my perspective, I would be pretty happy if we could cover the two edge
LTS releases at every point in time.
Then we have to decide also which JDK shall we use for the pull requests
and local dev environment. I think it makes sense to use the latest. People
like working on modern stuff and also it makes sense that newer releases
will also use newer versions. It would be pretty awkward if someone wants
to use the latest Hive version and it turns out that it can only run on
JDK8.
Best,
Stamatis
On Thu, Jun 1, 2023, 3:42 AM Sungwoo Park <gl...@gmail.com> wrote:
> Hi, everyone.
>
> I have not tested the master branch with Java 11/17 yet, but I would like
> to share my experience with testing a fork of branch-3.1 with Java 11/17
> (as part of developing Hive-MR3), in case that it can be useful for the
> discussion. I merged the patches listed in [1] HIVE-22415 and updated the
> Maven configuration for Java 11.
>
> 1. Building Hive was fine and I was able to run it with Java 11 as well as
> Java 17. So, it seems that the work reported in [1] is indeed complete for
> upgrading to Java 11 (and Java 17) and getting Hive to work.
>
> 2. However, there was a problem with running tests, so this can be
> additional work for upgrading to Java 11.
>
> 3. For performance, Java 17 gives about 8 percent of (free) performance
> improvement. When tested with 10TB TPC-DS, Java 8 takes 8074 seconds,
> whereas Java 17 takes 7415 seconds. Considering the maturity of Hive, I
> think this is not a small improvement because almost every query gets some
> speedup.
>
> Thanks,
>
> --- Sungwoo
>
> [1] https://issues.apache.org/jira/browse/HIVE-22415
>
>
> On Thu, Jun 1, 2023 at 3:53 AM Sai Hemanth Gantasala
> <sa...@cloudera.com.invalid> wrote:
>
> > Hi All,
> >
> > I would strongly advocate keeping support for JDK8.
> > Between JDK11 and JDK17, Depending on the amount of effort on the upgrade
> > I'm inclined towards JDK17 (JDK21 LTS will be released in Sep 2023).
> >
> > Thanks,
> > Sai.
> >
> > On Wed, May 31, 2023 at 5:39 AM László Bodor <bo...@gmail.com>
> > wrote:
> >
> > > *Hi!*
> > >
> > >
> > > *Should we support both JDK-11 & JDK-8?*
> > > IMO absolutely yes, let's not break up with JDK-8: according to its
> > > lifecycle, it's going to stay with us for a long time.
> > >
> > > I believe
> > > a) we should be able to compile on JDK8, JDK11, and JDK17 (github
> actions
> > > can cover this conveniently in precommit time, like tez
> > > <https://github.com/apache/tez/blob/master/.github/workflows/build.yml
> >)
> > > b) the release artifacts should be compatible with JDK8 as long as it
> is
> > > with us.
> > >
> > > Regards,
> > > Laszlo Bodor
> > >
> > >
> > > Butao Zhang <bu...@163.com> ezt írta (időpont: 2023. máj. 31.,
> > Sze,
> > > 14:33):
> > >
> > > > Thanks Ayush for driving this! Good to know that Hive is getting
> ready
> > > for
> > > > newer JDK.
> > > > From my opinon, if we have more community energy to put into it, we
> can
> > > > support both JDK-11 and JDK-17 like Spark[1]. If we have to make a
> > > choice
> > > > between a JDK-11 and JDK-17, i would like to choose the relatively
> new
> > > > version JDK-17, meanwhile, we should maintain compatibility with
> jdk8,
> > as
> > > > JDK-8 is still widely used in most big data platforms.
> > > >
> > > >
> > > > Thanks,
> > > > Butao Zhang
> > > >
> > > >
> > > > [1]https://issues.apache.org/jira/browse/SPARK-33772
> > > > ---- Replied Message ----
> > > > | From | Ayush Saxena<ay...@gmail.com> |
> > > > | Date | 5/31/2023 18:39 |
> > > > | To | dev<de...@hive.apache.org> |
> > > > | Subject | Move to JDK-11 |
> > > > Hi Everyone,
> > > > Want to pull in the attention of folks towards moving to JDK-11
> compile
> > > > time support in Hive. There was a ticket in the past [1] which talks
> > > about
> > > > it and If I could decode it right, it was blocked because the Hadoop
> > > > version used by Hive didn't had JDK-11 runtime support, But with [2]
> in
> > > we
> > > > have upgraded the Hadoop version, so that problem is sorted out. I
> > > couldn't
> > > > even see any unresolved tickets in the blocked state either.
> > > >
> > > > I quickly tried* a mvn clean install -DskipTests -Piceberg -Pitests
> > > > -Dmaven.javadoc.skip=true
> > > >
> > > > And no surprises it failed with some weird exceptions towards the
> end.
> > > But
> > > > I think that should be solvable.
> > > >
> > > > So, Questions?
> > > >
> > > > - What do folks think about this? Should we put in some effort
> towards
> > > > JDK-11
> > > > - Should we support both JDK-11 & JDK-8?
> > > > - Ditch JDK-11 and directly shoot for JDK-17?
> > > >
> > > > Let me know your thoughts, In case anyone has some experience in this
> > > area
> > > > and have tried something in the context, feel free to share or may be
> > if
> > > > someone has any potential action plan or so
> > > >
> > > > -Ayush
> > > >
> > > > [1] https://issues.apache.org/jira/browse/HIVE-22415
> > > > [2] https://issues.apache.org/jira/browse/HIVE-24484
> > > >
> > > > * changed the maven.compiler.source & maven.compiler.target to 11
> > > >
> > >
> >
>
Re: Move to JDK-11
Posted by Sungwoo Park <gl...@gmail.com>.
Hi, everyone.
I have not tested the master branch with Java 11/17 yet, but I would like
to share my experience with testing a fork of branch-3.1 with Java 11/17
(as part of developing Hive-MR3), in case that it can be useful for the
discussion. I merged the patches listed in [1] HIVE-22415 and updated the
Maven configuration for Java 11.
1. Building Hive was fine and I was able to run it with Java 11 as well as
Java 17. So, it seems that the work reported in [1] is indeed complete for
upgrading to Java 11 (and Java 17) and getting Hive to work.
2. However, there was a problem with running tests, so this can be
additional work for upgrading to Java 11.
3. For performance, Java 17 gives about 8 percent of (free) performance
improvement. When tested with 10TB TPC-DS, Java 8 takes 8074 seconds,
whereas Java 17 takes 7415 seconds. Considering the maturity of Hive, I
think this is not a small improvement because almost every query gets some
speedup.
Thanks,
--- Sungwoo
[1] https://issues.apache.org/jira/browse/HIVE-22415
On Thu, Jun 1, 2023 at 3:53 AM Sai Hemanth Gantasala
<sa...@cloudera.com.invalid> wrote:
> Hi All,
>
> I would strongly advocate keeping support for JDK8.
> Between JDK11 and JDK17, Depending on the amount of effort on the upgrade
> I'm inclined towards JDK17 (JDK21 LTS will be released in Sep 2023).
>
> Thanks,
> Sai.
>
> On Wed, May 31, 2023 at 5:39 AM László Bodor <bo...@gmail.com>
> wrote:
>
> > *Hi!*
> >
> >
> > *Should we support both JDK-11 & JDK-8?*
> > IMO absolutely yes, let's not break up with JDK-8: according to its
> > lifecycle, it's going to stay with us for a long time.
> >
> > I believe
> > a) we should be able to compile on JDK8, JDK11, and JDK17 (github actions
> > can cover this conveniently in precommit time, like tez
> > <https://github.com/apache/tez/blob/master/.github/workflows/build.yml>)
> > b) the release artifacts should be compatible with JDK8 as long as it is
> > with us.
> >
> > Regards,
> > Laszlo Bodor
> >
> >
> > Butao Zhang <bu...@163.com> ezt írta (időpont: 2023. máj. 31.,
> Sze,
> > 14:33):
> >
> > > Thanks Ayush for driving this! Good to know that Hive is getting ready
> > for
> > > newer JDK.
> > > From my opinon, if we have more community energy to put into it, we can
> > > support both JDK-11 and JDK-17 like Spark[1]. If we have to make a
> > choice
> > > between a JDK-11 and JDK-17, i would like to choose the relatively new
> > > version JDK-17, meanwhile, we should maintain compatibility with jdk8,
> as
> > > JDK-8 is still widely used in most big data platforms.
> > >
> > >
> > > Thanks,
> > > Butao Zhang
> > >
> > >
> > > [1]https://issues.apache.org/jira/browse/SPARK-33772
> > > ---- Replied Message ----
> > > | From | Ayush Saxena<ay...@gmail.com> |
> > > | Date | 5/31/2023 18:39 |
> > > | To | dev<de...@hive.apache.org> |
> > > | Subject | Move to JDK-11 |
> > > Hi Everyone,
> > > Want to pull in the attention of folks towards moving to JDK-11 compile
> > > time support in Hive. There was a ticket in the past [1] which talks
> > about
> > > it and If I could decode it right, it was blocked because the Hadoop
> > > version used by Hive didn't had JDK-11 runtime support, But with [2] in
> > we
> > > have upgraded the Hadoop version, so that problem is sorted out. I
> > couldn't
> > > even see any unresolved tickets in the blocked state either.
> > >
> > > I quickly tried* a mvn clean install -DskipTests -Piceberg -Pitests
> > > -Dmaven.javadoc.skip=true
> > >
> > > And no surprises it failed with some weird exceptions towards the end.
> > But
> > > I think that should be solvable.
> > >
> > > So, Questions?
> > >
> > > - What do folks think about this? Should we put in some effort towards
> > > JDK-11
> > > - Should we support both JDK-11 & JDK-8?
> > > - Ditch JDK-11 and directly shoot for JDK-17?
> > >
> > > Let me know your thoughts, In case anyone has some experience in this
> > area
> > > and have tried something in the context, feel free to share or may be
> if
> > > someone has any potential action plan or so
> > >
> > > -Ayush
> > >
> > > [1] https://issues.apache.org/jira/browse/HIVE-22415
> > > [2] https://issues.apache.org/jira/browse/HIVE-24484
> > >
> > > * changed the maven.compiler.source & maven.compiler.target to 11
> > >
> >
>
Re: Move to JDK-11
Posted by Sai Hemanth Gantasala <sa...@cloudera.com.INVALID>.
Hi All,
I would strongly advocate keeping support for JDK8.
Between JDK11 and JDK17, Depending on the amount of effort on the upgrade
I'm inclined towards JDK17 (JDK21 LTS will be released in Sep 2023).
Thanks,
Sai.
On Wed, May 31, 2023 at 5:39 AM László Bodor <bo...@gmail.com>
wrote:
> *Hi!*
>
>
> *Should we support both JDK-11 & JDK-8?*
> IMO absolutely yes, let's not break up with JDK-8: according to its
> lifecycle, it's going to stay with us for a long time.
>
> I believe
> a) we should be able to compile on JDK8, JDK11, and JDK17 (github actions
> can cover this conveniently in precommit time, like tez
> <https://github.com/apache/tez/blob/master/.github/workflows/build.yml>)
> b) the release artifacts should be compatible with JDK8 as long as it is
> with us.
>
> Regards,
> Laszlo Bodor
>
>
> Butao Zhang <bu...@163.com> ezt írta (időpont: 2023. máj. 31., Sze,
> 14:33):
>
> > Thanks Ayush for driving this! Good to know that Hive is getting ready
> for
> > newer JDK.
> > From my opinon, if we have more community energy to put into it, we can
> > support both JDK-11 and JDK-17 like Spark[1]. If we have to make a
> choice
> > between a JDK-11 and JDK-17, i would like to choose the relatively new
> > version JDK-17, meanwhile, we should maintain compatibility with jdk8, as
> > JDK-8 is still widely used in most big data platforms.
> >
> >
> > Thanks,
> > Butao Zhang
> >
> >
> > [1]https://issues.apache.org/jira/browse/SPARK-33772
> > ---- Replied Message ----
> > | From | Ayush Saxena<ay...@gmail.com> |
> > | Date | 5/31/2023 18:39 |
> > | To | dev<de...@hive.apache.org> |
> > | Subject | Move to JDK-11 |
> > Hi Everyone,
> > Want to pull in the attention of folks towards moving to JDK-11 compile
> > time support in Hive. There was a ticket in the past [1] which talks
> about
> > it and If I could decode it right, it was blocked because the Hadoop
> > version used by Hive didn't had JDK-11 runtime support, But with [2] in
> we
> > have upgraded the Hadoop version, so that problem is sorted out. I
> couldn't
> > even see any unresolved tickets in the blocked state either.
> >
> > I quickly tried* a mvn clean install -DskipTests -Piceberg -Pitests
> > -Dmaven.javadoc.skip=true
> >
> > And no surprises it failed with some weird exceptions towards the end.
> But
> > I think that should be solvable.
> >
> > So, Questions?
> >
> > - What do folks think about this? Should we put in some effort towards
> > JDK-11
> > - Should we support both JDK-11 & JDK-8?
> > - Ditch JDK-11 and directly shoot for JDK-17?
> >
> > Let me know your thoughts, In case anyone has some experience in this
> area
> > and have tried something in the context, feel free to share or may be if
> > someone has any potential action plan or so
> >
> > -Ayush
> >
> > [1] https://issues.apache.org/jira/browse/HIVE-22415
> > [2] https://issues.apache.org/jira/browse/HIVE-24484
> >
> > * changed the maven.compiler.source & maven.compiler.target to 11
> >
>
Re: Move to JDK-11
Posted by László Bodor <bo...@gmail.com>.
*Hi!*
*Should we support both JDK-11 & JDK-8?*
IMO absolutely yes, let's not break up with JDK-8: according to its
lifecycle, it's going to stay with us for a long time.
I believe
a) we should be able to compile on JDK8, JDK11, and JDK17 (github actions
can cover this conveniently in precommit time, like tez
<https://github.com/apache/tez/blob/master/.github/workflows/build.yml>)
b) the release artifacts should be compatible with JDK8 as long as it is
with us.
Regards,
Laszlo Bodor
Butao Zhang <bu...@163.com> ezt írta (időpont: 2023. máj. 31., Sze,
14:33):
> Thanks Ayush for driving this! Good to know that Hive is getting ready for
> newer JDK.
> From my opinon, if we have more community energy to put into it, we can
> support both JDK-11 and JDK-17 like Spark[1]. If we have to make a choice
> between a JDK-11 and JDK-17, i would like to choose the relatively new
> version JDK-17, meanwhile, we should maintain compatibility with jdk8, as
> JDK-8 is still widely used in most big data platforms.
>
>
> Thanks,
> Butao Zhang
>
>
> [1]https://issues.apache.org/jira/browse/SPARK-33772
> ---- Replied Message ----
> | From | Ayush Saxena<ay...@gmail.com> |
> | Date | 5/31/2023 18:39 |
> | To | dev<de...@hive.apache.org> |
> | Subject | Move to JDK-11 |
> Hi Everyone,
> Want to pull in the attention of folks towards moving to JDK-11 compile
> time support in Hive. There was a ticket in the past [1] which talks about
> it and If I could decode it right, it was blocked because the Hadoop
> version used by Hive didn't had JDK-11 runtime support, But with [2] in we
> have upgraded the Hadoop version, so that problem is sorted out. I couldn't
> even see any unresolved tickets in the blocked state either.
>
> I quickly tried* a mvn clean install -DskipTests -Piceberg -Pitests
> -Dmaven.javadoc.skip=true
>
> And no surprises it failed with some weird exceptions towards the end. But
> I think that should be solvable.
>
> So, Questions?
>
> - What do folks think about this? Should we put in some effort towards
> JDK-11
> - Should we support both JDK-11 & JDK-8?
> - Ditch JDK-11 and directly shoot for JDK-17?
>
> Let me know your thoughts, In case anyone has some experience in this area
> and have tried something in the context, feel free to share or may be if
> someone has any potential action plan or so
>
> -Ayush
>
> [1] https://issues.apache.org/jira/browse/HIVE-22415
> [2] https://issues.apache.org/jira/browse/HIVE-24484
>
> * changed the maven.compiler.source & maven.compiler.target to 11
>
Re: Move to JDK-11
Posted by Butao Zhang <bu...@163.com>.
Thanks Ayush for driving this! Good to know that Hive is getting ready for newer JDK.
From my opinon, if we have more community energy to put into it, we can support both JDK-11 and JDK-17 like Spark[1]. If we have to make a choice between a JDK-11 and JDK-17, i would like to choose the relatively new version JDK-17, meanwhile, we should maintain compatibility with jdk8, as JDK-8 is still widely used in most big data platforms.
Thanks,
Butao Zhang
[1]https://issues.apache.org/jira/browse/SPARK-33772
---- Replied Message ----
| From | Ayush Saxena<ay...@gmail.com> |
| Date | 5/31/2023 18:39 |
| To | dev<de...@hive.apache.org> |
| Subject | Move to JDK-11 |
Hi Everyone,
Want to pull in the attention of folks towards moving to JDK-11 compile
time support in Hive. There was a ticket in the past [1] which talks about
it and If I could decode it right, it was blocked because the Hadoop
version used by Hive didn't had JDK-11 runtime support, But with [2] in we
have upgraded the Hadoop version, so that problem is sorted out. I couldn't
even see any unresolved tickets in the blocked state either.
I quickly tried* a mvn clean install -DskipTests -Piceberg -Pitests
-Dmaven.javadoc.skip=true
And no surprises it failed with some weird exceptions towards the end. But
I think that should be solvable.
So, Questions?
- What do folks think about this? Should we put in some effort towards
JDK-11
- Should we support both JDK-11 & JDK-8?
- Ditch JDK-11 and directly shoot for JDK-17?
Let me know your thoughts, In case anyone has some experience in this area
and have tried something in the context, feel free to share or may be if
someone has any potential action plan or so
-Ayush
[1] https://issues.apache.org/jira/browse/HIVE-22415
[2] https://issues.apache.org/jira/browse/HIVE-24484
* changed the maven.compiler.source & maven.compiler.target to 11