You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Takanobu Asanuma <ta...@apache.org> on 2018/11/05 04:09:58 UTC
About supporting upcoming java versions
Hi all,
As discussed in HADOOP-11123, we stop fixing incompatibilities of JDK9 and
JDK10, and would like to focus on supporting JDK11.
This week, I plan to close HADOOP-11123(Support Java 9) and
HADOOP-11423(Support Java 10) as duplicate/superceded, and move the
unresolved sub-tasks to under HADOOP-15338(Support Java 11).
If there are any opinions, please reply this email or comment on the jiras.
Any help for supporting upcoming JDK would be highly appreciated.
Regards,
- Takanobu
Re: About supporting upcoming java versions
Posted by "J. Rottinghuis" <jr...@gmail.com>.
Not to throw fuel on the fire so to say, nor to make any statement about
not wanting anybody to spend time on JDK 9 or 10, but our general thinking
at Twitter is that we'll skip over these versions as well and move straight
to JDK 11 as well.
That said, this is still a bit of an aspiration for us rather than
something we're working on right away in the Hadoop team (there is some
other tech-debt to iron out first before we will get to that)
Cheers,
Joep
On Wed, Nov 7, 2018 at 2:18 AM Steve Loughran <st...@hortonworks.com>
wrote:
>
> If there are problems w/ JDK11 then we should be talking to oracle about
> them to have them fixed. Is there an ASF JIRA on this issue yet?
>
> As usual, the large physical clusters will be slow to upgrade,
>
> but the smaller cloud ones can get away with being agile, and as I believe
> that YARN does let you run code with a different path to the jvm, people
> can mix things.
> This makes it possible for people to run java 11+ apps even if hadoop
> itself is on java 8.
>
> And this time we may want to think about: which release we declare "ready
> for Java 11", being proactive rather than lagging behind the public
> releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the
> java 8 language for a while, but there's a lot more we can do there in our
> code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit
> when things are potentially null, and while its crippled by the java
> language itself (
> http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html
> ), its still something we can embrace (*)
>
>
> Takanobu,
>
> I've been watching the work you, Akira and others have been putting in for
> java 9+ support and its wonderful, If we had an annual award for
> "persevering in the presence of extreme suffering" it'd be the top
> candidate for this year's work.
>
> it means we are lined up to let people run on Hadoop 11 if they want, and
> gives that option of moving to java 11 sooner rather than later. I'm also
> looking at JUnit 5, wondering when I can embrace it fully (i.e. not worry
> about cherry picking code into junit 4 tests)
>
> Thanks for all your work
>
> -Steve
>
> (*) I also have in the test code of that branch a bonding of UG.doAs which
> takes closures
>
>
> https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
>
>
> lets me do things like
>
> assertEquals("FS username in doAs()",
> ALICE,
> doAs(bobUser, () -> fs.getUsername()))
>
> If someone wants to actually pull this support into UGI itself, happy to
> review. as moving our doAs code to things like bobUser.doAs(() ->
> fs.create(path)) will transform all those UGI code users.
>
> On 6 Nov 2018, at 05:57, Takanobu Asanuma <tasanuma@apache.org<mailto:
> tasanuma@apache.org>> wrote:
>
> Thanks for your reply, Owen.
>
> That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
> strict subset of the issues getting to JDK 11.
>
> Most of the issues that we have fixed are subset of the ones of JDK 11. But
> there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
> has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
> already been EOL. I wonder if we should treat such a kind of error going
> forward.
>
> I've hit at least one pretty serious JVM bug in JDK 11
> Could you please share that detail?
>
> In any case, we should be carefully that what version of Hadoop is ready
> for JDK 11. It will take some time yet. And we also need to keep supporting
> JDK 8 for a while.
>
> Regards,
> - Takanobu
>
>
>
>
Re: FW: About supporting upcoming java versions
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
> I must admit when the ORC unit tests failed with jdk11 and the debugger
showed that it was failing because it was in code that should have been
unreachable with the data values, I just went back to jdk10 rather than
diagnosing it.
Owen, could you check the bug you faced was the filed test failure as ORC-436?
https://jira.apache.org/jira/browse/ORC-436
Akira and I confirmed that it was not caused by JDK bug, and it has
workaround to avoid the root cause as reported on ORC-436.
Regards,
- Tsuyoshi
On Mon, Nov 12, 2018 at 10:08 AM Takanobu Asanuma <ta...@apache.org> wrote:
>
> I appreciate your help, Ewan and Tsuyoshi.
>
> Based on the discussion in this thread, after the bug reported by Owen
> and Tsuyoshi is resolved, I'll close the jiras of supporting JDK9/10
> and let's focus on supporting JDK 11 (and JDK 8).
>
> >
> >
> > on 2018/11/11 23:35, "Tsuyoshi Ozawa"<oz...@apache.org> wrote:
> >
> > Thanks Takanobu for stating the discussion, and thanks Ewan for
> > sharing the pointer.
> >
> > I reproduced the bug Owen said only with OpenJDK 11, so I filed it on
> > ORC jira for now:
> > https://jira.apache.org/jira/browse/ORC-436
> >
> > I looked over the bug, but I cannot identify the root cause - only on
> > OpenJDK 11, the test failed with the following exception:
> > > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
> >
> > This bug might affect the Hadoop. For the case that the bug affects
> > Hadoop, I talked offline with an OpenJDK developer who has a
> > permission to create jira ticket of OpenJDK. He said that we should
> > have a discussion with OpenJDK community before creating the ticket
> > since we don't know the cause in detail and we are also unsure whether
> > it is really JDK bug. The ticket on the OpenJDK jira should be
> > source-code level or stack-trace level of OpenJDK like this:
> > https://bugs.java.com/view_bug.do?bug_id=8196296
> >
> > He also gave us the pointer for the discussion:
> > jdk-dev@openjdk.java.net might be a good place to report. If someone
> > has interest in this problem of JDK11, please feel free to pick it up.
> >
> > Regards,
> > - Tsuyoshi
> > On Fri, Nov 9, 2018 at 8:35 AM Ewan Higgs <Ew...@wdc.com> wrote:
> > >
> > > Hi all,
> > > Reporting bugs to Java is a bit weird/nontrivial so I'm sympathetic to Owen's situation. The openjdk bug tracker requires users be committers so no one can comment unless they're already contributing.
> > >
> > > Their tracker is here:
> > > https://bugs.openjdk.java.net/projects/JDK/issues
> > >
> > > To actually file a bug, the form is here:
> > > https://bugreport.java.com/bugreport
> > >
> > > Yours,
> > > Ewan
> > >
> > > On 07/11/2018, 11:18, "Steve Loughran" <st...@hortonworks.com> wrote:
> > >
> > >
> > > If there are problems w/ JDK11 then we should be talking to oracle about them to have them fixed. Is there an ASF JIRA on this issue yet?
> > >
> > > As usual, the large physical clusters will be slow to upgrade,
> > >
> > > but the smaller cloud ones can get away with being agile, and as I believe that YARN does let you run code with a different path to the jvm, people can mix things.
> > > This makes it possible for people to run java 11+ apps even if hadoop itself is on java 8.
> > >
> > > And this time we may want to think about: which release we declare "ready for Java 11", being proactive rather than lagging behind the public releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the java 8 language for a while, but there's a lot more we can do there in our code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit when things are potentially null, and while its crippled by the java language itself (http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html ), its still something we can embrace (*)
> > >
> > >
> > > Takanobu,
> > >
> > > I've been watching the work you, Akira and others have been putting in for java 9+ support and its wonderful, If we had an annual award for "persevering in the presence of extreme suffering" it'd be the top candidate for this year's work.
> > >
> > > it means we are lined up to let people run on Hadoop 11 if they want, and gives that option of moving to java 11 sooner rather than later. I'm also looking at JUnit 5, wondering when I can embrace it fully (i.e. not worry about cherry picking code into junit 4 tests)
> > >
> > > Thanks for all your work
> > >
> > > -Steve
> > >
> > > (*) I also have in the test code of that branch a bonding of UG.doAs which takes closures
> > >
> > > https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
> > >
> > >
> > > lets me do things like
> > >
> > > assertEquals("FS username in doAs()",
> > > ALICE,
> > > doAs(bobUser, () -> fs.getUsername()))
> > >
> > > If someone wants to actually pull this support into UGI itself, happy to review. as moving our doAs code to things like bobUser.doAs(() -> fs.create(path)) will transform all those UGI code users.
> > >
> > > On 6 Nov 2018, at 05:57, Takanobu Asanuma <ta...@apache.org>> wrote:
> > >
> > > Thanks for your reply, Owen.
> > >
> > > That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
> > > strict subset of the issues getting to JDK 11.
> > >
> > > Most of the issues that we have fixed are subset of the ones of JDK 11. But
> > > there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
> > > has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
> > > already been EOL. I wonder if we should treat such a kind of error going
> > > forward.
> > >
> > > I've hit at least one pretty serious JVM bug in JDK 11
> > > Could you please share that detail?
> > >
> > > In any case, we should be carefully that what version of Hadoop is ready
> > > for JDK 11. It will take some time yet. And we also need to keep supporting
> > > JDK 8 for a while.
> > >
> > > Regards,
> > > - Takanobu
> > >
> > >
> > >
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: common-dev-help@hadoop.apache.org
> >
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: FW: About supporting upcoming java versions
Posted by Takanobu Asanuma <ta...@apache.org>.
I appreciate your help, Ewan and Tsuyoshi.
Based on the discussion in this thread, after the bug reported by Owen
and Tsuyoshi is resolved, I'll close the jiras of supporting JDK9/10
and let's focus on supporting JDK 11 (and JDK 8).
>
>
> on 2018/11/11 23:35, "Tsuyoshi Ozawa"<oz...@apache.org> wrote:
>
> Thanks Takanobu for stating the discussion, and thanks Ewan for
> sharing the pointer.
>
> I reproduced the bug Owen said only with OpenJDK 11, so I filed it on
> ORC jira for now:
> https://jira.apache.org/jira/browse/ORC-436
>
> I looked over the bug, but I cannot identify the root cause - only on
> OpenJDK 11, the test failed with the following exception:
> > Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
>
> This bug might affect the Hadoop. For the case that the bug affects
> Hadoop, I talked offline with an OpenJDK developer who has a
> permission to create jira ticket of OpenJDK. He said that we should
> have a discussion with OpenJDK community before creating the ticket
> since we don't know the cause in detail and we are also unsure whether
> it is really JDK bug. The ticket on the OpenJDK jira should be
> source-code level or stack-trace level of OpenJDK like this:
> https://bugs.java.com/view_bug.do?bug_id=8196296
>
> He also gave us the pointer for the discussion:
> jdk-dev@openjdk.java.net might be a good place to report. If someone
> has interest in this problem of JDK11, please feel free to pick it up.
>
> Regards,
> - Tsuyoshi
> On Fri, Nov 9, 2018 at 8:35 AM Ewan Higgs <Ew...@wdc.com> wrote:
> >
> > Hi all,
> > Reporting bugs to Java is a bit weird/nontrivial so I'm sympathetic to Owen's situation. The openjdk bug tracker requires users be committers so no one can comment unless they're already contributing.
> >
> > Their tracker is here:
> > https://bugs.openjdk.java.net/projects/JDK/issues
> >
> > To actually file a bug, the form is here:
> > https://bugreport.java.com/bugreport
> >
> > Yours,
> > Ewan
> >
> > On 07/11/2018, 11:18, "Steve Loughran" <st...@hortonworks.com> wrote:
> >
> >
> > If there are problems w/ JDK11 then we should be talking to oracle about them to have them fixed. Is there an ASF JIRA on this issue yet?
> >
> > As usual, the large physical clusters will be slow to upgrade,
> >
> > but the smaller cloud ones can get away with being agile, and as I believe that YARN does let you run code with a different path to the jvm, people can mix things.
> > This makes it possible for people to run java 11+ apps even if hadoop itself is on java 8.
> >
> > And this time we may want to think about: which release we declare "ready for Java 11", being proactive rather than lagging behind the public releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the java 8 language for a while, but there's a lot more we can do there in our code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit when things are potentially null, and while its crippled by the java language itself (http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html ), its still something we can embrace (*)
> >
> >
> > Takanobu,
> >
> > I've been watching the work you, Akira and others have been putting in for java 9+ support and its wonderful, If we had an annual award for "persevering in the presence of extreme suffering" it'd be the top candidate for this year's work.
> >
> > it means we are lined up to let people run on Hadoop 11 if they want, and gives that option of moving to java 11 sooner rather than later. I'm also looking at JUnit 5, wondering when I can embrace it fully (i.e. not worry about cherry picking code into junit 4 tests)
> >
> > Thanks for all your work
> >
> > -Steve
> >
> > (*) I also have in the test code of that branch a bonding of UG.doAs which takes closures
> >
> > https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
> >
> >
> > lets me do things like
> >
> > assertEquals("FS username in doAs()",
> > ALICE,
> > doAs(bobUser, () -> fs.getUsername()))
> >
> > If someone wants to actually pull this support into UGI itself, happy to review. as moving our doAs code to things like bobUser.doAs(() -> fs.create(path)) will transform all those UGI code users.
> >
> > On 6 Nov 2018, at 05:57, Takanobu Asanuma <ta...@apache.org>> wrote:
> >
> > Thanks for your reply, Owen.
> >
> > That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
> > strict subset of the issues getting to JDK 11.
> >
> > Most of the issues that we have fixed are subset of the ones of JDK 11. But
> > there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
> > has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
> > already been EOL. I wonder if we should treat such a kind of error going
> > forward.
> >
> > I've hit at least one pretty serious JVM bug in JDK 11
> > Could you please share that detail?
> >
> > In any case, we should be carefully that what version of Hadoop is ready
> > for JDK 11. It will take some time yet. And we also need to keep supporting
> > JDK 8 for a while.
> >
> > Regards,
> > - Takanobu
> >
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: About supporting upcoming java versions
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Thanks Takanobu for stating the discussion, and thanks Ewan for
sharing the pointer.
I reproduced the bug Owen said only with OpenJDK 11, so I filed it on
ORC jira for now:
https://jira.apache.org/jira/browse/ORC-436
I looked over the bug, but I cannot identify the root cause - only on
OpenJDK 11, the test failed with the following exception:
> Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
This bug might affect the Hadoop. For the case that the bug affects
Hadoop, I talked offline with an OpenJDK developer who has a
permission to create jira ticket of OpenJDK. He said that we should
have a discussion with OpenJDK community before creating the ticket
since we don't know the cause in detail and we are also unsure whether
it is really JDK bug. The ticket on the OpenJDK jira should be
source-code level or stack-trace level of OpenJDK like this:
https://bugs.java.com/view_bug.do?bug_id=8196296
He also gave us the pointer for the discussion:
jdk-dev@openjdk.java.net might be a good place to report. If someone
has interest in this problem of JDK11, please feel free to pick it up.
Regards,
- Tsuyoshi
On Fri, Nov 9, 2018 at 8:35 AM Ewan Higgs <Ew...@wdc.com> wrote:
>
> Hi all,
> Reporting bugs to Java is a bit weird/nontrivial so I'm sympathetic to Owen's situation. The openjdk bug tracker requires users be committers so no one can comment unless they're already contributing.
>
> Their tracker is here:
> https://bugs.openjdk.java.net/projects/JDK/issues
>
> To actually file a bug, the form is here:
> https://bugreport.java.com/bugreport
>
> Yours,
> Ewan
>
> On 07/11/2018, 11:18, "Steve Loughran" <st...@hortonworks.com> wrote:
>
>
> If there are problems w/ JDK11 then we should be talking to oracle about them to have them fixed. Is there an ASF JIRA on this issue yet?
>
> As usual, the large physical clusters will be slow to upgrade,
>
> but the smaller cloud ones can get away with being agile, and as I believe that YARN does let you run code with a different path to the jvm, people can mix things.
> This makes it possible for people to run java 11+ apps even if hadoop itself is on java 8.
>
> And this time we may want to think about: which release we declare "ready for Java 11", being proactive rather than lagging behind the public releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the java 8 language for a while, but there's a lot more we can do there in our code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit when things are potentially null, and while its crippled by the java language itself (http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html ), its still something we can embrace (*)
>
>
> Takanobu,
>
> I've been watching the work you, Akira and others have been putting in for java 9+ support and its wonderful, If we had an annual award for "persevering in the presence of extreme suffering" it'd be the top candidate for this year's work.
>
> it means we are lined up to let people run on Hadoop 11 if they want, and gives that option of moving to java 11 sooner rather than later. I'm also looking at JUnit 5, wondering when I can embrace it fully (i.e. not worry about cherry picking code into junit 4 tests)
>
> Thanks for all your work
>
> -Steve
>
> (*) I also have in the test code of that branch a bonding of UG.doAs which takes closures
>
> https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
>
>
> lets me do things like
>
> assertEquals("FS username in doAs()",
> ALICE,
> doAs(bobUser, () -> fs.getUsername()))
>
> If someone wants to actually pull this support into UGI itself, happy to review. as moving our doAs code to things like bobUser.doAs(() -> fs.create(path)) will transform all those UGI code users.
>
> On 6 Nov 2018, at 05:57, Takanobu Asanuma <ta...@apache.org>> wrote:
>
> Thanks for your reply, Owen.
>
> That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
> strict subset of the issues getting to JDK 11.
>
> Most of the issues that we have fixed are subset of the ones of JDK 11. But
> there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
> has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
> already been EOL. I wonder if we should treat such a kind of error going
> forward.
>
> I've hit at least one pretty serious JVM bug in JDK 11
> Could you please share that detail?
>
> In any case, we should be carefully that what version of Hadoop is ready
> for JDK 11. It will take some time yet. And we also need to keep supporting
> JDK 8 for a while.
>
> Regards,
> - Takanobu
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: About supporting upcoming java versions
Posted by Ewan Higgs <Ew...@wdc.com>.
Hi all,
Reporting bugs to Java is a bit weird/nontrivial so I'm sympathetic to Owen's situation. The openjdk bug tracker requires users be committers so no one can comment unless they're already contributing.
Their tracker is here:
https://bugs.openjdk.java.net/projects/JDK/issues
To actually file a bug, the form is here:
https://bugreport.java.com/bugreport
Yours,
Ewan
On 07/11/2018, 11:18, "Steve Loughran" <st...@hortonworks.com> wrote:
If there are problems w/ JDK11 then we should be talking to oracle about them to have them fixed. Is there an ASF JIRA on this issue yet?
As usual, the large physical clusters will be slow to upgrade,
but the smaller cloud ones can get away with being agile, and as I believe that YARN does let you run code with a different path to the jvm, people can mix things.
This makes it possible for people to run java 11+ apps even if hadoop itself is on java 8.
And this time we may want to think about: which release we declare "ready for Java 11", being proactive rather than lagging behind the public releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the java 8 language for a while, but there's a lot more we can do there in our code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit when things are potentially null, and while its crippled by the java language itself (http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html ), its still something we can embrace (*)
Takanobu,
I've been watching the work you, Akira and others have been putting in for java 9+ support and its wonderful, If we had an annual award for "persevering in the presence of extreme suffering" it'd be the top candidate for this year's work.
it means we are lined up to let people run on Hadoop 11 if they want, and gives that option of moving to java 11 sooner rather than later. I'm also looking at JUnit 5, wondering when I can embrace it fully (i.e. not worry about cherry picking code into junit 4 tests)
Thanks for all your work
-Steve
(*) I also have in the test code of that branch a bonding of UG.doAs which takes closures
https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
lets me do things like
assertEquals("FS username in doAs()",
ALICE,
doAs(bobUser, () -> fs.getUsername()))
If someone wants to actually pull this support into UGI itself, happy to review. as moving our doAs code to things like bobUser.doAs(() -> fs.create(path)) will transform all those UGI code users.
On 6 Nov 2018, at 05:57, Takanobu Asanuma <ta...@apache.org>> wrote:
Thanks for your reply, Owen.
That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
strict subset of the issues getting to JDK 11.
Most of the issues that we have fixed are subset of the ones of JDK 11. But
there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
already been EOL. I wonder if we should treat such a kind of error going
forward.
I've hit at least one pretty serious JVM bug in JDK 11
Could you please share that detail?
In any case, we should be carefully that what version of Hadoop is ready
for JDK 11. It will take some time yet. And we also need to keep supporting
JDK 8 for a while.
Regards,
- Takanobu
Re: About supporting upcoming java versions
Posted by Owen O'Malley <ow...@gmail.com>.
On Wed, Nov 7, 2018 at 2:18 AM Steve Loughran <st...@hortonworks.com>
wrote:
>
> If there are problems w/ JDK11 then we should be talking to oracle about
> them to have them fixed. Is there an ASF JIRA on this issue yet?
>
I must admit when the ORC unit tests failed with jdk11 and the debugger
showed that it was failing because it was in code that should have been
unreachable with the data values, I just went back to jdk10 rather than
diagnosing it.
For bugs with relatively simple workarounds, it is worth fixing them. For
example, in ORC I hit a jdk bug that is introduced in 8 and fixed in 9. I
worked around it with ORC-432.
.. Owen
> As usual, the large physical clusters will be slow to upgrade,
>
> but the smaller cloud ones can get away with being agile, and as I believe
> that YARN does let you run code with a different path to the jvm, people
> can mix things.
> This makes it possible for people to run java 11+ apps even if hadoop
> itself is on java 8.
>
> And this time we may want to think about: which release we declare "ready
> for Java 11", being proactive rather than lagging behind the public
> releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the
> java 8 language for a while, but there's a lot more we can do there in our
> code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit
> when things are potentially null, and while its crippled by the java
> language itself (
> http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html
> ), its still something we can embrace (*)
>
>
> Takanobu,
>
> I've been watching the work you, Akira and others have been putting in for
> java 9+ support and its wonderful, If we had an annual award for
> "persevering in the presence of extreme suffering" it'd be the top
> candidate for this year's work.
>
> it means we are lined up to let people run on Hadoop 11 if they want, and
> gives that option of moving to java 11 sooner rather than later. I'm also
> looking at JUnit 5, wondering when I can embrace it fully (i.e. not worry
> about cherry picking code into junit 4 tests)
>
> Thanks for all your work
>
> -Steve
>
> (*) I also have in the test code of that branch a bonding of UG.doAs which
> takes closures
>
>
> https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
>
>
> lets me do things like
>
> assertEquals("FS username in doAs()",
> ALICE,
> doAs(bobUser, () -> fs.getUsername()))
>
> If someone wants to actually pull this support into UGI itself, happy to
> review. as moving our doAs code to things like bobUser.doAs(() ->
> fs.create(path)) will transform all those UGI code users.
>
> On 6 Nov 2018, at 05:57, Takanobu Asanuma <tasanuma@apache.org<mailto:
> tasanuma@apache.org>> wrote:
>
> Thanks for your reply, Owen.
>
> That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
> strict subset of the issues getting to JDK 11.
>
> Most of the issues that we have fixed are subset of the ones of JDK 11. But
> there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
> has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
> already been EOL. I wonder if we should treat such a kind of error going
> forward.
>
> I've hit at least one pretty serious JVM bug in JDK 11
> Could you please share that detail?
>
> In any case, we should be carefully that what version of Hadoop is ready
> for JDK 11. It will take some time yet. And we also need to keep supporting
> JDK 8 for a while.
>
> Regards,
> - Takanobu
>
>
>
>
Re: FW: About supporting upcoming java versions
Posted by Takanobu Asanuma <ta...@apache.org>.
Steve,
Thank you very much. I am honored to hear you say that. I also thank to
Akira and the others for driving this work.
I agree to being proactive since software depending on Hadoop is
increasing. And it's good to adopt functional programming features
gradually.
Regards,
- Takanobu
> on 2018/11/07 19:18, "Steve Loughran"<st...@hortonworks.com> wrote:
>
>
> If there are problems w/ JDK11 then we should be talking to oracle
> about them to have them fixed. Is there an ASF JIRA on this issue yet?
>
> As usual, the large physical clusters will be slow to upgrade,
>
> but the smaller cloud ones can get away with being agile, and as I
> believe that YARN does let you run code with a different path to the jvm,
> people can mix things.
> This makes it possible for people to run java 11+ apps even if hadoop
> itself is on java 8.
>
> And this time we may want to think about: which release we declare
> "ready for Java 11", being proactive rather than lagging behind the public
> releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the
> java 8 language for a while, but there's a lot more we can do there in our
> code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit
> when things are potentially null, and while its crippled by the java
> language itself (
> http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html
> ), its still something we can embrace (*)
>
>
> Takanobu,
>
> I've been watching the work you, Akira and others have been putting in
> for java 9+ support and its wonderful, If we had an annual award for
> "persevering in the presence of extreme suffering" it'd be the top
> candidate for this year's work.
>
> it means we are lined up to let people run on Hadoop 11 if they want,
> and gives that option of moving to java 11 sooner rather than later. I'm
> also looking at JUnit 5, wondering when I can embrace it fully (i.e. not
> worry about cherry picking code into junit 4 tests)
>
> Thanks for all your work
>
> -Steve
>
> (*) I also have in the test code of that branch a bonding of UG.doAs
> which takes closures
>
>
> https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
>
>
> lets me do things like
>
> assertEquals("FS username in doAs()",
> ALICE,
> doAs(bobUser, () -> fs.getUsername()))
>
> If someone wants to actually pull this support into UGI itself, happy
> to review. as moving our doAs code to things like bobUser.doAs(() ->
> fs.create(path)) will transform all those UGI code users.
>
> On 6 Nov 2018, at 05:57, Takanobu Asanuma <tasanuma@apache.org<mailto:
> tasanuma@apache.org>> wrote:
>
> Thanks for your reply, Owen.
>
> That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
> strict subset of the issues getting to JDK 11.
>
> Most of the issues that we have fixed are subset of the ones of JDK
> 11. But
> there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10
> which
> has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
> already been EOL. I wonder if we should treat such a kind of error
> going
> forward.
>
> I've hit at least one pretty serious JVM bug in JDK 11
> Could you please share that detail?
>
> In any case, we should be carefully that what version of Hadoop is
> ready
> for JDK 11. It will take some time yet. And we also need to keep
> supporting
> JDK 8 for a while.
>
> Regards,
> - Takanobu
>
>
>
>
>
>
Re: About supporting upcoming java versions
Posted by Steve Loughran <st...@hortonworks.com>.
If there are problems w/ JDK11 then we should be talking to oracle about them to have them fixed. Is there an ASF JIRA on this issue yet?
As usual, the large physical clusters will be slow to upgrade,
but the smaller cloud ones can get away with being agile, and as I believe that YARN does let you run code with a different path to the jvm, people can mix things.
This makes it possible for people to run java 11+ apps even if hadoop itself is on java 8.
And this time we may want to think about: which release we declare "ready for Java 11", being proactive rather than lagging behind the public releases by many years (6=>7, 7=>8). Of course, we'll have to stay with the java 8 language for a while, but there's a lot more we can do there in our code. I'm currently (HADOOP-14556) embracing Optional, as it makes explicit when things are potentially null, and while its crippled by the java language itself (http://steveloughran.blogspot.com/2018/10/javas-use-of-checked-exceptions.html ), its still something we can embrace (*)
Takanobu,
I've been watching the work you, Akira and others have been putting in for java 9+ support and its wonderful, If we had an annual award for "persevering in the presence of extreme suffering" it'd be the top candidate for this year's work.
it means we are lined up to let people run on Hadoop 11 if they want, and gives that option of moving to java 11 sooner rather than later. I'm also looking at JUnit 5, wondering when I can embrace it fully (i.e. not worry about cherry picking code into junit 4 tests)
Thanks for all your work
-Steve
(*) I also have in the test code of that branch a bonding of UG.doAs which takes closures
https://github.com/steveloughran/hadoop/blob/s3/HADOOP-14556-delegation-token/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/LambdaTestUtils.java#L865
lets me do things like
assertEquals("FS username in doAs()",
ALICE,
doAs(bobUser, () -> fs.getUsername()))
If someone wants to actually pull this support into UGI itself, happy to review. as moving our doAs code to things like bobUser.doAs(() -> fs.create(path)) will transform all those UGI code users.
On 6 Nov 2018, at 05:57, Takanobu Asanuma <ta...@apache.org>> wrote:
Thanks for your reply, Owen.
That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
strict subset of the issues getting to JDK 11.
Most of the issues that we have fixed are subset of the ones of JDK 11. But
there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
already been EOL. I wonder if we should treat such a kind of error going
forward.
I've hit at least one pretty serious JVM bug in JDK 11
Could you please share that detail?
In any case, we should be carefully that what version of Hadoop is ready
for JDK 11. It will take some time yet. And we also need to keep supporting
JDK 8 for a while.
Regards,
- Takanobu
Re: About supporting upcoming java versions
Posted by Takanobu Asanuma <ta...@apache.org>.
Thanks for your reply, Owen.
> That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
strict subset of the issues getting to JDK 11.
Most of the issues that we have fixed are subset of the ones of JDK 11. But
there seem to be some exceptions. HADOOP-15905 is a bug of JDK 9/10 which
has been fixed in JDK 11. It is difficult to fix it since JDK 9/10 have
already been EOL. I wonder if we should treat such a kind of error going
forward.
> I've hit at least one pretty serious JVM bug in JDK 11
Could you please share that detail?
In any case, we should be carefully that what version of Hadoop is ready
for JDK 11. It will take some time yet. And we also need to keep supporting
JDK 8 for a while.
Regards,
- Takanobu
2018年11月6日(火) 2:21 Owen O'Malley <ow...@gmail.com>:
> I’m less convinced. I’ve hit at least one pretty serious JVM bug in JDK
> 11, so there is value in enabling JDK 9 and 10 even with blockers on JDK 11.
>
> That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a
> strict subset of the issues getting to JDK 11.
>
> .. Owen
>
> > On Nov 5, 2018, at 6:14 AM, Kihwal Lee <ki...@oath.com.INVALID> wrote:
> >
> > +1 It makes sense to converge on the LTS version.
> > Thanks for the initiative.
> >
> > 73
> > Kihwal
> >
> > On Sun, Nov 4, 2018 at 10:10 PM Takanobu Asanuma <ta...@apache.org>
> > wrote:
> >
> >> Hi all,
> >>
> >> As discussed in HADOOP-11123, we stop fixing incompatibilities of JDK9
> and
> >> JDK10, and would like to focus on supporting JDK11.
> >> This week, I plan to close HADOOP-11123(Support Java 9) and
> >> HADOOP-11423(Support Java 10) as duplicate/superceded, and move the
> >> unresolved sub-tasks to under HADOOP-15338(Support Java 11).
> >>
> >> If there are any opinions, please reply this email or comment on the
> jiras.
> >> Any help for supporting upcoming JDK would be highly appreciated.
> >>
> >> Regards,
> >> - Takanobu
> >>
>
>
Re: About supporting upcoming java versions
Posted by Owen O'Malley <ow...@gmail.com>.
I’m less convinced. I’ve hit at least one pretty serious JVM bug in JDK 11, so there is value in enabling JDK 9 and 10 even with blockers on JDK 11.
That said, I’d be surprised if the work items for JDK 9 and 10 aren’t a strict subset of the issues getting to JDK 11.
.. Owen
> On Nov 5, 2018, at 6:14 AM, Kihwal Lee <ki...@oath.com.INVALID> wrote:
>
> +1 It makes sense to converge on the LTS version.
> Thanks for the initiative.
>
> 73
> Kihwal
>
> On Sun, Nov 4, 2018 at 10:10 PM Takanobu Asanuma <ta...@apache.org>
> wrote:
>
>> Hi all,
>>
>> As discussed in HADOOP-11123, we stop fixing incompatibilities of JDK9 and
>> JDK10, and would like to focus on supporting JDK11.
>> This week, I plan to close HADOOP-11123(Support Java 9) and
>> HADOOP-11423(Support Java 10) as duplicate/superceded, and move the
>> unresolved sub-tasks to under HADOOP-15338(Support Java 11).
>>
>> If there are any opinions, please reply this email or comment on the jiras.
>> Any help for supporting upcoming JDK would be highly appreciated.
>>
>> Regards,
>> - Takanobu
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: About supporting upcoming java versions
Posted by Kihwal Lee <ki...@oath.com.INVALID>.
+1 It makes sense to converge on the LTS version.
Thanks for the initiative.
73
Kihwal
On Sun, Nov 4, 2018 at 10:10 PM Takanobu Asanuma <ta...@apache.org>
wrote:
> Hi all,
>
> As discussed in HADOOP-11123, we stop fixing incompatibilities of JDK9 and
> JDK10, and would like to focus on supporting JDK11.
> This week, I plan to close HADOOP-11123(Support Java 9) and
> HADOOP-11423(Support Java 10) as duplicate/superceded, and move the
> unresolved sub-tasks to under HADOOP-15338(Support Java 11).
>
> If there are any opinions, please reply this email or comment on the jiras.
> Any help for supporting upcoming JDK would be highly appreciated.
>
> Regards,
> - Takanobu
>