You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Tsuyoshi Ozawa <oz...@apache.org> on 2015/10/06 02:35:51 UTC
[DISCUSS] About the details of JDK-8 support
Hi commiters and users of Hadoop stack,
I’ll share the current status of JDK-8 support here. We can take a
two-step approach to support JDK-8 - runtime-level support and
source-level support.
About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
long as I tested, it works well completely since JDK-8 doesn’t have
any incompatibility at binary level. We can say Hadoop has supported
JDK8 runtime already. Do you have any concern about this? I’ve not
tested with HBase yet. I need help of HBase community. I think only
problem about runtime is HADOOP-11364, default value of
colntainer-killer of YARN. After fixing the issue, we can declare the
support of JDK on Hadoop Wiki to make it clear for users.
https://wiki.apache.org/hadoop/HadoopJavaVersions
About source-level, however, we have one big problem - upgrading
dependency of asm and cglib. We need to upgrade all libraries which
depends on asm to support new byte code introduced in JDK8[1]. The
dependencies which uses asm are jersey-server for compile and provide
scope, and cglib for test scope(I checked it with mvn dependency:tree
command). HADOOP-9613 is addressing the problem.
One complex problem I’ve faced is Jersey depends on grizzly - to
upgrade jersey to 1.19, which supports JDK8,
we need to upgrade grizzly to 2.2.16 to use
jersey-test-framework-grizzly2. I’d like to discuss which version we
will target this change. Can we do this in branch-2? Should we take
care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
confirm whether HADOOP-11993 means to remove Jersey, which depends on
asm, or not. I think we can collaborate with Yetus community here.
Also, another simple problem is that source code cannot be compiled
because javadoc format or variable identifier are illegal(e.g.
HADOOP-12457, HADOOP-11875). I think this can be solved
straightforwardly.
Please share any concern I’ve missed. The opinions of users are also welcome :-)
I'd like to go forward this step by step to make Hadoop user friendly.
Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
JDK-8.
Best regards,
- Tsuyoshi
[1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
[2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by SasakiKai <le...@me.com>.
Hello
Thank you for sharing information about JDK support discussion. I'd like to share what I know now.
In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed.
And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment.
Regard
Kai Sasaki
Sent from my tiny typewriter.
> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
>
> https://github.com/google/guice/wiki/Guice40
>
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
>
> Best regards,
> - Tsuyoshi
>
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> Hi commiters and users of Hadoop stack,
>>
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>>
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>>
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>>
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>>
>> Please share any concern I’ve missed. The opinions of users are also welcome :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>>
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>>
>> Best regards,
>> - Tsuyoshi
>>
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by SasakiKai <le...@me.com>.
Hello
Thank you for sharing information about JDK support discussion. I'd like to share what I know now.
In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed.
And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment.
Regard
Kai Sasaki
Sent from my tiny typewriter.
> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
>
> https://github.com/google/guice/wiki/Guice40
>
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
>
> Best regards,
> - Tsuyoshi
>
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> Hi commiters and users of Hadoop stack,
>>
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>>
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>>
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>>
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>>
>> Please share any concern I’ve missed. The opinions of users are also welcome :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>>
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>>
>> Best regards,
>> - Tsuyoshi
>>
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by SasakiKai <le...@me.com>.
Hello
Thank you for sharing information about JDK support discussion. I'd like to share what I know now.
In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed.
And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment.
Regard
Kai Sasaki
Sent from my tiny typewriter.
> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
>
> https://github.com/google/guice/wiki/Guice40
>
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
>
> Best regards,
> - Tsuyoshi
>
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> Hi commiters and users of Hadoop stack,
>>
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>>
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>>
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>>
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>>
>> Please share any concern I’ve missed. The opinions of users are also welcome :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>>
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>>
>> Best regards,
>> - Tsuyoshi
>>
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by SasakiKai <le...@me.com>.
Hello
Thank you for sharing information about JDK support discussion. I'd like to share what I know now.
In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed.
And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment.
Regard
Kai Sasaki
Sent from my tiny typewriter.
> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
>
> https://github.com/google/guice/wiki/Guice40
>
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
>
> Best regards,
> - Tsuyoshi
>
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> Hi commiters and users of Hadoop stack,
>>
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>>
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>>
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>>
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>>
>> Please share any concern I’ve missed. The opinions of users are also welcome :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>>
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>>
>> Best regards,
>> - Tsuyoshi
>>
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by SasakiKai <le...@me.com>.
Hello
Thank you for sharing information about JDK support discussion. I'd like to share what I know now.
In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed.
And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment.
Regard
Kai Sasaki
Sent from my tiny typewriter.
> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
>
> https://github.com/google/guice/wiki/Guice40
>
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
>
> Best regards,
> - Tsuyoshi
>
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> Hi commiters and users of Hadoop stack,
>>
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>>
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>>
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>>
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>>
>> Please share any concern I’ve missed. The opinions of users are also welcome :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>>
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>>
>> Best regards,
>> - Tsuyoshi
>>
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by SasakiKai <le...@me.com>.
Hello
Thank you for sharing information about JDK support discussion. I'd like to share what I know now.
In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed.
And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment.
Regard
Kai Sasaki
Sent from my tiny typewriter.
> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
>
> https://github.com/google/guice/wiki/Guice40
>
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
>
> Best regards,
> - Tsuyoshi
>
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> Hi commiters and users of Hadoop stack,
>>
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>>
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>>
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>>
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>>
>> Please share any concern I’ve missed. The opinions of users are also welcome :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>>
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>>
>> Best regards,
>> - Tsuyoshi
>>
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by SasakiKai <le...@me.com>.
Hello
Thank you for sharing information about JDK support discussion. I'd like to share what I know now.
In terms of runtime level support, we've tried JDK8 with HDP2.3. Although of course there may be several diffs between HDP and upstream(trunk, branch-2), it is generally working well. We are running hadoop, mapreduce, yarn, hdfs, hive(HS2, command line). So I also think runtime level support of JDK8 is generally completed.
And new features such as Hive LLAP or HDFS erasure coding also work well currently from the viewpoint of runtime environment.
Regard
Kai Sasaki
Sent from my tiny typewriter.
> On Oct 6, 2015, at 3:43 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> I forgot to mention - we also need to upgrade Google Guice to 0.4.0
> for source-level JDK8 support in addition to asm and cglib.
>
> https://github.com/google/guice/wiki/Guice40
>
>> Some highlights:
>> Java8 runtime compatibility for Guice core & all extensions.
>
> Best regards,
> - Tsuyoshi
>
>> On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> Hi commiters and users of Hadoop stack,
>>
>> I’ll share the current status of JDK-8 support here. We can take a
>> two-step approach to support JDK-8 - runtime-level support and
>> source-level support.
>>
>> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
>> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
>> long as I tested, it works well completely since JDK-8 doesn’t have
>> any incompatibility at binary level. We can say Hadoop has supported
>> JDK8 runtime already. Do you have any concern about this? I’ve not
>> tested with HBase yet. I need help of HBase community. I think only
>> problem about runtime is HADOOP-11364, default value of
>> colntainer-killer of YARN. After fixing the issue, we can declare the
>> support of JDK on Hadoop Wiki to make it clear for users.
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About source-level, however, we have one big problem - upgrading
>> dependency of asm and cglib. We need to upgrade all libraries which
>> depends on asm to support new byte code introduced in JDK8[1]. The
>> dependencies which uses asm are jersey-server for compile and provide
>> scope, and cglib for test scope(I checked it with mvn dependency:tree
>> command). HADOOP-9613 is addressing the problem.
>>
>> One complex problem I’ve faced is Jersey depends on grizzly - to
>> upgrade jersey to 1.19, which supports JDK8,
>> we need to upgrade grizzly to 2.2.16 to use
>> jersey-test-framework-grizzly2. I’d like to discuss which version we
>> will target this change. Can we do this in branch-2? Should we take
>> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
>> confirm whether HADOOP-11993 means to remove Jersey, which depends on
>> asm, or not. I think we can collaborate with Yetus community here.
>>
>> Also, another simple problem is that source code cannot be compiled
>> because javadoc format or variable identifier are illegal(e.g.
>> HADOOP-12457, HADOOP-11875). I think this can be solved
>> straightforwardly.
>>
>> Please share any concern I’ve missed. The opinions of users are also welcome :-)
>> I'd like to go forward this step by step to make Hadoop user friendly.
>>
>> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
>> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
>> JDK-8.
>>
>> Best regards,
>> - Tsuyoshi
>>
>> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
>> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
+1
On Thu, Oct 15, 2015 at 7:57 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
> >
> > On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com>
> wrote:
> >
> >>
> >> If people want, I could setup a cut off of yetus master to run the
> jenkins
> >> test-patch. (multiple maven repos, docker support, multijdk support, …
> )
> >> Yetus would get some real world testing out of it and hadoop common-dev
> >> could stop spinning in circles over some of the same issues month after
> >> month. ;)
> >>
> >
> > Seems like a step in the right direction.
> >
> > Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before
> monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Oct 17, 2015, at 8:55 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>
> thx for doing this, I'll keep an eye on the builds
Thanks for helping out too. You never know how software actually runs in the wild. I’m already hitting bugs that didn’t show up in testing. *sigh*
> one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
That’s a great question. The directory isn’t labeled and interrogating the JDK isn’t being helpful either. Here’s the first ever run from Yetus…
https://issues.apache.org/jira/browse/HADOOP-11820?focusedCommentId=14961948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14961948
Unfortunately, I hit a bug when I enabled Docker mode a bit ago brought on by a recent code change. Until that gets fixed, we’re sort of at the mercy of what’s on the Jenkins box.
FWIW, I’m in the process of testing the per-instance maven repos now. While I’m there, I think I'm going to bump the JDK7 version up to something newer (we’re on 55, go to 79).
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Oct 17, 2015, at 8:55 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>
> thx for doing this, I'll keep an eye on the builds
Thanks for helping out too. You never know how software actually runs in the wild. I’m already hitting bugs that didn’t show up in testing. *sigh*
> one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
That’s a great question. The directory isn’t labeled and interrogating the JDK isn’t being helpful either. Here’s the first ever run from Yetus…
https://issues.apache.org/jira/browse/HADOOP-11820?focusedCommentId=14961948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14961948
Unfortunately, I hit a bug when I enabled Docker mode a bit ago brought on by a recent code change. Until that gets fixed, we’re sort of at the mercy of what’s on the Jenkins box.
FWIW, I’m in the process of testing the per-instance maven repos now. While I’m there, I think I'm going to bump the JDK7 version up to something newer (we’re on 55, go to 79).
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Oct 17, 2015, at 8:55 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>
> thx for doing this, I'll keep an eye on the builds
Thanks for helping out too. You never know how software actually runs in the wild. I’m already hitting bugs that didn’t show up in testing. *sigh*
> one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
That’s a great question. The directory isn’t labeled and interrogating the JDK isn’t being helpful either. Here’s the first ever run from Yetus…
https://issues.apache.org/jira/browse/HADOOP-11820?focusedCommentId=14961948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14961948
Unfortunately, I hit a bug when I enabled Docker mode a bit ago brought on by a recent code change. Until that gets fixed, we’re sort of at the mercy of what’s on the Jenkins box.
FWIW, I’m in the process of testing the per-instance maven repos now. While I’m there, I think I'm going to bump the JDK7 version up to something newer (we’re on 55, go to 79).
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
On Oct 17, 2015, at 8:55 AM, Steve Loughran <st...@hortonworks.com> wrote:
>>
>
> thx for doing this, I'll keep an eye on the builds
Thanks for helping out too. You never know how software actually runs in the wild. I’m already hitting bugs that didn’t show up in testing. *sigh*
> one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
That’s a great question. The directory isn’t labeled and interrogating the JDK isn’t being helpful either. Here’s the first ever run from Yetus…
https://issues.apache.org/jira/browse/HADOOP-11820?focusedCommentId=14961948&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14961948
Unfortunately, I hit a bug when I enabled Docker mode a bit ago brought on by a recent code change. Until that gets fixed, we’re sort of at the mercy of what’s on the Jenkins box.
FWIW, I’m in the process of testing the per-instance maven repos now. While I’m there, I think I'm going to bump the JDK7 version up to something newer (we’re on 55, go to 79).
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 17 Oct 2015, at 15:42, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
>
>
> FYI:
>
> I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
>
thx for doing this, I'll keep an eye on the builds
one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
> At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
>
> I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
>
> Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 17 Oct 2015, at 15:42, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
>
>
> FYI:
>
> I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
>
thx for doing this, I'll keep an eye on the builds
one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
> At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
>
> I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
>
> Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 17 Oct 2015, at 15:42, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
>
>
> FYI:
>
> I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
>
thx for doing this, I'll keep an eye on the builds
one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
> At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
>
> I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
>
> Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 17 Oct 2015, at 15:42, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
>
>
> FYI:
>
> I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
>
thx for doing this, I'll keep an eye on the builds
one question: which jdk8? Some of the Hadoop builds (example YARN) are crashing the JVM..there's a whole list of jdk8 options, including, for ubuntu hosts only, openjdk. Should we switch the jvm for our nightly builds to open jdk 8?
> At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
>
> I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
>
> Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
FYI:
I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
FYI:
I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
FYI:
I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
FYI:
I’ve switched HADOOP to Apache Yetus (from the YETUS-83 branch) in multijdk mode with jdk7 and jdk8 turned on. (So java patches will take *twice as long* since they aren’t done in parallel!). I’ll be ramping up the other big features (docker and multiple maven repos) and watching the results of various jobs throughout the day. Note that the Jenkins precommit bits are now a bit hacky for the time being so that if necessary I can easily back it out in case of a big bug.
At some point, we’ll switch the other projects over, dealing with any feedback from the hadoop-common trial. Hopefully by then, we’ll have our first Apache Yetus release as well as clean up the jobs to be less hacky.
I’ll drop some notes on the new functionality later after I see how far I can ramp up the features.
Thanks.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
OK. I will summarise it on HADOOP-11090.
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
Thanks very much, Allen.
> Seems like a step in the right direction.
> Should we expect a downtime and is there a good/bad time to do this?
>
> Weekends are the times to work on Jenkins
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
> Two days time, then?
+1
> sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
> This has been fixed in jdk8u60.
>
> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
Thank for the sharing, Kihwal. We should describe it on Wiki and
documents. I'll do it based on information based on this thread.
Thanks,
- Tsuyoshi
On Thu, Oct 15, 2015 at 11:57 PM, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>>
>> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>>>
>>> If people want, I could setup a cut off of yetus master to run the jenkins
>>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>>> Yetus would get some real world testing out of it and hadoop common-dev
>>> could stop spinning in circles over some of the same issues month after
>>> month. ;)
>>>
>>
>> Seems like a step in the right direction.
>>
>> Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
+1
On Thu, Oct 15, 2015 at 7:57 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
> >
> > On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com>
> wrote:
> >
> >>
> >> If people want, I could setup a cut off of yetus master to run the
> jenkins
> >> test-patch. (multiple maven repos, docker support, multijdk support, …
> )
> >> Yetus would get some real world testing out of it and hadoop common-dev
> >> could stop spinning in circles over some of the same issues month after
> >> month. ;)
> >>
> >
> > Seems like a step in the right direction.
> >
> > Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before
> monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
+1
On Thu, Oct 15, 2015 at 7:57 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
> >
> > On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com>
> wrote:
> >
> >>
> >> If people want, I could setup a cut off of yetus master to run the
> jenkins
> >> test-patch. (multiple maven repos, docker support, multijdk support, …
> )
> >> Yetus would get some real world testing out of it and hadoop common-dev
> >> could stop spinning in circles over some of the same issues month after
> >> month. ;)
> >>
> >
> > Seems like a step in the right direction.
> >
> > Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before
> monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
OK. I will summarise it on HADOOP-11090.
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
Thanks very much, Allen.
> Seems like a step in the right direction.
> Should we expect a downtime and is there a good/bad time to do this?
>
> Weekends are the times to work on Jenkins
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
> Two days time, then?
+1
> sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
> This has been fixed in jdk8u60.
>
> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
Thank for the sharing, Kihwal. We should describe it on Wiki and
documents. I'll do it based on information based on this thread.
Thanks,
- Tsuyoshi
On Thu, Oct 15, 2015 at 11:57 PM, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>>
>> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>>>
>>> If people want, I could setup a cut off of yetus master to run the jenkins
>>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>>> Yetus would get some real world testing out of it and hadoop common-dev
>>> could stop spinning in circles over some of the same issues month after
>>> month. ;)
>>>
>>
>> Seems like a step in the right direction.
>>
>> Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
+1
On Thu, Oct 15, 2015 at 7:57 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
> >
> > On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com>
> wrote:
> >
> >>
> >> If people want, I could setup a cut off of yetus master to run the
> jenkins
> >> test-patch. (multiple maven repos, docker support, multijdk support, …
> )
> >> Yetus would get some real world testing out of it and hadoop common-dev
> >> could stop spinning in circles over some of the same issues month after
> >> month. ;)
> >>
> >
> > Seems like a step in the right direction.
> >
> > Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before
> monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
OK. I will summarise it on HADOOP-11090.
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
Thanks very much, Allen.
> Seems like a step in the right direction.
> Should we expect a downtime and is there a good/bad time to do this?
>
> Weekends are the times to work on Jenkins
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
> Two days time, then?
+1
> sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
> This has been fixed in jdk8u60.
>
> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
Thank for the sharing, Kihwal. We should describe it on Wiki and
documents. I'll do it based on information based on this thread.
Thanks,
- Tsuyoshi
On Thu, Oct 15, 2015 at 11:57 PM, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>>
>> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>>>
>>> If people want, I could setup a cut off of yetus master to run the jenkins
>>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>>> Yetus would get some real world testing out of it and hadoop common-dev
>>> could stop spinning in circles over some of the same issues month after
>>> month. ;)
>>>
>>
>> Seems like a step in the right direction.
>>
>> Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
OK. I will summarise it on HADOOP-11090.
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
Thanks very much, Allen.
> Seems like a step in the right direction.
> Should we expect a downtime and is there a good/bad time to do this?
>
> Weekends are the times to work on Jenkins
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
> Two days time, then?
+1
> sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
> This has been fixed in jdk8u60.
>
> http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
Thank for the sharing, Kihwal. We should describe it on Wiki and
documents. I'll do it based on information based on this thread.
Thanks,
- Tsuyoshi
On Thu, Oct 15, 2015 at 11:57 PM, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>>
>> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>>
>>>
>>> If people want, I could setup a cut off of yetus master to run the jenkins
>>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>>> Yetus would get some real world testing out of it and hadoop common-dev
>>> could stop spinning in circles over some of the same issues month after
>>> month. ;)
>>>
>>
>> Seems like a step in the right direction.
>>
>> Should we expect a downtime and is there a good/bad time to do this?
>
>
>
> Weekends are the times to work on Jenkins
>
> do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
>
> Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>
> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>>
>> If people want, I could setup a cut off of yetus master to run the jenkins
>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>> Yetus would get some real world testing out of it and hadoop common-dev
>> could stop spinning in circles over some of the same issues month after
>> month. ;)
>>
>
> Seems like a step in the right direction.
>
> Should we expect a downtime and is there a good/bad time to do this?
Weekends are the times to work on Jenkins
do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>
> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>>
>> If people want, I could setup a cut off of yetus master to run the jenkins
>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>> Yetus would get some real world testing out of it and hadoop common-dev
>> could stop spinning in circles over some of the same issues month after
>> month. ;)
>>
>
> Seems like a step in the right direction.
>
> Should we expect a downtime and is there a good/bad time to do this?
Weekends are the times to work on Jenkins
do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>
> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>>
>> If people want, I could setup a cut off of yetus master to run the jenkins
>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>> Yetus would get some real world testing out of it and hadoop common-dev
>> could stop spinning in circles over some of the same issues month after
>> month. ;)
>>
>
> Seems like a step in the right direction.
>
> Should we expect a downtime and is there a good/bad time to do this?
Weekends are the times to work on Jenkins
do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 14:42, Karthik Kambatla <ka...@cloudera.com> wrote:
>
> On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>>
>> If people want, I could setup a cut off of yetus master to run the jenkins
>> test-patch. (multiple maven repos, docker support, multijdk support, … )
>> Yetus would get some real world testing out of it and hadoop common-dev
>> could stop spinning in circles over some of the same issues month after
>> month. ;)
>>
>
> Seems like a step in the right direction.
>
> Should we expect a downtime and is there a good/bad time to do this?
Weekends are the times to work on Jenkins
do it on a saturday morning (PST) and the'res 24-48 h to stabilise before monday morning PST.
Two days time, then?
Re: [DISCUSS] About the details of JDK-8 support
Posted by Karthik Kambatla <ka...@cloudera.com>.
On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
>
Seems like a step in the right direction.
Should we expect a downtime and is there a good/bad time to do this?
>
>
> On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
> > The only problem with trying to get the JDK 8 trunk builds green (or
> blue I
> > guess) is that it's like trying to hit a moving target because of how
> many
> > new commits keep coming in. I was looking at fixing these a while ago,
> and
> > managed to at least make them compile and fixed (or worked with others to
> > fix) some of the unit tests. I've been really busy on other tasks and
> > haven't had time to continue working on this in quite a while though.
> >
> > Currently, it looks like Common is still green mostly, Yarn is having a
> > build failure with checkstyle, MR has between 1 and 10 test failures, and
> > HDFS had between 3 and 10 test failures.
> >
> > I think it's going to be difficult to get these green, and to keep them
> > green, unless we get more buy in from everyone on new commits being
> tested
> > against JDK 8. Otherwise, it's too hard to keep up with the number of
> > commits coming in, even if we do get it green. Perhaps we could have
> > test-patch also run the patch against JDK 8?
> >
> >
> > - Robert
> >
> > On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> >>
> >>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >>>
> >>> Just to echo Steve's idea -- if we're seriously considering supporting
> >>> JDK 8, maybe the first thing to do is to set up the jenkins to run
> >>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> >>> need to play around with all the Jenkins knob?
> >>
> >> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> >> the Java 7 build, which I will happily do. The POM can be changed to
> set
> >> the minimum JVM version -though that's most likely to be visible to
> people
> >> building locally, as you'll need to make sure that you have access to
> java
> >> 7 and java 8 JVMs if you want to build and test for both.
> >>
> >> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> >> failing an not enough people are caring
> >>
> >>
> >>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
> >>
> >> Please, lets fix this
> >>
> >>
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 00:28, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
> If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
>
>
+1.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 00:28, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
> If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
>
>
+1.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Karthik Kambatla <ka...@cloudera.com>.
On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
>
Seems like a step in the right direction.
Should we expect a downtime and is there a good/bad time to do this?
>
>
> On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
> > The only problem with trying to get the JDK 8 trunk builds green (or
> blue I
> > guess) is that it's like trying to hit a moving target because of how
> many
> > new commits keep coming in. I was looking at fixing these a while ago,
> and
> > managed to at least make them compile and fixed (or worked with others to
> > fix) some of the unit tests. I've been really busy on other tasks and
> > haven't had time to continue working on this in quite a while though.
> >
> > Currently, it looks like Common is still green mostly, Yarn is having a
> > build failure with checkstyle, MR has between 1 and 10 test failures, and
> > HDFS had between 3 and 10 test failures.
> >
> > I think it's going to be difficult to get these green, and to keep them
> > green, unless we get more buy in from everyone on new commits being
> tested
> > against JDK 8. Otherwise, it's too hard to keep up with the number of
> > commits coming in, even if we do get it green. Perhaps we could have
> > test-patch also run the patch against JDK 8?
> >
> >
> > - Robert
> >
> > On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> >>
> >>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >>>
> >>> Just to echo Steve's idea -- if we're seriously considering supporting
> >>> JDK 8, maybe the first thing to do is to set up the jenkins to run
> >>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> >>> need to play around with all the Jenkins knob?
> >>
> >> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> >> the Java 7 build, which I will happily do. The POM can be changed to
> set
> >> the minimum JVM version -though that's most likely to be visible to
> people
> >> building locally, as you'll need to make sure that you have access to
> java
> >> 7 and java 8 JVMs if you want to build and test for both.
> >>
> >> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> >> failing an not enough people are caring
> >>
> >>
> >>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
> >>
> >> Please, lets fix this
> >>
> >>
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 00:28, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
> If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
>
>
+1.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Karthik Kambatla <ka...@cloudera.com>.
On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
>
Seems like a step in the right direction.
Should we expect a downtime and is there a good/bad time to do this?
>
>
> On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
> > The only problem with trying to get the JDK 8 trunk builds green (or
> blue I
> > guess) is that it's like trying to hit a moving target because of how
> many
> > new commits keep coming in. I was looking at fixing these a while ago,
> and
> > managed to at least make them compile and fixed (or worked with others to
> > fix) some of the unit tests. I've been really busy on other tasks and
> > haven't had time to continue working on this in quite a while though.
> >
> > Currently, it looks like Common is still green mostly, Yarn is having a
> > build failure with checkstyle, MR has between 1 and 10 test failures, and
> > HDFS had between 3 and 10 test failures.
> >
> > I think it's going to be difficult to get these green, and to keep them
> > green, unless we get more buy in from everyone on new commits being
> tested
> > against JDK 8. Otherwise, it's too hard to keep up with the number of
> > commits coming in, even if we do get it green. Perhaps we could have
> > test-patch also run the patch against JDK 8?
> >
> >
> > - Robert
> >
> > On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> >>
> >>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >>>
> >>> Just to echo Steve's idea -- if we're seriously considering supporting
> >>> JDK 8, maybe the first thing to do is to set up the jenkins to run
> >>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> >>> need to play around with all the Jenkins knob?
> >>
> >> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> >> the Java 7 build, which I will happily do. The POM can be changed to
> set
> >> the minimum JVM version -though that's most likely to be visible to
> people
> >> building locally, as you'll need to make sure that you have access to
> java
> >> 7 and java 8 JVMs if you want to build and test for both.
> >>
> >> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> >> failing an not enough people are caring
> >>
> >>
> >>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
> >>
> >> Please, lets fix this
> >>
> >>
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Oct 2015, at 00:28, Allen Wittenauer <aw...@altiscale.com> wrote:
>
>
> If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
>
>
+1.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Karthik Kambatla <ka...@cloudera.com>.
On Wed, Oct 14, 2015 at 4:28 PM, Allen Wittenauer <aw...@altiscale.com> wrote:
>
> If people want, I could setup a cut off of yetus master to run the jenkins
> test-patch. (multiple maven repos, docker support, multijdk support, … )
> Yetus would get some real world testing out of it and hadoop common-dev
> could stop spinning in circles over some of the same issues month after
> month. ;)
>
Seems like a step in the right direction.
Should we expect a downtime and is there a good/bad time to do this?
>
>
> On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
> > The only problem with trying to get the JDK 8 trunk builds green (or
> blue I
> > guess) is that it's like trying to hit a moving target because of how
> many
> > new commits keep coming in. I was looking at fixing these a while ago,
> and
> > managed to at least make them compile and fixed (or worked with others to
> > fix) some of the unit tests. I've been really busy on other tasks and
> > haven't had time to continue working on this in quite a while though.
> >
> > Currently, it looks like Common is still green mostly, Yarn is having a
> > build failure with checkstyle, MR has between 1 and 10 test failures, and
> > HDFS had between 3 and 10 test failures.
> >
> > I think it's going to be difficult to get these green, and to keep them
> > green, unless we get more buy in from everyone on new commits being
> tested
> > against JDK 8. Otherwise, it's too hard to keep up with the number of
> > commits coming in, even if we do get it green. Perhaps we could have
> > test-patch also run the patch against JDK 8?
> >
> >
> > - Robert
> >
> > On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> > wrote:
> >
> >>
> >>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >>>
> >>> Just to echo Steve's idea -- if we're seriously considering supporting
> >>> JDK 8, maybe the first thing to do is to set up the jenkins to run
> >>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> >>> need to play around with all the Jenkins knob?
> >>
> >> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> >> the Java 7 build, which I will happily do. The POM can be changed to
> set
> >> the minimum JVM version -though that's most likely to be visible to
> people
> >> building locally, as you'll need to make sure that you have access to
> java
> >> 7 and java 8 JVMs if you want to build and test for both.
> >>
> >> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> >> failing an not enough people are caring
> >>
> >>
> >>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
> >>
> >> Please, lets fix this
> >>
> >>
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
> The only problem with trying to get the JDK 8 trunk builds green (or blue I
> guess) is that it's like trying to hit a moving target because of how many
> new commits keep coming in. I was looking at fixing these a while ago, and
> managed to at least make them compile and fixed (or worked with others to
> fix) some of the unit tests. I've been really busy on other tasks and
> haven't had time to continue working on this in quite a while though.
>
> Currently, it looks like Common is still green mostly, Yarn is having a
> build failure with checkstyle, MR has between 1 and 10 test failures, and
> HDFS had between 3 and 10 test failures.
>
> I think it's going to be difficult to get these green, and to keep them
> green, unless we get more buy in from everyone on new commits being tested
> against JDK 8. Otherwise, it's too hard to keep up with the number of
> commits coming in, even if we do get it green. Perhaps we could have
> test-patch also run the patch against JDK 8?
>
>
> - Robert
>
> On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>>>
>>> Just to echo Steve's idea -- if we're seriously considering supporting
>>> JDK 8, maybe the first thing to do is to set up the jenkins to run
>>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
>>> need to play around with all the Jenkins knob?
>>
>> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
>> the Java 7 build, which I will happily do. The POM can be changed to set
>> the minimum JVM version -though that's most likely to be visible to people
>> building locally, as you'll need to make sure that you have access to java
>> 7 and java 8 JVMs if you want to build and test for both.
>>
>> Jenkins-wise, the big issue is one I've mentioned before: the builds are
>> failing an not enough people are caring
>>
>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>>
>> Please, lets fix this
>>
>>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
> The only problem with trying to get the JDK 8 trunk builds green (or blue I
> guess) is that it's like trying to hit a moving target because of how many
> new commits keep coming in. I was looking at fixing these a while ago, and
> managed to at least make them compile and fixed (or worked with others to
> fix) some of the unit tests. I've been really busy on other tasks and
> haven't had time to continue working on this in quite a while though.
>
> Currently, it looks like Common is still green mostly, Yarn is having a
> build failure with checkstyle, MR has between 1 and 10 test failures, and
> HDFS had between 3 and 10 test failures.
>
> I think it's going to be difficult to get these green, and to keep them
> green, unless we get more buy in from everyone on new commits being tested
> against JDK 8. Otherwise, it's too hard to keep up with the number of
> commits coming in, even if we do get it green. Perhaps we could have
> test-patch also run the patch against JDK 8?
>
>
> - Robert
>
> On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>>>
>>> Just to echo Steve's idea -- if we're seriously considering supporting
>>> JDK 8, maybe the first thing to do is to set up the jenkins to run
>>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
>>> need to play around with all the Jenkins knob?
>>
>> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
>> the Java 7 build, which I will happily do. The POM can be changed to set
>> the minimum JVM version -though that's most likely to be visible to people
>> building locally, as you'll need to make sure that you have access to java
>> 7 and java 8 JVMs if you want to build and test for both.
>>
>> Jenkins-wise, the big issue is one I've mentioned before: the builds are
>> failing an not enough people are caring
>>
>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>>
>> Please, lets fix this
>>
>>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
> The only problem with trying to get the JDK 8 trunk builds green (or blue I
> guess) is that it's like trying to hit a moving target because of how many
> new commits keep coming in. I was looking at fixing these a while ago, and
> managed to at least make them compile and fixed (or worked with others to
> fix) some of the unit tests. I've been really busy on other tasks and
> haven't had time to continue working on this in quite a while though.
>
> Currently, it looks like Common is still green mostly, Yarn is having a
> build failure with checkstyle, MR has between 1 and 10 test failures, and
> HDFS had between 3 and 10 test failures.
>
> I think it's going to be difficult to get these green, and to keep them
> green, unless we get more buy in from everyone on new commits being tested
> against JDK 8. Otherwise, it's too hard to keep up with the number of
> commits coming in, even if we do get it green. Perhaps we could have
> test-patch also run the patch against JDK 8?
>
>
> - Robert
>
> On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>>>
>>> Just to echo Steve's idea -- if we're seriously considering supporting
>>> JDK 8, maybe the first thing to do is to set up the jenkins to run
>>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
>>> need to play around with all the Jenkins knob?
>>
>> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
>> the Java 7 build, which I will happily do. The POM can be changed to set
>> the minimum JVM version -though that's most likely to be visible to people
>> building locally, as you'll need to make sure that you have access to java
>> 7 and java 8 JVMs if you want to build and test for both.
>>
>> Jenkins-wise, the big issue is one I've mentioned before: the builds are
>> failing an not enough people are caring
>>
>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>>
>> Please, lets fix this
>>
>>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Allen Wittenauer <aw...@altiscale.com>.
If people want, I could setup a cut off of yetus master to run the jenkins test-patch. (multiple maven repos, docker support, multijdk support, … ) Yetus would get some real world testing out of it and hadoop common-dev could stop spinning in circles over some of the same issues month after month. ;)
On Oct 14, 2015, at 3:05 PM, Robert Kanter <rk...@cloudera.com> wrote:
> The only problem with trying to get the JDK 8 trunk builds green (or blue I
> guess) is that it's like trying to hit a moving target because of how many
> new commits keep coming in. I was looking at fixing these a while ago, and
> managed to at least make them compile and fixed (or worked with others to
> fix) some of the unit tests. I've been really busy on other tasks and
> haven't had time to continue working on this in quite a while though.
>
> Currently, it looks like Common is still green mostly, Yarn is having a
> build failure with checkstyle, MR has between 1 and 10 test failures, and
> HDFS had between 3 and 10 test failures.
>
> I think it's going to be difficult to get these green, and to keep them
> green, unless we get more buy in from everyone on new commits being tested
> against JDK 8. Otherwise, it's too hard to keep up with the number of
> commits coming in, even if we do get it green. Perhaps we could have
> test-patch also run the patch against JDK 8?
>
>
> - Robert
>
> On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>>> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>>>
>>> Just to echo Steve's idea -- if we're seriously considering supporting
>>> JDK 8, maybe the first thing to do is to set up the jenkins to run
>>> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
>>> need to play around with all the Jenkins knob?
>>
>> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
>> the Java 7 build, which I will happily do. The POM can be changed to set
>> the minimum JVM version -though that's most likely to be visible to people
>> building locally, as you'll need to make sure that you have access to java
>> 7 and java 8 JVMs if you want to build and test for both.
>>
>> Jenkins-wise, the big issue is one I've mentioned before: the builds are
>> failing an not enough people are caring
>>
>>
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>>
>> Please, lets fix this
>>
>>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Robert Kanter <rk...@cloudera.com>.
The only problem with trying to get the JDK 8 trunk builds green (or blue I
guess) is that it's like trying to hit a moving target because of how many
new commits keep coming in. I was looking at fixing these a while ago, and
managed to at least make them compile and fixed (or worked with others to
fix) some of the unit tests. I've been really busy on other tasks and
haven't had time to continue working on this in quite a while though.
Currently, it looks like Common is still green mostly, Yarn is having a
build failure with checkstyle, MR has between 1 and 10 test failures, and
HDFS had between 3 and 10 test failures.
I think it's going to be difficult to get these green, and to keep them
green, unless we get more buy in from everyone on new commits being tested
against JDK 8. Otherwise, it's too hard to keep up with the number of
commits coming in, even if we do get it green. Perhaps we could have
test-patch also run the patch against JDK 8?
- Robert
On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >
> > Just to echo Steve's idea -- if we're seriously considering supporting
> > JDK 8, maybe the first thing to do is to set up the jenkins to run
> > with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> > need to play around with all the Jenkins knob?
>
> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> the Java 7 build, which I will happily do. The POM can be changed to set
> the minimum JVM version -though that's most likely to be visible to people
> building locally, as you'll need to make sure that you have access to java
> 7 and java 8 JVMs if you want to build and test for both.
>
> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> failing an not enough people are caring
>
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>
> Please, lets fix this
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Robert Kanter <rk...@cloudera.com>.
The only problem with trying to get the JDK 8 trunk builds green (or blue I
guess) is that it's like trying to hit a moving target because of how many
new commits keep coming in. I was looking at fixing these a while ago, and
managed to at least make them compile and fixed (or worked with others to
fix) some of the unit tests. I've been really busy on other tasks and
haven't had time to continue working on this in quite a while though.
Currently, it looks like Common is still green mostly, Yarn is having a
build failure with checkstyle, MR has between 1 and 10 test failures, and
HDFS had between 3 and 10 test failures.
I think it's going to be difficult to get these green, and to keep them
green, unless we get more buy in from everyone on new commits being tested
against JDK 8. Otherwise, it's too hard to keep up with the number of
commits coming in, even if we do get it green. Perhaps we could have
test-patch also run the patch against JDK 8?
- Robert
On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >
> > Just to echo Steve's idea -- if we're seriously considering supporting
> > JDK 8, maybe the first thing to do is to set up the jenkins to run
> > with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> > need to play around with all the Jenkins knob?
>
> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> the Java 7 build, which I will happily do. The POM can be changed to set
> the minimum JVM version -though that's most likely to be visible to people
> building locally, as you'll need to make sure that you have access to java
> 7 and java 8 JVMs if you want to build and test for both.
>
> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> failing an not enough people are caring
>
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>
> Please, lets fix this
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Robert Kanter <rk...@cloudera.com>.
The only problem with trying to get the JDK 8 trunk builds green (or blue I
guess) is that it's like trying to hit a moving target because of how many
new commits keep coming in. I was looking at fixing these a while ago, and
managed to at least make them compile and fixed (or worked with others to
fix) some of the unit tests. I've been really busy on other tasks and
haven't had time to continue working on this in quite a while though.
Currently, it looks like Common is still green mostly, Yarn is having a
build failure with checkstyle, MR has between 1 and 10 test failures, and
HDFS had between 3 and 10 test failures.
I think it's going to be difficult to get these green, and to keep them
green, unless we get more buy in from everyone on new commits being tested
against JDK 8. Otherwise, it's too hard to keep up with the number of
commits coming in, even if we do get it green. Perhaps we could have
test-patch also run the patch against JDK 8?
- Robert
On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >
> > Just to echo Steve's idea -- if we're seriously considering supporting
> > JDK 8, maybe the first thing to do is to set up the jenkins to run
> > with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> > need to play around with all the Jenkins knob?
>
> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> the Java 7 build, which I will happily do. The POM can be changed to set
> the minimum JVM version -though that's most likely to be visible to people
> building locally, as you'll need to make sure that you have access to java
> 7 and java 8 JVMs if you want to build and test for both.
>
> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> failing an not enough people are caring
>
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>
> Please, lets fix this
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Robert Kanter <rk...@cloudera.com>.
The only problem with trying to get the JDK 8 trunk builds green (or blue I
guess) is that it's like trying to hit a moving target because of how many
new commits keep coming in. I was looking at fixing these a while ago, and
managed to at least make them compile and fixed (or worked with others to
fix) some of the unit tests. I've been really busy on other tasks and
haven't had time to continue working on this in quite a while though.
Currently, it looks like Common is still green mostly, Yarn is having a
build failure with checkstyle, MR has between 1 and 10 test failures, and
HDFS had between 3 and 10 test failures.
I think it's going to be difficult to get these green, and to keep them
green, unless we get more buy in from everyone on new commits being tested
against JDK 8. Otherwise, it's too hard to keep up with the number of
commits coming in, even if we do get it green. Perhaps we could have
test-patch also run the patch against JDK 8?
- Robert
On Wed, Oct 14, 2015 at 8:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
> >
> > Just to echo Steve's idea -- if we're seriously considering supporting
> > JDK 8, maybe the first thing to do is to set up the jenkins to run
> > with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> > need to play around with all the Jenkins knob?
>
> Jenkins is building with JAva 7 and 8. all that's needed is to turn off
> the Java 7 build, which I will happily do. The POM can be changed to set
> the minimum JVM version -though that's most likely to be visible to people
> building locally, as you'll need to make sure that you have access to java
> 7 and java 8 JVMs if you want to build and test for both.
>
> Jenkins-wise, the big issue is one I've mentioned before: the builds are
> failing an not enough people are caring
>
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
>
> Please, lets fix this
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>
> Just to echo Steve's idea -- if we're seriously considering supporting
> JDK 8, maybe the first thing to do is to set up the jenkins to run
> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> need to play around with all the Jenkins knob?
Jenkins is building with JAva 7 and 8. all that's needed is to turn off the Java 7 build, which I will happily do. The POM can be changed to set the minimum JVM version -though that's most likely to be visible to people building locally, as you'll need to make sure that you have access to java 7 and java 8 JVMs if you want to build and test for both.
Jenkins-wise, the big issue is one I've mentioned before: the builds are failing an not enough people are caring
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
Please, lets fix this
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>
> Just to echo Steve's idea -- if we're seriously considering supporting
> JDK 8, maybe the first thing to do is to set up the jenkins to run
> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> need to play around with all the Jenkins knob?
Jenkins is building with JAva 7 and 8. all that's needed is to turn off the Java 7 build, which I will happily do. The POM can be changed to set the minimum JVM version -though that's most likely to be visible to people building locally, as you'll need to make sure that you have access to java 7 and java 8 JVMs if you want to build and test for both.
Jenkins-wise, the big issue is one I've mentioned before: the builds are failing an not enough people are caring
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
Please, lets fix this
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>
> Just to echo Steve's idea -- if we're seriously considering supporting
> JDK 8, maybe the first thing to do is to set up the jenkins to run
> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> need to play around with all the Jenkins knob?
Jenkins is building with JAva 7 and 8. all that's needed is to turn off the Java 7 build, which I will happily do. The POM can be changed to set the minimum JVM version -though that's most likely to be visible to people building locally, as you'll need to make sure that you have access to java 7 and java 8 JVMs if you want to build and test for both.
Jenkins-wise, the big issue is one I've mentioned before: the builds are failing an not enough people are caring
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
Please, lets fix this
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 13 Oct 2015, at 17:32, Haohui Mai <ri...@gmail.com> wrote:
>
> Just to echo Steve's idea -- if we're seriously considering supporting
> JDK 8, maybe the first thing to do is to set up the jenkins to run
> with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
> need to play around with all the Jenkins knob?
Jenkins is building with JAva 7 and 8. all that's needed is to turn off the Java 7 build, which I will happily do. The POM can be changed to set the minimum JVM version -though that's most likely to be visible to people building locally, as you'll need to make sure that you have access to java 7 and java 8 JVMs if you want to build and test for both.
Jenkins-wise, the big issue is one I've mentioned before: the builds are failing an not enough people are caring
https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/488/
Please, lets fix this
Re: [DISCUSS] About the details of JDK-8 support
Posted by Haohui Mai <ri...@gmail.com>.
Just to echo Steve's idea -- if we're seriously considering supporting
JDK 8, maybe the first thing to do is to set up the jenkins to run
with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
need to play around with all the Jenkins knob?
~Haohui
On Tue, Oct 13, 2015 at 8:24 AM, Vinod Vavilapalli
<vi...@hortonworks.com> wrote:
> If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
>
> In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
>
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
>
> Thanks
> +Vinod
>
>
>> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>
>> Thank you for sharing lots information and joining discussion.
>>
>> About the runtime support of JDK-8, let's describe it on wiki. It
>> would be great for users to clarify which JDK version they should use.
>> It's also useful to describe to use "1.8.0_40 or above" explicitly.
>> Andrew, Elliott, could you update wiki or can I update wiki to add the
>> description?
>>
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About the source code level, I summarize opinions by users and
>> developers as follows:
>>
>> 1. Moving trunk to the java-8 compatible dependencies.
>> 2. Creating branch-2-java-8 after 1.
>> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
>> Currently, we don't have any consensus.
>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>>
>> This way looks good to me.
>> Steve, do you know how to do this? In fact, I don't know so much about
>> how to change and update nightly builds. Should we contact to Yetus
>> community or Apache Infra team?
>>
>> Best,
>> - Tsuyoshi
>>
>>
>>
>> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>>> source-wise) **at this point** would be an issue. I concur with the
>>> sentiment that we should preserve the java 7 support on branch-2 (not not
>>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>
>>>>
>>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>>
>>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>>
>>>>>> Do you have any concern about this? I’ve not
>>>>>> tested with HBase yet.
>>>>>>
>>>>>
>>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>>> quite a while. Everything has been very stable for us. We're running and
>>>>> compiling with jdk8.
>>>>>
>>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>>> didn't do too well.
>>>>>
>>>>
>>>> maybe related to the initial memory requirements being higher?
>>>>
>>>> otherwise: did you file a JIRA?
>>>>
>>>>
>>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>>> would be very weird for that to happen on anything but a major release.
>>>>
>>>>
>>>> Past discussion (including a big contrib from twitter) was that breaking
>>>> Java 7 support breaks all client apps too, which is not something people
>>>> were ready for.
>>>>
>>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>>> have the jenkins nighly builds only before java 8; we'd still have the
>>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>>> attention to the fact.
>>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Haohui Mai <ri...@gmail.com>.
Just to echo Steve's idea -- if we're seriously considering supporting
JDK 8, maybe the first thing to do is to set up the jenkins to run
with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
need to play around with all the Jenkins knob?
~Haohui
On Tue, Oct 13, 2015 at 8:24 AM, Vinod Vavilapalli
<vi...@hortonworks.com> wrote:
> If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
>
> In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
>
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
>
> Thanks
> +Vinod
>
>
>> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>
>> Thank you for sharing lots information and joining discussion.
>>
>> About the runtime support of JDK-8, let's describe it on wiki. It
>> would be great for users to clarify which JDK version they should use.
>> It's also useful to describe to use "1.8.0_40 or above" explicitly.
>> Andrew, Elliott, could you update wiki or can I update wiki to add the
>> description?
>>
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About the source code level, I summarize opinions by users and
>> developers as follows:
>>
>> 1. Moving trunk to the java-8 compatible dependencies.
>> 2. Creating branch-2-java-8 after 1.
>> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
>> Currently, we don't have any consensus.
>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>>
>> This way looks good to me.
>> Steve, do you know how to do this? In fact, I don't know so much about
>> how to change and update nightly builds. Should we contact to Yetus
>> community or Apache Infra team?
>>
>> Best,
>> - Tsuyoshi
>>
>>
>>
>> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>>> source-wise) **at this point** would be an issue. I concur with the
>>> sentiment that we should preserve the java 7 support on branch-2 (not not
>>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>
>>>>
>>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>>
>>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>>
>>>>>> Do you have any concern about this? I’ve not
>>>>>> tested with HBase yet.
>>>>>>
>>>>>
>>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>>> quite a while. Everything has been very stable for us. We're running and
>>>>> compiling with jdk8.
>>>>>
>>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>>> didn't do too well.
>>>>>
>>>>
>>>> maybe related to the initial memory requirements being higher?
>>>>
>>>> otherwise: did you file a JIRA?
>>>>
>>>>
>>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>>> would be very weird for that to happen on anything but a major release.
>>>>
>>>>
>>>> Past discussion (including a big contrib from twitter) was that breaking
>>>> Java 7 support breaks all client apps too, which is not something people
>>>> were ready for.
>>>>
>>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>>> have the jenkins nighly builds only before java 8; we'd still have the
>>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>>> attention to the fact.
>>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Haohui Mai <ri...@gmail.com>.
Just to echo Steve's idea -- if we're seriously considering supporting
JDK 8, maybe the first thing to do is to set up the jenkins to run
with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
need to play around with all the Jenkins knob?
~Haohui
On Tue, Oct 13, 2015 at 8:24 AM, Vinod Vavilapalli
<vi...@hortonworks.com> wrote:
> If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
>
> In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
>
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
>
> Thanks
> +Vinod
>
>
>> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>
>> Thank you for sharing lots information and joining discussion.
>>
>> About the runtime support of JDK-8, let's describe it on wiki. It
>> would be great for users to clarify which JDK version they should use.
>> It's also useful to describe to use "1.8.0_40 or above" explicitly.
>> Andrew, Elliott, could you update wiki or can I update wiki to add the
>> description?
>>
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About the source code level, I summarize opinions by users and
>> developers as follows:
>>
>> 1. Moving trunk to the java-8 compatible dependencies.
>> 2. Creating branch-2-java-8 after 1.
>> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
>> Currently, we don't have any consensus.
>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>>
>> This way looks good to me.
>> Steve, do you know how to do this? In fact, I don't know so much about
>> how to change and update nightly builds. Should we contact to Yetus
>> community or Apache Infra team?
>>
>> Best,
>> - Tsuyoshi
>>
>>
>>
>> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>>> source-wise) **at this point** would be an issue. I concur with the
>>> sentiment that we should preserve the java 7 support on branch-2 (not not
>>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>
>>>>
>>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>>
>>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>>
>>>>>> Do you have any concern about this? I’ve not
>>>>>> tested with HBase yet.
>>>>>>
>>>>>
>>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>>> quite a while. Everything has been very stable for us. We're running and
>>>>> compiling with jdk8.
>>>>>
>>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>>> didn't do too well.
>>>>>
>>>>
>>>> maybe related to the initial memory requirements being higher?
>>>>
>>>> otherwise: did you file a JIRA?
>>>>
>>>>
>>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>>> would be very weird for that to happen on anything but a major release.
>>>>
>>>>
>>>> Past discussion (including a big contrib from twitter) was that breaking
>>>> Java 7 support breaks all client apps too, which is not something people
>>>> were ready for.
>>>>
>>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>>> have the jenkins nighly builds only before java 8; we'd still have the
>>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>>> attention to the fact.
>>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Haohui Mai <ri...@gmail.com>.
Just to echo Steve's idea -- if we're seriously considering supporting
JDK 8, maybe the first thing to do is to set up the jenkins to run
with JDK 8? I'm happy to help. Does anyone know who I can talk to if I
need to play around with all the Jenkins knob?
~Haohui
On Tue, Oct 13, 2015 at 8:24 AM, Vinod Vavilapalli
<vi...@hortonworks.com> wrote:
> If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
>
> In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
>
> We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
>
> Thanks
> +Vinod
>
>
>> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>
>> Thank you for sharing lots information and joining discussion.
>>
>> About the runtime support of JDK-8, let's describe it on wiki. It
>> would be great for users to clarify which JDK version they should use.
>> It's also useful to describe to use "1.8.0_40 or above" explicitly.
>> Andrew, Elliott, could you update wiki or can I update wiki to add the
>> description?
>>
>> https://wiki.apache.org/hadoop/HadoopJavaVersions
>>
>> About the source code level, I summarize opinions by users and
>> developers as follows:
>>
>> 1. Moving trunk to the java-8 compatible dependencies.
>> 2. Creating branch-2-java-8 after 1.
>> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
>> Currently, we don't have any consensus.
>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>>
>> This way looks good to me.
>> Steve, do you know how to do this? In fact, I don't know so much about
>> how to change and update nightly builds. Should we contact to Yetus
>> community or Apache Infra team?
>>
>> Best,
>> - Tsuyoshi
>>
>>
>>
>> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>>> source-wise) **at this point** would be an issue. I concur with the
>>> sentiment that we should preserve the java 7 support on branch-2 (not not
>>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>>> wrote:
>>>
>>>>
>>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>>
>>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>>
>>>>>> Do you have any concern about this? I’ve not
>>>>>> tested with HBase yet.
>>>>>>
>>>>>
>>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>>> quite a while. Everything has been very stable for us. We're running and
>>>>> compiling with jdk8.
>>>>>
>>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>>> didn't do too well.
>>>>>
>>>>
>>>> maybe related to the initial memory requirements being higher?
>>>>
>>>> otherwise: did you file a JIRA?
>>>>
>>>>
>>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>>> would be very weird for that to happen on anything but a major release.
>>>>
>>>>
>>>> Past discussion (including a big contrib from twitter) was that breaking
>>>> Java 7 support breaks all client apps too, which is not something people
>>>> were ready for.
>>>>
>>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>>> have the jenkins nighly builds only before java 8; we'd still have the
>>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>>> attention to the fact.
>>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Vinod Vavilapalli <vi...@hortonworks.com>.
If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
Thanks
+Vinod
> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Thank you for sharing lots information and joining discussion.
>
> About the runtime support of JDK-8, let's describe it on wiki. It
> would be great for users to clarify which JDK version they should use.
> It's also useful to describe to use "1.8.0_40 or above" explicitly.
> Andrew, Elliott, could you update wiki or can I update wiki to add the
> description?
>
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About the source code level, I summarize opinions by users and
> developers as follows:
>
> 1. Moving trunk to the java-8 compatible dependencies.
> 2. Creating branch-2-java-8 after 1.
> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
> Currently, we don't have any consensus.
>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>
> This way looks good to me.
> Steve, do you know how to do this? In fact, I don't know so much about
> how to change and update nightly builds. Should we contact to Yetus
> community or Apache Infra team?
>
> Best,
> - Tsuyoshi
>
>
>
> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>> source-wise) **at this point** would be an issue. I concur with the
>> sentiment that we should preserve the java 7 support on branch-2 (not not
>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>
>> Thanks,
>> Sangjin
>>
>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>>
>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>
>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>
>>>>> Do you have any concern about this? I’ve not
>>>>> tested with HBase yet.
>>>>>
>>>>
>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>> quite a while. Everything has been very stable for us. We're running and
>>>> compiling with jdk8.
>>>>
>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>> didn't do too well.
>>>>
>>>
>>> maybe related to the initial memory requirements being higher?
>>>
>>> otherwise: did you file a JIRA?
>>>
>>>
>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>> would be very weird for that to happen on anything but a major release.
>>>
>>>
>>> Past discussion (including a big contrib from twitter) was that breaking
>>> Java 7 support breaks all client apps too, which is not something people
>>> were ready for.
>>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>> have the jenkins nighly builds only before java 8; we'd still have the
>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>> attention to the fact.
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Vinod Vavilapalli <vi...@hortonworks.com>.
If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
Thanks
+Vinod
> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Thank you for sharing lots information and joining discussion.
>
> About the runtime support of JDK-8, let's describe it on wiki. It
> would be great for users to clarify which JDK version they should use.
> It's also useful to describe to use "1.8.0_40 or above" explicitly.
> Andrew, Elliott, could you update wiki or can I update wiki to add the
> description?
>
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About the source code level, I summarize opinions by users and
> developers as follows:
>
> 1. Moving trunk to the java-8 compatible dependencies.
> 2. Creating branch-2-java-8 after 1.
> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
> Currently, we don't have any consensus.
>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>
> This way looks good to me.
> Steve, do you know how to do this? In fact, I don't know so much about
> how to change and update nightly builds. Should we contact to Yetus
> community or Apache Infra team?
>
> Best,
> - Tsuyoshi
>
>
>
> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>> source-wise) **at this point** would be an issue. I concur with the
>> sentiment that we should preserve the java 7 support on branch-2 (not not
>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>
>> Thanks,
>> Sangjin
>>
>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>>
>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>
>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>
>>>>> Do you have any concern about this? I’ve not
>>>>> tested with HBase yet.
>>>>>
>>>>
>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>> quite a while. Everything has been very stable for us. We're running and
>>>> compiling with jdk8.
>>>>
>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>> didn't do too well.
>>>>
>>>
>>> maybe related to the initial memory requirements being higher?
>>>
>>> otherwise: did you file a JIRA?
>>>
>>>
>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>> would be very weird for that to happen on anything but a major release.
>>>
>>>
>>> Past discussion (including a big contrib from twitter) was that breaking
>>> Java 7 support breaks all client apps too, which is not something people
>>> were ready for.
>>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>> have the jenkins nighly builds only before java 8; we'd still have the
>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>> attention to the fact.
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Vinod Vavilapalli <vi...@hortonworks.com>.
If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
Thanks
+Vinod
> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Thank you for sharing lots information and joining discussion.
>
> About the runtime support of JDK-8, let's describe it on wiki. It
> would be great for users to clarify which JDK version they should use.
> It's also useful to describe to use "1.8.0_40 or above" explicitly.
> Andrew, Elliott, could you update wiki or can I update wiki to add the
> description?
>
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About the source code level, I summarize opinions by users and
> developers as follows:
>
> 1. Moving trunk to the java-8 compatible dependencies.
> 2. Creating branch-2-java-8 after 1.
> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
> Currently, we don't have any consensus.
>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>
> This way looks good to me.
> Steve, do you know how to do this? In fact, I don't know so much about
> how to change and update nightly builds. Should we contact to Yetus
> community or Apache Infra team?
>
> Best,
> - Tsuyoshi
>
>
>
> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>> source-wise) **at this point** would be an issue. I concur with the
>> sentiment that we should preserve the java 7 support on branch-2 (not not
>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>
>> Thanks,
>> Sangjin
>>
>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>>
>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>
>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>
>>>>> Do you have any concern about this? I’ve not
>>>>> tested with HBase yet.
>>>>>
>>>>
>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>> quite a while. Everything has been very stable for us. We're running and
>>>> compiling with jdk8.
>>>>
>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>> didn't do too well.
>>>>
>>>
>>> maybe related to the initial memory requirements being higher?
>>>
>>> otherwise: did you file a JIRA?
>>>
>>>
>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>> would be very weird for that to happen on anything but a major release.
>>>
>>>
>>> Past discussion (including a big contrib from twitter) was that breaking
>>> Java 7 support breaks all client apps too, which is not something people
>>> were ready for.
>>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>> have the jenkins nighly builds only before java 8; we'd still have the
>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>> attention to the fact.
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Vinod Vavilapalli <vi...@hortonworks.com>.
If you see the community discussion thread on 2.8, my proposal was to support *both* JDK 7 and JDK 8 first. The last time we had discussion about dropping JDKs it wasn’t fun, so let’s not go there for now.
In terms of runtime support for JDK 8, yes, there is vast evidence that things already work as they are in Hadoop and all the way up to dependent projects. This was already the case as early as Hadoop 2.6/2.7 if I remember correctly, so nothing new here.
We should target source-level support of JDK 8 too, around which you outlined a bunch of issues around dependencies. I also found a bunch of issues around generating documentation, site etc. I propose that we track them under the umbrella JIRA and make progress there first.
Thanks
+Vinod
> On Oct 13, 2015, at 1:17 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
> Thank you for sharing lots information and joining discussion.
>
> About the runtime support of JDK-8, let's describe it on wiki. It
> would be great for users to clarify which JDK version they should use.
> It's also useful to describe to use "1.8.0_40 or above" explicitly.
> Andrew, Elliott, could you update wiki or can I update wiki to add the
> description?
>
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About the source code level, I summarize opinions by users and
> developers as follows:
>
> 1. Moving trunk to the java-8 compatible dependencies.
> 2. Creating branch-2-java-8 after 1.
> 3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
> Currently, we don't have any consensus.
>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
>
> This way looks good to me.
> Steve, do you know how to do this? In fact, I don't know so much about
> how to change and update nightly builds. Should we contact to Yetus
> community or Apache Infra team?
>
> Best,
> - Tsuyoshi
>
>
>
> On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
>> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
>> source-wise) **at this point** would be an issue. I concur with the
>> sentiment that we should preserve the java 7 support on branch-2 (not not
>> move to java 8 source level) but can consider it for trunk. My 2 cents.
>>
>> Thanks,
>> Sangjin
>>
>> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>>
>>>> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>>>>
>>>> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>>>>
>>>>> Do you have any concern about this? I’ve not
>>>>> tested with HBase yet.
>>>>>
>>>>
>>>> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>>>> quite a while. Everything has been very stable for us. We're running and
>>>> compiling with jdk8.
>>>>
>>>> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>>>> didn't do too well.
>>>>
>>>
>>> maybe related to the initial memory requirements being higher?
>>>
>>> otherwise: did you file a JIRA?
>>>
>>>
>>>> I'd be +1 on dropping jdk7 support. However as downstream developer it
>>>> would be very weird for that to happen on anything but a major release.
>>>
>>>
>>> Past discussion (including a big contrib from twitter) was that breaking
>>> Java 7 support breaks all client apps too, which is not something people
>>> were ready for.
>>>
>>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>>> have the jenkins nighly builds only before java 8; we'd still have the
>>> patch and branch-2 runs on Java 7. That way, people will only have one
>>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>>> attention to the fact.
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Thank you for sharing lots information and joining discussion.
About the runtime support of JDK-8, let's describe it on wiki. It
would be great for users to clarify which JDK version they should use.
It's also useful to describe to use "1.8.0_40 or above" explicitly.
Andrew, Elliott, could you update wiki or can I update wiki to add the
description?
https://wiki.apache.org/hadoop/HadoopJavaVersions
About the source code level, I summarize opinions by users and
developers as follows:
1. Moving trunk to the java-8 compatible dependencies.
2. Creating branch-2-java-8 after 1.
3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
Currently, we don't have any consensus.
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
This way looks good to me.
Steve, do you know how to do this? In fact, I don't know so much about
how to change and update nightly builds. Should we contact to Yetus
community or Apache Infra team?
Best,
- Tsuyoshi
On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
> source-wise) **at this point** would be an issue. I concur with the
> sentiment that we should preserve the java 7 support on branch-2 (not not
> move to java 8 source level) but can consider it for trunk. My 2 cents.
>
> Thanks,
> Sangjin
>
> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>> >
>> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> >
>> >> Do you have any concern about this? I’ve not
>> >> tested with HBase yet.
>> >>
>> >
>> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>> > quite a while. Everything has been very stable for us. We're running and
>> > compiling with jdk8.
>> >
>> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>> > didn't do too well.
>> >
>>
>> maybe related to the initial memory requirements being higher?
>>
>> otherwise: did you file a JIRA?
>>
>>
>> > I'd be +1 on dropping jdk7 support. However as downstream developer it
>> > would be very weird for that to happen on anything but a major release.
>>
>>
>> Past discussion (including a big contrib from twitter) was that breaking
>> Java 7 support breaks all client apps too, which is not something people
>> were ready for.
>>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>> have the jenkins nighly builds only before java 8; we'd still have the
>> patch and branch-2 runs on Java 7. That way, people will only have one
>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Thank you for sharing lots information and joining discussion.
About the runtime support of JDK-8, let's describe it on wiki. It
would be great for users to clarify which JDK version they should use.
It's also useful to describe to use "1.8.0_40 or above" explicitly.
Andrew, Elliott, could you update wiki or can I update wiki to add the
description?
https://wiki.apache.org/hadoop/HadoopJavaVersions
About the source code level, I summarize opinions by users and
developers as follows:
1. Moving trunk to the java-8 compatible dependencies.
2. Creating branch-2-java-8 after 1.
3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
Currently, we don't have any consensus.
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
This way looks good to me.
Steve, do you know how to do this? In fact, I don't know so much about
how to change and update nightly builds. Should we contact to Yetus
community or Apache Infra team?
Best,
- Tsuyoshi
On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
> source-wise) **at this point** would be an issue. I concur with the
> sentiment that we should preserve the java 7 support on branch-2 (not not
> move to java 8 source level) but can consider it for trunk. My 2 cents.
>
> Thanks,
> Sangjin
>
> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>> >
>> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> >
>> >> Do you have any concern about this? I’ve not
>> >> tested with HBase yet.
>> >>
>> >
>> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>> > quite a while. Everything has been very stable for us. We're running and
>> > compiling with jdk8.
>> >
>> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>> > didn't do too well.
>> >
>>
>> maybe related to the initial memory requirements being higher?
>>
>> otherwise: did you file a JIRA?
>>
>>
>> > I'd be +1 on dropping jdk7 support. However as downstream developer it
>> > would be very weird for that to happen on anything but a major release.
>>
>>
>> Past discussion (including a big contrib from twitter) was that breaking
>> Java 7 support breaks all client apps too, which is not something people
>> were ready for.
>>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>> have the jenkins nighly builds only before java 8; we'd still have the
>> patch and branch-2 runs on Java 7. That way, people will only have one
>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Thank you for sharing lots information and joining discussion.
About the runtime support of JDK-8, let's describe it on wiki. It
would be great for users to clarify which JDK version they should use.
It's also useful to describe to use "1.8.0_40 or above" explicitly.
Andrew, Elliott, could you update wiki or can I update wiki to add the
description?
https://wiki.apache.org/hadoop/HadoopJavaVersions
About the source code level, I summarize opinions by users and
developers as follows:
1. Moving trunk to the java-8 compatible dependencies.
2. Creating branch-2-java-8 after 1.
3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
Currently, we don't have any consensus.
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
This way looks good to me.
Steve, do you know how to do this? In fact, I don't know so much about
how to change and update nightly builds. Should we contact to Yetus
community or Apache Infra team?
Best,
- Tsuyoshi
On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
> source-wise) **at this point** would be an issue. I concur with the
> sentiment that we should preserve the java 7 support on branch-2 (not not
> move to java 8 source level) but can consider it for trunk. My 2 cents.
>
> Thanks,
> Sangjin
>
> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>> >
>> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> >
>> >> Do you have any concern about this? I’ve not
>> >> tested with HBase yet.
>> >>
>> >
>> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>> > quite a while. Everything has been very stable for us. We're running and
>> > compiling with jdk8.
>> >
>> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>> > didn't do too well.
>> >
>>
>> maybe related to the initial memory requirements being higher?
>>
>> otherwise: did you file a JIRA?
>>
>>
>> > I'd be +1 on dropping jdk7 support. However as downstream developer it
>> > would be very weird for that to happen on anything but a major release.
>>
>>
>> Past discussion (including a big contrib from twitter) was that breaking
>> Java 7 support breaks all client apps too, which is not something people
>> were ready for.
>>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>> have the jenkins nighly builds only before java 8; we'd still have the
>> patch and branch-2 runs on Java 7. That way, people will only have one
>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Thank you for sharing lots information and joining discussion.
About the runtime support of JDK-8, let's describe it on wiki. It
would be great for users to clarify which JDK version they should use.
It's also useful to describe to use "1.8.0_40 or above" explicitly.
Andrew, Elliott, could you update wiki or can I update wiki to add the
description?
https://wiki.apache.org/hadoop/HadoopJavaVersions
About the source code level, I summarize opinions by users and
developers as follows:
1. Moving trunk to the java-8 compatible dependencies.
2. Creating branch-2-java-8 after 1.
3. Dropping Java 7 support (maybe when we support JDK 9?) after 2.
Currently, we don't have any consensus.
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
This way looks good to me.
Steve, do you know how to do this? In fact, I don't know so much about
how to change and update nightly builds. Should we contact to Yetus
community or Apache Infra team?
Best,
- Tsuyoshi
On Sat, Oct 10, 2015 at 7:17 AM, Sangjin Lee <sj...@apache.org> wrote:
> Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
> source-wise) **at this point** would be an issue. I concur with the
> sentiment that we should preserve the java 7 support on branch-2 (not not
> move to java 8 source level) but can consider it for trunk. My 2 cents.
>
> Thanks,
> Sangjin
>
> On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>>
>> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>> >
>> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>> >
>> >> Do you have any concern about this? I’ve not
>> >> tested with HBase yet.
>> >>
>> >
>> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
>> > quite a while. Everything has been very stable for us. We're running and
>> > compiling with jdk8.
>> >
>> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
>> > didn't do too well.
>> >
>>
>> maybe related to the initial memory requirements being higher?
>>
>> otherwise: did you file a JIRA?
>>
>>
>> > I'd be +1 on dropping jdk7 support. However as downstream developer it
>> > would be very weird for that to happen on anything but a major release.
>>
>>
>> Past discussion (including a big contrib from twitter) was that breaking
>> Java 7 support breaks all client apps too, which is not something people
>> were ready for.
>>
>> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
>> have the jenkins nighly builds only before java 8; we'd still have the
>> patch and branch-2 runs on Java 7. That way, people will only have one
>> nightly mail from Jenkins saying the build is broken, and maybe pay more
>> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
source-wise) **at this point** would be an issue. I concur with the
sentiment that we should preserve the java 7 support on branch-2 (not not
move to java 8 source level) but can consider it for trunk. My 2 cents.
Thanks,
Sangjin
On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
> >
> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> >> Do you have any concern about this? I’ve not
> >> tested with HBase yet.
> >>
> >
> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> > quite a while. Everything has been very stable for us. We're running and
> > compiling with jdk8.
> >
> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> > didn't do too well.
> >
>
> maybe related to the initial memory requirements being higher?
>
> otherwise: did you file a JIRA?
>
>
> > I'd be +1 on dropping jdk7 support. However as downstream developer it
> > would be very weird for that to happen on anything but a major release.
>
>
> Past discussion (including a big contrib from twitter) was that breaking
> Java 7 support breaks all client apps too, which is not something people
> were ready for.
>
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
> have the jenkins nighly builds only before java 8; we'd still have the
> patch and branch-2 runs on Java 7. That way, people will only have one
> nightly mail from Jenkins saying the build is broken, and maybe pay more
> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
source-wise) **at this point** would be an issue. I concur with the
sentiment that we should preserve the java 7 support on branch-2 (not not
move to java 8 source level) but can consider it for trunk. My 2 cents.
Thanks,
Sangjin
On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
> >
> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> >> Do you have any concern about this? I’ve not
> >> tested with HBase yet.
> >>
> >
> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> > quite a while. Everything has been very stable for us. We're running and
> > compiling with jdk8.
> >
> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> > didn't do too well.
> >
>
> maybe related to the initial memory requirements being higher?
>
> otherwise: did you file a JIRA?
>
>
> > I'd be +1 on dropping jdk7 support. However as downstream developer it
> > would be very weird for that to happen on anything but a major release.
>
>
> Past discussion (including a big contrib from twitter) was that breaking
> Java 7 support breaks all client apps too, which is not something people
> were ready for.
>
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
> have the jenkins nighly builds only before java 8; we'd still have the
> patch and branch-2 runs on Java 7. That way, people will only have one
> nightly mail from Jenkins saying the build is broken, and maybe pay more
> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
source-wise) **at this point** would be an issue. I concur with the
sentiment that we should preserve the java 7 support on branch-2 (not not
move to java 8 source level) but can consider it for trunk. My 2 cents.
Thanks,
Sangjin
On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
> >
> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> >> Do you have any concern about this? I’ve not
> >> tested with HBase yet.
> >>
> >
> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> > quite a while. Everything has been very stable for us. We're running and
> > compiling with jdk8.
> >
> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> > didn't do too well.
> >
>
> maybe related to the initial memory requirements being higher?
>
> otherwise: did you file a JIRA?
>
>
> > I'd be +1 on dropping jdk7 support. However as downstream developer it
> > would be very weird for that to happen on anything but a major release.
>
>
> Past discussion (including a big contrib from twitter) was that breaking
> Java 7 support breaks all client apps too, which is not something people
> were ready for.
>
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
> have the jenkins nighly builds only before java 8; we'd still have the
> patch and branch-2 runs on Java 7. That way, people will only have one
> nightly mail from Jenkins saying the build is broken, and maybe pay more
> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Sangjin Lee <sj...@apache.org>.
Yes, at least for us, dropping the java 7 support (e.g. moving to java 8
source-wise) **at this point** would be an issue. I concur with the
sentiment that we should preserve the java 7 support on branch-2 (not not
move to java 8 source level) but can consider it for trunk. My 2 cents.
Thanks,
Sangjin
On Fri, Oct 9, 2015 at 10:43 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
> >
> > On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> >
> >> Do you have any concern about this? I’ve not
> >> tested with HBase yet.
> >>
> >
> > We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> > quite a while. Everything has been very stable for us. We're running and
> > compiling with jdk8.
> >
> > We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> > didn't do too well.
> >
>
> maybe related to the initial memory requirements being higher?
>
> otherwise: did you file a JIRA?
>
>
> > I'd be +1 on dropping jdk7 support. However as downstream developer it
> > would be very weird for that to happen on anything but a major release.
>
>
> Past discussion (including a big contrib from twitter) was that breaking
> Java 7 support breaks all client apps too, which is not something people
> were ready for.
>
> I'd be +1 for moving trunk to the java-8 compatible dependencies, and to
> have the jenkins nighly builds only before java 8; we'd still have the
> patch and branch-2 runs on Java 7. That way, people will only have one
> nightly mail from Jenkins saying the build is broken, and maybe pay more
> attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>
> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
>> Do you have any concern about this? I’ve not
>> tested with HBase yet.
>>
>
> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> quite a while. Everything has been very stable for us. We're running and
> compiling with jdk8.
>
> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> didn't do too well.
>
maybe related to the initial memory requirements being higher?
otherwise: did you file a JIRA?
> I'd be +1 on dropping jdk7 support. However as downstream developer it
> would be very weird for that to happen on anything but a major release.
Past discussion (including a big contrib from twitter) was that breaking Java 7 support breaks all client apps too, which is not something people were ready for.
I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>
> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
>> Do you have any concern about this? I’ve not
>> tested with HBase yet.
>>
>
> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> quite a while. Everything has been very stable for us. We're running and
> compiling with jdk8.
>
> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> didn't do too well.
>
maybe related to the initial memory requirements being higher?
otherwise: did you file a JIRA?
> I'd be +1 on dropping jdk7 support. However as downstream developer it
> would be very weird for that to happen on anything but a major release.
Past discussion (including a big contrib from twitter) was that breaking Java 7 support breaks all client apps too, which is not something people were ready for.
I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>
> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
>> Do you have any concern about this? I’ve not
>> tested with HBase yet.
>>
>
> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> quite a while. Everything has been very stable for us. We're running and
> compiling with jdk8.
>
> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> didn't do too well.
>
maybe related to the initial memory requirements being higher?
otherwise: did you file a JIRA?
> I'd be +1 on dropping jdk7 support. However as downstream developer it
> would be very weird for that to happen on anything but a major release.
Past discussion (including a big contrib from twitter) was that breaking Java 7 support breaks all client apps too, which is not something people were ready for.
I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 22:39, Elliott Clark <ec...@apache.org> wrote:
>
> On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
>
>> Do you have any concern about this? I’ve not
>> tested with HBase yet.
>>
>
> We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
> quite a while. Everything has been very stable for us. We're running and
> compiling with jdk8.
>
> We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
> didn't do too well.
>
maybe related to the initial memory requirements being higher?
otherwise: did you file a JIRA?
> I'd be +1 on dropping jdk7 support. However as downstream developer it
> would be very weird for that to happen on anything but a major release.
Past discussion (including a big contrib from twitter) was that breaking Java 7 support breaks all client apps too, which is not something people were ready for.
I'd be +1 for moving trunk to the java-8 compatible dependencies, and to have the jenkins nighly builds only before java 8; we'd still have the patch and branch-2 runs on Java 7. That way, people will only have one nightly mail from Jenkins saying the build is broken, and maybe pay more attention to the fact.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.
sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
This has been fixed in jdk8u60.
http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
One of the visible symptom is RPC reader thread getting ArrayIndexOutOfBoundsException from
sun.security.provider.DigestBase.engineUpdate. More concerning case is a reader operating on a wrong digest.
Kihwal
________________________________
From: Jean-Baptiste Note <jb...@gmail.com>
To: common-dev@hadoop.apache.org
Cc: "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; dev <de...@hbase.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
Sent: Tuesday, October 13, 2015 5:00 AM
Subject: Re: [DISCUSS] About the details of JDK-8 support
Hi,
As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732
I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.
JB
Re: [DISCUSS] About the details of JDK-8 support
Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.
sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
This has been fixed in jdk8u60.
http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
One of the visible symptom is RPC reader thread getting ArrayIndexOutOfBoundsException from
sun.security.provider.DigestBase.engineUpdate. More concerning case is a reader operating on a wrong digest.
Kihwal
________________________________
From: Jean-Baptiste Note <jb...@gmail.com>
To: common-dev@hadoop.apache.org
Cc: "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; dev <de...@hbase.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
Sent: Tuesday, October 13, 2015 5:00 AM
Subject: Re: [DISCUSS] About the details of JDK-8 support
Hi,
As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732
I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.
JB
Re: [DISCUSS] About the details of JDK-8 support
Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.
sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
This has been fixed in jdk8u60.
http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
One of the visible symptom is RPC reader thread getting ArrayIndexOutOfBoundsException from
sun.security.provider.DigestBase.engineUpdate. More concerning case is a reader operating on a wrong digest.
Kihwal
________________________________
From: Jean-Baptiste Note <jb...@gmail.com>
To: common-dev@hadoop.apache.org
Cc: "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; dev <de...@hbase.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
Sent: Tuesday, October 13, 2015 5:00 AM
Subject: Re: [DISCUSS] About the details of JDK-8 support
Hi,
As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732
I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.
JB
Re: [DISCUSS] About the details of JDK-8 support
Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.
sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
This has been fixed in jdk8u60.
http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
One of the visible symptom is RPC reader thread getting ArrayIndexOutOfBoundsException from
sun.security.provider.DigestBase.engineUpdate. More concerning case is a reader operating on a wrong digest.
Kihwal
________________________________
From: Jean-Baptiste Note <jb...@gmail.com>
To: common-dev@hadoop.apache.org
Cc: "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; dev <de...@hbase.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
Sent: Tuesday, October 13, 2015 5:00 AM
Subject: Re: [DISCUSS] About the details of JDK-8 support
Hi,
As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732
I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.
JB
Re: [DISCUSS] About the details of JDK-8 support
Posted by Chen He <ai...@gmail.com>.
On Tue, Oct 13, 2015 at 3:00 AM, Jean-Baptiste Note <jb...@gmail.com>
wrote:
> Hi,
>
> As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
> production with security enabled.
> We reported some gripes with some specific java versions:
> https://issues.cloudera.org/browse/DISTRO-732
>
> I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
> problems with SPNEGO SSO.
>
> Agree with Jean, according our experiences, transferring file from HDFS to
swift store, j8u45 is 6 time slower than java7. But j8u60 is fine.
> JB
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.
sun.security.krb5.KrbApReq was creating a static MD5 digest object and not synchronizing access.
This has been fixed in jdk8u60.
http://hg.openjdk.java.net/jdk8u/jdk8u60/jdk/rev/02d6b1096e89
One of the visible symptom is RPC reader thread getting ArrayIndexOutOfBoundsException from
sun.security.provider.DigestBase.engineUpdate. More concerning case is a reader operating on a wrong digest.
Kihwal
________________________________
From: Jean-Baptiste Note <jb...@gmail.com>
To: common-dev@hadoop.apache.org
Cc: "mapreduce-dev@hadoop.apache.org" <ma...@hadoop.apache.org>; "hdfs-dev@hadoop.apache.org" <hd...@hadoop.apache.org>; dev <de...@hbase.apache.org>; "yarn-dev@hadoop.apache.org" <ya...@hadoop.apache.org>
Sent: Tuesday, October 13, 2015 5:00 AM
Subject: Re: [DISCUSS] About the details of JDK-8 support
Hi,
As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732
I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.
JB
Re: [DISCUSS] About the details of JDK-8 support
Posted by Jean-Baptiste Note <jb...@gmail.com>.
Hi,
As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732
I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.
JB
Re: [DISCUSS] About the details of JDK-8 support
Posted by Jean-Baptiste Note <jb...@gmail.com>.
Hi,
As far as security is concerned we (Criteo) are using CDH5 with JDK8 in
production with security enabled.
We reported some gripes with some specific java versions:
https://issues.cloudera.org/browse/DISTRO-732
I would bump the dependency to _u51 or later; _u40 _u45 do have a lot of
problems with SPNEGO SSO.
JB
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
>
> > On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
> >
> > We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> > the full stack including HBase), so I agree that we're good there.
> >
>
>
> with Kerberos on?
>
> Yea, I haven't been that involved with our internal JDK validation
efforts, but I know there have been an assortment of JDK8 bugs related to
Kerberos. Our latest docs currently recommend 1.8.0_40 or above:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp_unique_1
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
>
> > On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
> >
> > We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> > the full stack including HBase), so I agree that we're good there.
> >
>
>
> with Kerberos on?
>
> Yea, I haven't been that involved with our internal JDK validation
efforts, but I know there have been an assortment of JDK8 bugs related to
Kerberos. Our latest docs currently recommend 1.8.0_40 or above:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp_unique_1
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
>
> > On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
> >
> > We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> > the full stack including HBase), so I agree that we're good there.
> >
>
>
> with Kerberos on?
>
> Yea, I haven't been that involved with our internal JDK validation
efforts, but I know there have been an assortment of JDK8 bugs related to
Kerberos. Our latest docs currently recommend 1.8.0_40 or above:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp_unique_1
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
>
> > On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
> >
> > We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> > the full stack including HBase), so I agree that we're good there.
> >
>
>
> with Kerberos on?
>
> Yea, I haven't been that involved with our internal JDK validation
efforts, but I know there have been an assortment of JDK8 bugs related to
Kerberos. Our latest docs currently recommend 1.8.0_40 or above:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp_unique_1
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
>
> > On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
> >
> > We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> > the full stack including HBase), so I agree that we're good there.
> >
>
>
> with Kerberos on?
>
> Yea, I haven't been that involved with our internal JDK validation
efforts, but I know there have been an assortment of JDK8 bugs related to
Kerberos. Our latest docs currently recommend 1.8.0_40 or above:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_req_supported_versions.html#concept_pdd_kzf_vp_unique_1
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
>
> We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> the full stack including HBase), so I agree that we're good there.
>
with Kerberos on?
> I'm against dropping JDK7 support though in branch-2.
+1. We're only talking about dependencies here
> Even bumping
> dependency versions scares me, since it often leads to downstream pain. Any
> comment about the compatibility of said bump? We need to have very high
> confidence if it's targeted for branch-2.
>
Which is why we need to test things downstream with the bumped jersey/grizzly libs before making any commitment to branch-2.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
>
> We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> the full stack including HBase), so I agree that we're good there.
>
with Kerberos on?
> I'm against dropping JDK7 support though in branch-2.
+1. We're only talking about dependencies here
> Even bumping
> dependency versions scares me, since it often leads to downstream pain. Any
> comment about the compatibility of said bump? We need to have very high
> confidence if it's targeted for branch-2.
>
Which is why we need to test things downstream with the bumped jersey/grizzly libs before making any commitment to branch-2.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
>
> We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> the full stack including HBase), so I agree that we're good there.
>
with Kerberos on?
> I'm against dropping JDK7 support though in branch-2.
+1. We're only talking about dependencies here
> Even bumping
> dependency versions scares me, since it often leads to downstream pain. Any
> comment about the compatibility of said bump? We need to have very high
> confidence if it's targeted for branch-2.
>
Which is why we need to test things downstream with the bumped jersey/grizzly libs before making any commitment to branch-2.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
>
> We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> the full stack including HBase), so I agree that we're good there.
>
with Kerberos on?
> I'm against dropping JDK7 support though in branch-2.
+1. We're only talking about dependencies here
> Even bumping
> dependency versions scares me, since it often leads to downstream pain. Any
> comment about the compatibility of said bump? We need to have very high
> confidence if it's targeted for branch-2.
>
Which is why we need to test things downstream with the bumped jersey/grizzly libs before making any commitment to branch-2.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 17:23, Andrew Wang <an...@cloudera.com> wrote:
>
> We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
> the full stack including HBase), so I agree that we're good there.
>
with Kerberos on?
> I'm against dropping JDK7 support though in branch-2.
+1. We're only talking about dependencies here
> Even bumping
> dependency versions scares me, since it often leads to downstream pain. Any
> comment about the compatibility of said bump? We need to have very high
> confidence if it's targeted for branch-2.
>
Which is why we need to test things downstream with the bumped jersey/grizzly libs before making any commitment to branch-2.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Andrew Wang <an...@cloudera.com>.
We've been supporting JDK8 as a runtime for CDH5 for a while now (meaning
the full stack including HBase), so I agree that we're good there.
I'm against dropping JDK7 support though in branch-2. Even bumping
dependency versions scares me, since it often leads to downstream pain. Any
comment about the compatibility of said bump? We need to have very high
confidence if it's targeted for branch-2.
Best,
Andrew
On Wed, Oct 7, 2015 at 2:27 AM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> > On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp>
> wrote:
> >
> > Thanks for clear summary, Tsuyoshi.
> >
> > I read some related past discussions.
> >
> > https://wiki.apache.org/hadoop/MovingToJdk7and8
> > http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> > http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
> >
> > Though there seems to be no consensus about when to drop java 7 support
> yet,
> > it would not be 2.8 for which the preparation is already started.
> > If the works for making source compatible with java 8 does not result in
> > dropping java 7 support, it would be nice and easy to backport to
> branch-2.
> >
> >
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2?
> >
> > At lease, the newest grizzly, jersey and asm seems to support java 7 too
> > and HADOOP-11993 may work in branch-2.
> >
>
> Certainly for trunk, I'm +1 for making the leap. For branch 2, how
> backwards compatible/incompatible is the change?
>
> I think we'd have to test it downstream; I can use slider & spark as test
> builds locally —YARN apps are the failure points. Someone else would have
> to try HBase.
>
> In that world, we could think of having a short-lived branch-2-java-8
> branch, which cherry picks the grizzly changes from trunk, and which we can
> then use for that downstream testing
>
> >
> > Masatake Iwasaki
> >
> >
> > On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > > Hi commiters and users of Hadoop stack,
> > >
> > > I’ll share the current status of JDK-8 support here. We can take a
> > > two-step approach to support JDK-8 - runtime-level support and
> > > source-level support.
> > >
> > > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > > long as I tested, it works well completely since JDK-8 doesn’t have
> > > any incompatibility at binary level. We can say Hadoop has supported
> > > JDK8 runtime already. Do you have any concern about this? I’ve not
> > > tested with HBase yet. I need help of HBase community. I think only
> > > problem about runtime is HADOOP-11364, default value of
> > > colntainer-killer of YARN. After fixing the issue, we can declare the
> > > support of JDK on Hadoop Wiki to make it clear for users.
> > > https://wiki.apache.org/hadoop/HadoopJavaVersions
> > >
> > > About source-level, however, we have one big problem - upgrading
> > > dependency of asm and cglib. We need to upgrade all libraries which
> > > depends on asm to support new byte code introduced in JDK8[1]. The
> > > dependencies which uses asm are jersey-server for compile and provide
> > > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > > command). HADOOP-9613 is addressing the problem.
> > >
> > > One complex problem I’ve faced is Jersey depends on grizzly - to
> > > upgrade jersey to 1.19, which supports JDK8,
> > > we need to upgrade grizzly to 2.2.16 to use
> > > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > > will target this change. Can we do this in branch-2? Should we take
> > > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > > asm, or not. I think we can collaborate with Yetus community here.
> > >
> > > Also, another simple problem is that source code cannot be compiled
> > > because javadoc format or variable identifier are illegal(e.g.
> > > HADOOP-12457, HADOOP-11875). I think this can be solved
> > > straightforwardly.
> > >
> > > Please share any concern I’ve missed. The opinions of users are also
> welcome :-)
> > > I'd like to go forward this step by step to make Hadoop user friendly.
> > >
> > > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > > JDK-8.
> > >
> > > Best regards,
> > > - Tsuyoshi
> > >
> > > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
> >
> >
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Steve Loughran <st...@hortonworks.com>.
> On 7 Oct 2015, at 07:29, Masatake Iwasaki <iw...@oss.nttdata.co.jp> wrote:
>
> Thanks for clear summary, Tsuyoshi.
>
> I read some related past discussions.
>
> https://wiki.apache.org/hadoop/MovingToJdk7and8
> http://search-hadoop.com/m/uOzYtGSiCs1acRnh
> http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
>
> Though there seems to be no consensus about when to drop java 7 support yet,
> it would not be 2.8 for which the preparation is already started.
> If the works for making source compatible with java 8 does not result in
> dropping java 7 support, it would be nice and easy to backport to branch-2.
>
>
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2?
>
> At lease, the newest grizzly, jersey and asm seems to support java 7 too
> and HADOOP-11993 may work in branch-2.
>
Certainly for trunk, I'm +1 for making the leap. For branch 2, how backwards compatible/incompatible is the change?
I think we'd have to test it downstream; I can use slider & spark as test builds locally —YARN apps are the failure points. Someone else would have to try HBase.
In that world, we could think of having a short-lived branch-2-java-8 branch, which cherry picks the grizzly changes from trunk, and which we can then use for that downstream testing
>
> Masatake Iwasaki
>
>
> On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> > Hi commiters and users of Hadoop stack,
> >
> > I’ll share the current status of JDK-8 support here. We can take a
> > two-step approach to support JDK-8 - runtime-level support and
> > source-level support.
> >
> > About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> > MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> > long as I tested, it works well completely since JDK-8 doesn’t have
> > any incompatibility at binary level. We can say Hadoop has supported
> > JDK8 runtime already. Do you have any concern about this? I’ve not
> > tested with HBase yet. I need help of HBase community. I think only
> > problem about runtime is HADOOP-11364, default value of
> > colntainer-killer of YARN. After fixing the issue, we can declare the
> > support of JDK on Hadoop Wiki to make it clear for users.
> > https://wiki.apache.org/hadoop/HadoopJavaVersions
> >
> > About source-level, however, we have one big problem - upgrading
> > dependency of asm and cglib. We need to upgrade all libraries which
> > depends on asm to support new byte code introduced in JDK8[1]. The
> > dependencies which uses asm are jersey-server for compile and provide
> > scope, and cglib for test scope(I checked it with mvn dependency:tree
> > command). HADOOP-9613 is addressing the problem.
> >
> > One complex problem I’ve faced is Jersey depends on grizzly - to
> > upgrade jersey to 1.19, which supports JDK8,
> > we need to upgrade grizzly to 2.2.16 to use
> > jersey-test-framework-grizzly2. I’d like to discuss which version we
> > will target this change. Can we do this in branch-2? Should we take
> > care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> > confirm whether HADOOP-11993 means to remove Jersey, which depends on
> > asm, or not. I think we can collaborate with Yetus community here.
> >
> > Also, another simple problem is that source code cannot be compiled
> > because javadoc format or variable identifier are illegal(e.g.
> > HADOOP-12457, HADOOP-11875). I think this can be solved
> > straightforwardly.
> >
> > Please share any concern I’ve missed. The opinions of users are also welcome :-)
> > I'd like to go forward this step by step to make Hadoop user friendly.
> >
> > Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> > Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> > JDK-8.
> >
> > Best regards,
> > - Tsuyoshi
> >
> > [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> > [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
>
>
Re: [DISCUSS] About the details of JDK-8 support
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Thanks for clear summary, Tsuyoshi.
I read some related past discussions.
https://wiki.apache.org/hadoop/MovingToJdk7and8
http://search-hadoop.com/m/uOzYtGSiCs1acRnh
http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
Though there seems to be no consensus about when to drop java 7 support yet,
it would not be 2.8 for which the preparation is already started.
If the works for making source compatible with java 8 does not result in
dropping java 7 support, it would be nice and easy to backport to branch-2.
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2?
At lease, the newest grizzly, jersey and asm seems to support java 7 too
and HADOOP-11993 may work in branch-2.
Masatake Iwasaki
On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also
welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Thanks for clear summary, Tsuyoshi.
I read some related past discussions.
https://wiki.apache.org/hadoop/MovingToJdk7and8
http://search-hadoop.com/m/uOzYtGSiCs1acRnh
http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
Though there seems to be no consensus about when to drop java 7 support yet,
it would not be 2.8 for which the preparation is already started.
If the works for making source compatible with java 8 does not result in
dropping java 7 support, it would be nice and easy to backport to branch-2.
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2?
At lease, the newest grizzly, jersey and asm seems to support java 7 too
and HADOOP-11993 may work in branch-2.
Masatake Iwasaki
On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also
welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Thanks for clear summary, Tsuyoshi.
I read some related past discussions.
https://wiki.apache.org/hadoop/MovingToJdk7and8
http://search-hadoop.com/m/uOzYtGSiCs1acRnh
http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
Though there seems to be no consensus about when to drop java 7 support yet,
it would not be 2.8 for which the preparation is already started.
If the works for making source compatible with java 8 does not result in
dropping java 7 support, it would be nice and easy to backport to branch-2.
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2?
At lease, the newest grizzly, jersey and asm seems to support java 7 too
and HADOOP-11993 may work in branch-2.
Masatake Iwasaki
On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also
welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Thanks for clear summary, Tsuyoshi.
I read some related past discussions.
https://wiki.apache.org/hadoop/MovingToJdk7and8
http://search-hadoop.com/m/uOzYtGSiCs1acRnh
http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
Though there seems to be no consensus about when to drop java 7 support yet,
it would not be 2.8 for which the preparation is already started.
If the works for making source compatible with java 8 does not result in
dropping java 7 support, it would be nice and easy to backport to branch-2.
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2?
At lease, the newest grizzly, jersey and asm seems to support java 7 too
and HADOOP-11993 may work in branch-2.
Masatake Iwasaki
On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also
welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Elliott Clark <ec...@apache.org>.
On Mon, Oct 5, 2015 at 5:35 PM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Do you have any concern about this? I’ve not
> tested with HBase yet.
>
We've been running JDK 8u60 in production with Hadoop 2.6.X and HBase for
quite a while. Everything has been very stable for us. We're running and
compiling with jdk8.
We had to turn off yarn.nodemanager.vmem-check-enabled. Otherwise mr jobs
didn't do too well.
I'd be +1 on dropping jdk7 support. However as downstream developer it
would be very weird for that to happen on anything but a major release.
Re: [DISCUSS] About the details of JDK-8 support
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
I forgot to mention - we also need to upgrade Google Guice to 0.4.0
for source-level JDK8 support in addition to asm and cglib.
https://github.com/google/guice/wiki/Guice40
> Some highlights:
> Java8 runtime compatibility for Guice core & all extensions.
Best regards,
- Tsuyoshi
On Tue, Oct 6, 2015 at 9:35 AM, Tsuyoshi Ozawa <oz...@apache.org> wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker
Re: [DISCUSS] About the details of JDK-8 support
Posted by Masatake Iwasaki <iw...@oss.nttdata.co.jp>.
Thanks for clear summary, Tsuyoshi.
I read some related past discussions.
https://wiki.apache.org/hadoop/MovingToJdk7and8
http://search-hadoop.com/m/uOzYtGSiCs1acRnh
http://search-hadoop.com/m/uOzYthdWJqpGdSZ1
Though there seems to be no consensus about when to drop java 7 support yet,
it would not be 2.8 for which the preparation is already started.
If the works for making source compatible with java 8 does not result in
dropping java 7 support, it would be nice and easy to backport to branch-2.
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2?
At lease, the newest grizzly, jersey and asm seems to support java 7 too
and HADOOP-11993 may work in branch-2.
Masatake Iwasaki
On 10/6/15 09:35, Tsuyoshi Ozawa wrote:
> Hi commiters and users of Hadoop stack,
>
> I’ll share the current status of JDK-8 support here. We can take a
> two-step approach to support JDK-8 - runtime-level support and
> source-level support.
>
> About runtime-level support, I’ve tested Hadoop stack with JDK-8 e.g.
> MapReduce, Spark, Tez, Flink on YARN and HDFS for a few months. As
> long as I tested, it works well completely since JDK-8 doesn’t have
> any incompatibility at binary level. We can say Hadoop has supported
> JDK8 runtime already. Do you have any concern about this? I’ve not
> tested with HBase yet. I need help of HBase community. I think only
> problem about runtime is HADOOP-11364, default value of
> colntainer-killer of YARN. After fixing the issue, we can declare the
> support of JDK on Hadoop Wiki to make it clear for users.
> https://wiki.apache.org/hadoop/HadoopJavaVersions
>
> About source-level, however, we have one big problem - upgrading
> dependency of asm and cglib. We need to upgrade all libraries which
> depends on asm to support new byte code introduced in JDK8[1]. The
> dependencies which uses asm are jersey-server for compile and provide
> scope, and cglib for test scope(I checked it with mvn dependency:tree
> command). HADOOP-9613 is addressing the problem.
>
> One complex problem I’ve faced is Jersey depends on grizzly - to
> upgrade jersey to 1.19, which supports JDK8,
> we need to upgrade grizzly to 2.2.16 to use
> jersey-test-framework-grizzly2. I’d like to discuss which version we
> will target this change. Can we do this in branch-2? Should we take
> care of HADOOP-11656 and HADOOP-11993 at the same time? I’d also
> confirm whether HADOOP-11993 means to remove Jersey, which depends on
> asm, or not. I think we can collaborate with Yetus community here.
>
> Also, another simple problem is that source code cannot be compiled
> because javadoc format or variable identifier are illegal(e.g.
> HADOOP-12457, HADOOP-11875). I think this can be solved
> straightforwardly.
>
> Please share any concern I’ve missed. The opinions of users are also
welcome :-)
> I'd like to go forward this step by step to make Hadoop user friendly.
>
> Thanks Steve, Sean, Allen, Robert, Brahma, Akira, Larry, Allen, Andrew
> Purtell, Tsz-wo Sze, Sethen and other guys for having lots works about
> JDK-8.
>
> Best regards,
> - Tsuyoshi
>
> [1] http://product.hubspot.com/blog/upgrading-to-java-8-at-scale
> [2] http://ispras.linuxbase.org/index.php/Java_API_Compliance_Checker