You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sean Owen <so...@cloudera.com> on 2017/02/10 16:47:06 UTC
Request for comments: Java 7 removal
As you have seen, there's a WIP PR to implement removal of Java 7 support:
https://github.com/apache/spark/pull/16871
I have heard several +1s at
https://issues.apache.org/jira/browse/SPARK-19493 but am asking for
concerns too, now that there's a concrete change to review.
If this goes in for 2.2 it can be followed by more extensive update of the
Java code to take advantage of Java 8; this is more or less the baseline
change.
We also just removed Hadoop 2.5 support. I know there was talk about
removing Python 2.6. I have no opinion on that myself, but, might be time
to revive that conversation too.
Re: Request for comments: Java 7 removal
Posted by Yuming Wang <wg...@gmail.com>.
There is a way only Spark use Java 8, Hadoop still use Java 7:
spark-conf.jpg
(58K)
<https://mail.google.com/mail/u/0/?ui=2&ik=a9af0e7eb1&view=att&th=15a3f68a367de778&attid=0.1&disp=safe&realattid=f_iz6aduk80&zw>
By the way, I have a way to install any spark version on CM5.4 - CM5.7 by
custom CSD <https://github.com/wangyum/cm_csds/tree/master/SPARK> and
custom Spark parcel <https://github.com/wangyum/spark-parcel>.
On Wed, Feb 15, 2017 at 6:46 AM, Koert Kuipers <ko...@tresata.com> wrote:
> what about the conversation about dropping scala 2.10?
>
> On Fri, Feb 10, 2017 at 11:47 AM, Sean Owen <so...@cloudera.com> wrote:
>
>> As you have seen, there's a WIP PR to implement removal of Java 7
>> support: https://github.com/apache/spark/pull/16871
>>
>> I have heard several +1s at https://issues.apache.org/j
>> ira/browse/SPARK-19493 but am asking for concerns too, now that there's
>> a concrete change to review.
>>
>> If this goes in for 2.2 it can be followed by more extensive update of
>> the Java code to take advantage of Java 8; this is more or less the
>> baseline change.
>>
>> We also just removed Hadoop 2.5 support. I know there was talk about
>> removing Python 2.6. I have no opinion on that myself, but, might be time
>> to revive that conversation too.
>>
>
>
Re: Request for comments: Java 7 removal
Posted by Koert Kuipers <ko...@tresata.com>.
what about the conversation about dropping scala 2.10?
On Fri, Feb 10, 2017 at 11:47 AM, Sean Owen <so...@cloudera.com> wrote:
> As you have seen, there's a WIP PR to implement removal of Java 7 support:
> https://github.com/apache/spark/pull/16871
>
> I have heard several +1s at https://issues.apache.org/
> jira/browse/SPARK-19493 but am asking for concerns too, now that there's
> a concrete change to review.
>
> If this goes in for 2.2 it can be followed by more extensive update of the
> Java code to take advantage of Java 8; this is more or less the baseline
> change.
>
> We also just removed Hadoop 2.5 support. I know there was talk about
> removing Python 2.6. I have no opinion on that myself, but, might be time
> to revive that conversation too.
>
Re: Request for comments: Java 7 removal
Posted by Sean Owen <so...@cloudera.com>.
Yes, that's a key concern about the Java dependency, that its update is a
function of the OS packages and those who control them, which is often not
the end user. I think that's why this has been delayed a while. My general
position is that, of course, someone in that boat can use Spark 2.1.x. It's
likely going to see maintenance releases through the end of the year, even.
On the flip side, no (non-paid) support has been available for Java 7 for a
while. It wouldn't surprise me if some people are yet still stuck on Java
7; it would surprise me if they expect to use the latest of any package at
this stage. Taking your CDH example, yes it's been a couple years since
people have been able to deploy it on Java 8. Spark 2 isn't supported
before 5.7 anyway. The default is Java 8.
Scala 2.10 is a good point that we are dealing with now. It's not really a
question of whether it will run -- it's all libraries and bytecode to the
JVM and it will happily deal with a mix of 7 and 8 bytecode. It's a
question of whether the build for 2.10 will succeed. I believe it's 'yes'
but am following up on some tests there.
On Tue, Feb 14, 2017 at 1:15 AM Charles Allen <ch...@metamarkets.com>
wrote:
> I think the biggest concern is enterprise users/operators who do not have
> the authority or access to upgrade hadoop/yarn clusters to java8. As a
> reference point, apparently CDH 5.3
> <https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_53.html#concept_c1n_bln_tj>
> shipped with java 8 in December 2014. I would be surprised if such users
> were active consumers of the dev mailing list, though. Unfortunately
> there's a bit of a selection bias in this list.
>
> The other concern is if there is guaranteed compatibility between scala
> and java8 for all versions you want to use (which is somewhat touched upon
> in the PR). Are you thinking about supporting scala 2.10 against java 8
> byte code?
>
> See https://groups.google.com/d/msg/druid-user/aTGQlnF1KLk/NvBPfmigAAAJ for
> the similar discussion that went forward in the Druid community.
>
>
> On Fri, Feb 10, 2017 at 8:47 AM Sean Owen <so...@cloudera.com> wrote:
>
> As you have seen, there's a WIP PR to implement removal of Java 7 support:
> https://github.com/apache/spark/pull/16871
>
> I have heard several +1s at
> https://issues.apache.org/jira/browse/SPARK-19493 but am asking for
> concerns too, now that there's a concrete change to review.
>
> If this goes in for 2.2 it can be followed by more extensive update of the
> Java code to take advantage of Java 8; this is more or less the baseline
> change.
>
> We also just removed Hadoop 2.5 support. I know there was talk about
> removing Python 2.6. I have no opinion on that myself, but, might be time
> to revive that conversation too.
>
>
Re: Request for comments: Java 7 removal
Posted by Charles Allen <ch...@metamarkets.com>.
I think the biggest concern is enterprise users/operators who do not have
the authority or access to upgrade hadoop/yarn clusters to java8. As a
reference point, apparently CDH 5.3
<https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_new_in_cdh_53.html#concept_c1n_bln_tj>
shipped with java 8 in December 2014. I would be surprised if such users
were active consumers of the dev mailing list, though. Unfortunately
there's a bit of a selection bias in this list.
The other concern is if there is guaranteed compatibility between scala and
java8 for all versions you want to use (which is somewhat touched upon in
the PR). Are you thinking about supporting scala 2.10 against java 8 byte
code?
See https://groups.google.com/d/msg/druid-user/aTGQlnF1KLk/NvBPfmigAAAJ for
the similar discussion that went forward in the Druid community.
On Fri, Feb 10, 2017 at 8:47 AM Sean Owen <so...@cloudera.com> wrote:
> As you have seen, there's a WIP PR to implement removal of Java 7 support:
> https://github.com/apache/spark/pull/16871
>
> I have heard several +1s at
> https://issues.apache.org/jira/browse/SPARK-19493 but am asking for
> concerns too, now that there's a concrete change to review.
>
> If this goes in for 2.2 it can be followed by more extensive update of the
> Java code to take advantage of Java 8; this is more or less the baseline
> change.
>
> We also just removed Hadoop 2.5 support. I know there was talk about
> removing Python 2.6. I have no opinion on that myself, but, might be time
> to revive that conversation too.
>
Re: Request for comments: Java 7 removal
Posted by Sean Owen <so...@cloudera.com>.
As usual I think maintenance release branches are created ad-hoc when there
seems to be some demand. I personally would guess there will be at least
one more 2.0.x and 2.1.x maintenance release. In that sense, yeah it's not
really even the end of actively supporting a Java 7-compatible release.
On Fri, Feb 10, 2017 at 7:03 PM Denis Bolshakov <bo...@gmail.com>
wrote:
> Hello Sean,
>
> Thanks for asking.
>
> From my point of view it Ok to remove Java 7 support from Spark since 2.2
> release.
> But as a lot of users still use java 7 could you please share your vision
> about bug fix releases for 2.0 and 2.1?
>
> About python 2.6
> https://www.python.org/download/releases/2.6/
> Python 2.6 (final) was released on October 1st, 2008.
>
> If supporting python 2.6 has any costs I would definitely remove that.
>
> Kind regards,
> Denis
>
> On 10 February 2017 at 19:47, Sean Owen <so...@cloudera.com> wrote:
>
> As you have seen, there's a WIP PR to implement removal of Java 7 support:
> https://github.com/apache/spark/pull/16871
>
> I have heard several +1s at
> https://issues.apache.org/jira/browse/SPARK-19493 but am asking for
> concerns too, now that there's a concrete change to review.
>
> If this goes in for 2.2 it can be followed by more extensive update of the
> Java code to take advantage of Java 8; this is more or less the baseline
> change.
>
> We also just removed Hadoop 2.5 support. I know there was talk about
> removing Python 2.6. I have no opinion on that myself, but, might be time
> to revive that conversation too.
>
>
>
>
> --
> //with Best Regards
> --Denis Bolshakov
> e-mail: bolshakov.denis@gmail.com
>
Re: Request for comments: Java 7 removal
Posted by Denis Bolshakov <bo...@gmail.com>.
Hello Sean,
Thanks for asking.
From my point of view it Ok to remove Java 7 support from Spark since 2.2
release.
But as a lot of users still use java 7 could you please share your vision
about bug fix releases for 2.0 and 2.1?
About python 2.6
https://www.python.org/download/releases/2.6/
Python 2.6 (final) was released on October 1st, 2008.
If supporting python 2.6 has any costs I would definitely remove that.
Kind regards,
Denis
On 10 February 2017 at 19:47, Sean Owen <so...@cloudera.com> wrote:
> As you have seen, there's a WIP PR to implement removal of Java 7 support:
> https://github.com/apache/spark/pull/16871
>
> I have heard several +1s at https://issues.apache.org/
> jira/browse/SPARK-19493 but am asking for concerns too, now that there's
> a concrete change to review.
>
> If this goes in for 2.2 it can be followed by more extensive update of the
> Java code to take advantage of Java 8; this is more or less the baseline
> change.
>
> We also just removed Hadoop 2.5 support. I know there was talk about
> removing Python 2.6. I have no opinion on that myself, but, might be time
> to revive that conversation too.
>
--
//with Best Regards
--Denis Bolshakov
e-mail: bolshakov.denis@gmail.com