You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2016/07/23 21:54:09 UTC
drop java 7 support for spark 2.1.x or spark 2.2.x
dropping java 7 support was considered for spark 2.0.x but we decided
against it.
ideally dropping support for a java version should be communicated far in
advance to facilitate the transition.
is this the right time to make that decision and start communicating it
(mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
my general sense is that most cluster admins have plans to migrate to java
8 before end of year. so that could line up nicely with spark 2.2
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Sean Owen <so...@cloudera.com>.
I had favored this for 2.0 even, so favor it sooner than later.
The general shape of the argument is:
- supporting Java 7 is starting to pinch a little because of extra
builds and the inevitable gap between what the PR builder (7) tests
and what the later Java 8 tests runs show
- requiring Java 8 would allow a few minor cleanups in the code
- Java 7 is already EOL (no, I don't count paying Oracle for support)
but mostly importantly,
- anyone can keep using 2.0 if they want to use Java 7. This does
*not* require people to use Java 8. It requires them to use Java 8 to
use the very latest version.
On Sat, Jul 23, 2016 at 10:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
> dropping java 7 support was considered for spark 2.0.x but we decided
> against it.
>
> ideally dropping support for a java version should be communicated far in
> advance to facilitate the transition.
>
> is this the right time to make that decision and start communicating it
> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>
> my general sense is that most cluster admins have plans to migrate to java 8
> before end of year. so that could line up nicely with spark 2.2
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Ofir Manor <of...@equalum.io>.
BTW - "signalling ahead of time" is called deprecating, not dropping
support...
(personally I only use JDK 8 / Scala 2.11 so I'm for it)
Ofir Manor
Co-Founder & CTO | Equalum
Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io
On Sun, Jul 24, 2016 at 1:50 AM, Koert Kuipers <ko...@tresata.com> wrote:
> i care about signalling it in advance mostly. and given the performance
> differences we do have some interest in pushing towards java 8
>
> On Jul 23, 2016 6:10 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
>
> Why the push to remove Java 7 support as soon as possible (which is how I
> read your "cluster admins plan to migrate by date X, so Spark should end
> Java 7 support then, too")? First, I don't think we should be removing
> Java 7 support until some time after all or nearly all relevant clusters
> are actually no longer running on Java 6, and that targeting removal of
> support at our best guess about when admins are just *planning* to migrate
> isn't a very good idea. Second, I don't see the significant difficulty or
> harm in continuing to support Java 7 for a while longer.
>
> On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> dropping java 7 support was considered for spark 2.0.x but we decided
>> against it.
>>
>> ideally dropping support for a java version should be communicated far in
>> advance to facilitate the transition.
>>
>> is this the right time to make that decision and start communicating it
>> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>>
>> my general sense is that most cluster admins have plans to migrate to
>> java 8 before end of year. so that could line up nicely with spark 2.2
>>
>>
>
>
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Dongjoon Hyun <do...@apache.org>.
Hi, All.
What about providing a official benchmark result between `Apache Spark on
JDK7` and `Apache Spark on JDK8`?
I think that is enough for this issue since we cannot drive users.
We had better let users choose one of JDK7/JDK8 for their own benefits.
Bests,
Dongjoon.
On Sat, Jul 23, 2016 at 4:35 PM, Koert Kuipers <ko...@tresata.com> wrote:
> They don't require dropping it sooner rather than later. But signalling in
> some way that java 8 is (strongly) recommend would be good.
>
> On Jul 23, 2016 6:59 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
>
>> Sure, signalling well ahead of time is good, as is getting better
>> performance from Java 8; but do either of those interests really require
>> dropping Java 7 support sooner rather than later?
>>
>> Now, to retroactively copy edit myself, when I previously wrote "after
>> all or nearly all relevant clusters are actually no longer running on Java
>> 6", I meant "...no longer running on Java 7". We should be at a point now
>> where there aren't many Java 6 clusters left, but my sense is that there
>> are still quite a number of Java 7 clusters around, and that there will be
>> for a good while still.
>>
>> On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> i care about signalling it in advance mostly. and given the performance
>>> differences we do have some interest in pushing towards java 8
>>>
>>> On Jul 23, 2016 6:10 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
>>>
>>> Why the push to remove Java 7 support as soon as possible (which is how
>>> I read your "cluster admins plan to migrate by date X, so Spark should end
>>> Java 7 support then, too")? First, I don't think we should be removing
>>> Java 7 support until some time after all or nearly all relevant clusters
>>> are actually no longer running on Java 6, and that targeting removal of
>>> support at our best guess about when admins are just *planning* to migrate
>>> isn't a very good idea. Second, I don't see the significant difficulty or
>>> harm in continuing to support Java 7 for a while longer.
>>>
>>> On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com>
>>> wrote:
>>>
>>>> dropping java 7 support was considered for spark 2.0.x but we decided
>>>> against it.
>>>>
>>>> ideally dropping support for a java version should be communicated far
>>>> in advance to facilitate the transition.
>>>>
>>>> is this the right time to make that decision and start communicating it
>>>> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>>>>
>>>> my general sense is that most cluster admins have plans to migrate to
>>>> java 8 before end of year. so that could line up nicely with spark 2.2
>>>>
>>>>
>>>
>>>
>>
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Koert Kuipers <ko...@tresata.com>.
They don't require dropping it sooner rather than later. But signalling in
some way that java 8 is (strongly) recommend would be good.
On Jul 23, 2016 6:59 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
> Sure, signalling well ahead of time is good, as is getting better
> performance from Java 8; but do either of those interests really require
> dropping Java 7 support sooner rather than later?
>
> Now, to retroactively copy edit myself, when I previously wrote "after
> all or nearly all relevant clusters are actually no longer running on Java
> 6", I meant "...no longer running on Java 7". We should be at a point now
> where there aren't many Java 6 clusters left, but my sense is that there
> are still quite a number of Java 7 clusters around, and that there will be
> for a good while still.
>
> On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> i care about signalling it in advance mostly. and given the performance
>> differences we do have some interest in pushing towards java 8
>>
>> On Jul 23, 2016 6:10 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
>>
>> Why the push to remove Java 7 support as soon as possible (which is how I
>> read your "cluster admins plan to migrate by date X, so Spark should end
>> Java 7 support then, too")? First, I don't think we should be removing
>> Java 7 support until some time after all or nearly all relevant clusters
>> are actually no longer running on Java 6, and that targeting removal of
>> support at our best guess about when admins are just *planning* to migrate
>> isn't a very good idea. Second, I don't see the significant difficulty or
>> harm in continuing to support Java 7 for a while longer.
>>
>> On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> dropping java 7 support was considered for spark 2.0.x but we decided
>>> against it.
>>>
>>> ideally dropping support for a java version should be communicated far
>>> in advance to facilitate the transition.
>>>
>>> is this the right time to make that decision and start communicating it
>>> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>>>
>>> my general sense is that most cluster admins have plans to migrate to
>>> java 8 before end of year. so that could line up nicely with spark 2.2
>>>
>>>
>>
>>
>
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Steve Loughran <st...@hortonworks.com>.
On 23 Jul 2016, at 23:59, Mark Hamstra <ma...@clearstorydata.com>> wrote:
Sure, signalling well ahead of time is good, as is getting better performance from Java 8; but do either of those interests really require dropping Java 7 support sooner rather than later?
Now, to retroactively copy edit myself, when I previously wrote "after all or nearly all relevant clusters are actually no longer running on Java 6", I meant "...no longer running on Java 7". We should be at a point now where there aren't many Java 6 clusters left, but my sense is that there are still quite a number of Java 7 clusters around, and that there will be for a good while still.
1. There are a lot of Hadoop clusters on Java 7; the migration to Java 8 there is pretty slow due to inertia of existing apps.
2. But...you can run Java 8 code in a Hadoop cluster, if you set up the JVM properly
3. And orgs playing with spark tend not to be the ones running older Hadoop versions
Ignoring runtime issues, mixing versions of: Java & Scala complicate the dev and test matrix; getting a Java 7 JDK to build and test on is getting harder and harder, so most developers are using Java 8. Even if using java 7 language, there's the differences in the JRE libraries. Scala code has it easier language wise, but there are enough diffs from 2.10 and 2.11 that again, life is complex. There's invariably a conflict between developers "I want to play with the latest toys!" and ops "It worked in 2012, let's keep using it"
Meanwhile, Java 9 is coming, which is going to break a lot of things, log4j 1.x included.
This is going to be fairly traumatic all round, as log4j 2, while a better design, doesn't parse the 1.x properties files (https://blogs.apache.org/logging/entry/moving_on_to_log4j_2).
While Java 9 is something nobody will use in production this year, it's coming, and it's something which has a lot of consequences up the stack. Being able to stop worrying about java 7 backwards compatibility would save time which can now be directed to worrying about java 9. Indeed, the ASF builds team are being offered Java 9 EARs, along with Java 8u112"
---------------
From: Rory O'Donnell <ro...@oracle.com>>
Subject: Early Access builds of JDK 8u112 b03, JDK 9 b128 are available on java.net<http://java.net>
Date: 22 July 2016 at 10:30:19 BST
To: <st...@yahoo.de>>
Cc: <ro...@oracle.com>>, Dalibor Topic <da...@oracle.com>>, Balchandra Vaidya <ba...@oracle.com>>, Muneer Kolarkunnu <ab...@oracle.com>>, <ga...@16degrees.com.au>>, <bu...@apache.org>>
Reply-To: <bu...@apache.org>>
Hi Mark & Gavin,
Early Access b128 <https://jdk9.java.net/download/> for JDK 9 is available on java.net<http://java.net>, summary of changes are listed here <http://www.java.net/download/java/jdk9/changes/jdk-9+128.html>.
Early Access b127 <https://jdk9.java.net/jigsaw/> (#5304) for JDK 9 with Project Jigsaw is available on java.net<http://java.net>, summary of changes are listed here <http://download.java.net/java/jigsaw/archive/127/binaries/jdk-9+127.html>
Early Access b03 <https://jdk8.java.net/download.html> for JDK 8u112 is available on java.net<http://java.net>, summary of changes are listed here <http://www.java.net/download/java/jdk8u112/changes/jdk8u112-b03.html>
Alan Bateman posted new EA builds contain initial implementation of current proposals , more info [0]
The jigsaw/jake forest has been updated with an initial
implementation of the proposals that Mark brought to the
jpms-spec-experts mailing list last week. For those that don't build
from source then the EA build/downloads [1] has also been refreshed.
Rgds,Rory
[0] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008467.html
[1] https://jdk9.java.net/jigsaw/
--
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Reynold Xin <rx...@databricks.com>.
BTW I created a JIRA ticket for tracking:
https://issues.apache.org/jira/browse/SPARK-19493
We of course shouldn't do anything until we achieve consensus.
On Tue, Feb 7, 2017 at 3:47 PM, Reynold Xin <rx...@databricks.com> wrote:
> Bumping this.
>
> Given we see the occassional build breaks with Java 8, we should
> reconsider this and do it for 2.2 or 2.3. By the time 2.2 is released, it
> will almost be an year since this thread started.
>
>
> On Sun, Jul 24, 2016 at 12:59 AM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>
>> Sure, signalling well ahead of time is good, as is getting better
>> performance from Java 8; but do either of those interests really require
>> dropping Java 7 support sooner rather than later?
>>
>> Now, to retroactively copy edit myself, when I previously wrote "after
>> all or nearly all relevant clusters are actually no longer running on Java
>> 6", I meant "...no longer running on Java 7". We should be at a point now
>> where there aren't many Java 6 clusters left, but my sense is that there
>> are still quite a number of Java 7 clusters around, and that there will be
>> for a good while still.
>>
>> On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> i care about signalling it in advance mostly. and given the performance
>>> differences we do have some interest in pushing towards java 8
>>>
>>> On Jul 23, 2016 6:10 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
>>>
>>> Why the push to remove Java 7 support as soon as possible (which is how
>>> I read your "cluster admins plan to migrate by date X, so Spark should end
>>> Java 7 support then, too")? First, I don't think we should be removing
>>> Java 7 support until some time after all or nearly all relevant clusters
>>> are actually no longer running on Java 6, and that targeting removal of
>>> support at our best guess about when admins are just *planning* to migrate
>>> isn't a very good idea. Second, I don't see the significant difficulty or
>>> harm in continuing to support Java 7 for a while longer.
>>>
>>> On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com>
>>> wrote:
>>>
>>>> dropping java 7 support was considered for spark 2.0.x but we decided
>>>> against it.
>>>>
>>>> ideally dropping support for a java version should be communicated far
>>>> in advance to facilitate the transition.
>>>>
>>>> is this the right time to make that decision and start communicating it
>>>> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>>>>
>>>> my general sense is that most cluster admins have plans to migrate to
>>>> java 8 before end of year. so that could line up nicely with spark 2.2
>>>>
>>>>
>>>
>>>
>>
>
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Reynold Xin <rx...@databricks.com>.
Bumping this.
Given we see the occassional build breaks with Java 8, we should reconsider
this and do it for 2.2 or 2.3. By the time 2.2 is released, it will almost
be an year since this thread started.
On Sun, Jul 24, 2016 at 12:59 AM, Mark Hamstra <ma...@clearstorydata.com>
wrote:
> Sure, signalling well ahead of time is good, as is getting better
> performance from Java 8; but do either of those interests really require
> dropping Java 7 support sooner rather than later?
>
> Now, to retroactively copy edit myself, when I previously wrote "after
> all or nearly all relevant clusters are actually no longer running on Java
> 6", I meant "...no longer running on Java 7". We should be at a point now
> where there aren't many Java 6 clusters left, but my sense is that there
> are still quite a number of Java 7 clusters around, and that there will be
> for a good while still.
>
> On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> i care about signalling it in advance mostly. and given the performance
>> differences we do have some interest in pushing towards java 8
>>
>> On Jul 23, 2016 6:10 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
>>
>> Why the push to remove Java 7 support as soon as possible (which is how I
>> read your "cluster admins plan to migrate by date X, so Spark should end
>> Java 7 support then, too")? First, I don't think we should be removing
>> Java 7 support until some time after all or nearly all relevant clusters
>> are actually no longer running on Java 6, and that targeting removal of
>> support at our best guess about when admins are just *planning* to migrate
>> isn't a very good idea. Second, I don't see the significant difficulty or
>> harm in continuing to support Java 7 for a while longer.
>>
>> On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>>> dropping java 7 support was considered for spark 2.0.x but we decided
>>> against it.
>>>
>>> ideally dropping support for a java version should be communicated far
>>> in advance to facilitate the transition.
>>>
>>> is this the right time to make that decision and start communicating it
>>> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>>>
>>> my general sense is that most cluster admins have plans to migrate to
>>> java 8 before end of year. so that could line up nicely with spark 2.2
>>>
>>>
>>
>>
>
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Sure, signalling well ahead of time is good, as is getting better
performance from Java 8; but do either of those interests really require
dropping Java 7 support sooner rather than later?
Now, to retroactively copy edit myself, when I previously wrote "after all
or nearly all relevant clusters are actually no longer running on Java 6",
I meant "...no longer running on Java 7". We should be at a point now
where there aren't many Java 6 clusters left, but my sense is that there
are still quite a number of Java 7 clusters around, and that there will be
for a good while still.
On Sat, Jul 23, 2016 at 3:50 PM, Koert Kuipers <ko...@tresata.com> wrote:
> i care about signalling it in advance mostly. and given the performance
> differences we do have some interest in pushing towards java 8
>
> On Jul 23, 2016 6:10 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
>
> Why the push to remove Java 7 support as soon as possible (which is how I
> read your "cluster admins plan to migrate by date X, so Spark should end
> Java 7 support then, too")? First, I don't think we should be removing
> Java 7 support until some time after all or nearly all relevant clusters
> are actually no longer running on Java 6, and that targeting removal of
> support at our best guess about when admins are just *planning* to migrate
> isn't a very good idea. Second, I don't see the significant difficulty or
> harm in continuing to support Java 7 for a while longer.
>
> On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> dropping java 7 support was considered for spark 2.0.x but we decided
>> against it.
>>
>> ideally dropping support for a java version should be communicated far in
>> advance to facilitate the transition.
>>
>> is this the right time to make that decision and start communicating it
>> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>>
>> my general sense is that most cluster admins have plans to migrate to
>> java 8 before end of year. so that could line up nicely with spark 2.2
>>
>>
>
>
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Koert Kuipers <ko...@tresata.com>.
i care about signalling it in advance mostly. and given the performance
differences we do have some interest in pushing towards java 8
On Jul 23, 2016 6:10 PM, "Mark Hamstra" <ma...@clearstorydata.com> wrote:
Why the push to remove Java 7 support as soon as possible (which is how I
read your "cluster admins plan to migrate by date X, so Spark should end
Java 7 support then, too")? First, I don't think we should be removing
Java 7 support until some time after all or nearly all relevant clusters
are actually no longer running on Java 6, and that targeting removal of
support at our best guess about when admins are just *planning* to migrate
isn't a very good idea. Second, I don't see the significant difficulty or
harm in continuing to support Java 7 for a while longer.
On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
> dropping java 7 support was considered for spark 2.0.x but we decided
> against it.
>
> ideally dropping support for a java version should be communicated far in
> advance to facilitate the transition.
>
> is this the right time to make that decision and start communicating it
> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>
> my general sense is that most cluster admins have plans to migrate to java
> 8 before end of year. so that could line up nicely with spark 2.2
>
>
Re: drop java 7 support for spark 2.1.x or spark 2.2.x
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Why the push to remove Java 7 support as soon as possible (which is how I
read your "cluster admins plan to migrate by date X, so Spark should end
Java 7 support then, too")? First, I don't think we should be removing
Java 7 support until some time after all or nearly all relevant clusters
are actually no longer running on Java 6, and that targeting removal of
support at our best guess about when admins are just *planning* to migrate
isn't a very good idea. Second, I don't see the significant difficulty or
harm in continuing to support Java 7 for a while longer.
On Sat, Jul 23, 2016 at 2:54 PM, Koert Kuipers <ko...@tresata.com> wrote:
> dropping java 7 support was considered for spark 2.0.x but we decided
> against it.
>
> ideally dropping support for a java version should be communicated far in
> advance to facilitate the transition.
>
> is this the right time to make that decision and start communicating it
> (mailing list, jira, etc.)? perhaps for spark 2.1.x or spark 2.2.x?
>
> my general sense is that most cluster admins have plans to migrate to java
> 8 before end of year. so that could line up nicely with spark 2.2
>
>