You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sean Owen <so...@cloudera.com> on 2017/02/25 19:10:20 UTC
Re: Straw poll: dropping support for things like Scala 2.10
I want to bring up the issue of Scala 2.10 support again, to see how people
feel about it. Key opinions from the previous responses, I think:
Cody: only drop 2.10 support when 2.12 support is added
Koert: we need all dependencies to support 2.12; Scala updates are pretty
transparent to IT/ops
Ofir: make sure to deprecate 2.10 in Spark 2.1
Reynold: let’s maybe remove support for Scala 2.10 and Java 7 in Spark 2.2
Matei: let’s not remove things unless they’re burdensome for the project;
some people are still on old environments that their IT can’t easily update
Scala 2.10 support was deprecated in 2.1, and we did remove Java 7 support
for 2.2. https://issues.apache.org/jira/browse/SPARK-14220 tracks the work
to support 2.12, and there is progress, especially in dependencies
supporting 2.12.
It looks like 2.12 support may even entail a breaking change as documented
in https://issues.apache.org/jira/browse/SPARK-14643 and will mean dropping
Kafka 0.8, for example. In any event it’s going to take some surgery and a
few hacks to make one code base work across 2.11 and 2.12. I don’t see this
happening for Spark 2.2.0 because there are just a few weeks left.
Supporting three versions at once is probably infeasible, so dropping 2.10
should precede 2.12 support. Right now, I would like to make progress
towards changes that 2.12 will require but that 2.11/2.10 can support. For
example, we have to update scalatest, breeze, chill, etc. and can do that
before 2.12 is enabled. However I’m finding making those changes tricky or
maybe impossible in one case while 2.10 is still supported.
For 2.2.0, I’m wondering if it makes sense to go ahead and drop 2.10
support, and even get in additional prep work for 2.12, into the 2.2.0
release. The move to support 2.12 in 2.3.0 would then be a smaller change.
It isn’t strictly necessary. We could delay all of that until after 2.2.0
and get it all done between 2.2.0 and 2.3.0. But I wonder if 2.10 is legacy
enough at this stage to drop for Spark 2.2.0?
I don’t feel strongly about it but there are some reasonable arguments for
dropping it:
- 2.10 doesn’t technically support Java 8, though we do have it working
still even after requiring Java 8
- Safe to say virtually all common _2.10 libraries has a _2.11 counterpart
at this point?
- 2.10.x was “EOL” in September 2015 with the final 2.10.6 release
- For a vendor viewpoint: CDH only supports Scala 2.11 with Spark 2.x
Before I open a JIRA, just soliciting opinions.
On Tue, Oct 25, 2016 at 4:36 PM Sean Owen <so...@cloudera.com> wrote:
> I'd like to gauge where people stand on the issue of dropping support for
> a few things that were considered for 2.0.
>
> First: Scala 2.10. We've seen a number of build breakages this week
> because the PR builder only tests 2.11. No big deal at this stage, but, it
> did cause me to wonder whether it's time to plan to drop 2.10 support,
> especially with 2.12 coming soon.
>
> Next, Java 7. It's reasonably old and out of public updates at this stage.
> It's not that painful to keep supporting, to be honest. It would simplify
> some bits of code, some scripts, some testing.
>
> Hadoop versions: I think the the general argument is that most anyone
> would be using, at the least, 2.6, and it would simplify some code that has
> to reflect to use not-even-that-new APIs. It would remove some moderate
> complexity in the build.
>
>
> "When" is a tricky question. Although it's a little aggressive for minor
> releases, I think these will all happen before 3.x regardless. 2.1.0 is not
> out of the question, though coming soon. What about ... 2.2.0?
>
>
> Although I tend to favor dropping support, I'm mostly asking for current
> opinions.
>
Re: Straw poll: dropping support for things like Scala 2.10
Posted by Sean Owen <so...@cloudera.com>.
Let's track further discussion at
https://issues.apache.org/jira/browse/SPARK-19810
I am also in favor of removing Scala 2.10 support, and will open a WIP to
discuss the change, but am not yet sure whether there are objections or
deeper support for this.
On Thu, Mar 2, 2017 at 7:51 PM Russell Spitzer <ru...@gmail.com>
wrote:
> +1 on removing 2.10
>
>
> On Thu, Mar 2, 2017 at 8:51 AM Koert Kuipers <ko...@tresata.com> wrote:
>
> given the issues with scala 2.10 and java 8 i am in favor of dropping
> scala 2.10 in next release
>
> On Sat, Feb 25, 2017 at 2:10 PM, Sean Owen <so...@cloudera.com> wrote:
>
> I want to bring up the issue of Scala 2.10 support again, to see how
> people feel about it. Key opinions from the previous responses, I think:
>
> Cody: only drop 2.10 support when 2.12 support is added
> Koert: we need all dependencies to support 2.12; Scala updates are pretty
> transparent to IT/ops
> Ofir: make sure to deprecate 2.10 in Spark 2.1
> Reynold: let’s maybe remove support for Scala 2.10 and Java 7 in Spark 2.2
> Matei: let’s not remove things unless they’re burdensome for the project;
> some people are still on old environments that their IT can’t easily update
>
> Scala 2.10 support was deprecated in 2.1, and we did remove Java 7 support
> for 2.2. https://issues.apache.org/jira/browse/SPARK-14220 tracks the
> work to support 2.12, and there is progress, especially in dependencies
> supporting 2.12.
>
> It looks like 2.12 support may even entail a breaking change as documented
> in https://issues.apache.org/jira/browse/SPARK-14643 and will mean
> dropping Kafka 0.8, for example. In any event it’s going to take some
> surgery and a few hacks to make one code base work across 2.11 and 2.12. I
> don’t see this happening for Spark 2.2.0 because there are just a few weeks
> left.
>
> Supporting three versions at once is probably infeasible, so dropping 2.10
> should precede 2.12 support. Right now, I would like to make progress
> towards changes that 2.12 will require but that 2.11/2.10 can support. For
> example, we have to update scalatest, breeze, chill, etc. and can do that
> before 2.12 is enabled. However I’m finding making those changes tricky or
> maybe impossible in one case while 2.10 is still supported.
>
> For 2.2.0, I’m wondering if it makes sense to go ahead and drop 2.10
> support, and even get in additional prep work for 2.12, into the 2.2.0
> release. The move to support 2.12 in 2.3.0 would then be a smaller change.
> It isn’t strictly necessary. We could delay all of that until after 2.2.0
> and get it all done between 2.2.0 and 2.3.0. But I wonder if 2.10 is legacy
> enough at this stage to drop for Spark 2.2.0?
>
> I don’t feel strongly about it but there are some reasonable arguments for
> dropping it:
>
> - 2.10 doesn’t technically support Java 8, though we do have it working
> still even after requiring Java 8
> - Safe to say virtually all common _2.10 libraries has a _2.11 counterpart
> at this point?
> - 2.10.x was “EOL” in September 2015 with the final 2.10.6 release
> - For a vendor viewpoint: CDH only supports Scala 2.11 with Spark 2.x
>
> Before I open a JIRA, just soliciting opinions.
>
>
> On Tue, Oct 25, 2016 at 4:36 PM Sean Owen <so...@cloudera.com> wrote:
>
> I'd like to gauge where people stand on the issue of dropping support for
> a few things that were considered for 2.0.
>
> First: Scala 2.10. We've seen a number of build breakages this week
> because the PR builder only tests 2.11. No big deal at this stage, but, it
> did cause me to wonder whether it's time to plan to drop 2.10 support,
> especially with 2.12 coming soon.
>
> Next, Java 7. It's reasonably old and out of public updates at this stage.
> It's not that painful to keep supporting, to be honest. It would simplify
> some bits of code, some scripts, some testing.
>
> Hadoop versions: I think the the general argument is that most anyone
> would be using, at the least, 2.6, and it would simplify some code that has
> to reflect to use not-even-that-new APIs. It would remove some moderate
> complexity in the build.
>
>
> "When" is a tricky question. Although it's a little aggressive for minor
> releases, I think these will all happen before 3.x regardless. 2.1.0 is not
> out of the question, though coming soon. What about ... 2.2.0?
>
>
> Although I tend to favor dropping support, I'm mostly asking for current
> opinions.
>
>
>
Re: Straw poll: dropping support for things like Scala 2.10
Posted by Russell Spitzer <ru...@gmail.com>.
+1 on removing 2.10
On Thu, Mar 2, 2017 at 8:51 AM Koert Kuipers <ko...@tresata.com> wrote:
given the issues with scala 2.10 and java 8 i am in favor of dropping scala
2.10 in next release
On Sat, Feb 25, 2017 at 2:10 PM, Sean Owen <so...@cloudera.com> wrote:
I want to bring up the issue of Scala 2.10 support again, to see how people
feel about it. Key opinions from the previous responses, I think:
Cody: only drop 2.10 support when 2.12 support is added
Koert: we need all dependencies to support 2.12; Scala updates are pretty
transparent to IT/ops
Ofir: make sure to deprecate 2.10 in Spark 2.1
Reynold: let’s maybe remove support for Scala 2.10 and Java 7 in Spark 2.2
Matei: let’s not remove things unless they’re burdensome for the project;
some people are still on old environments that their IT can’t easily update
Scala 2.10 support was deprecated in 2.1, and we did remove Java 7 support
for 2.2. https://issues.apache.org/jira/browse/SPARK-14220 tracks the work
to support 2.12, and there is progress, especially in dependencies
supporting 2.12.
It looks like 2.12 support may even entail a breaking change as documented
in https://issues.apache.org/jira/browse/SPARK-14643 and will mean dropping
Kafka 0.8, for example. In any event it’s going to take some surgery and a
few hacks to make one code base work across 2.11 and 2.12. I don’t see this
happening for Spark 2.2.0 because there are just a few weeks left.
Supporting three versions at once is probably infeasible, so dropping 2.10
should precede 2.12 support. Right now, I would like to make progress
towards changes that 2.12 will require but that 2.11/2.10 can support. For
example, we have to update scalatest, breeze, chill, etc. and can do that
before 2.12 is enabled. However I’m finding making those changes tricky or
maybe impossible in one case while 2.10 is still supported.
For 2.2.0, I’m wondering if it makes sense to go ahead and drop 2.10
support, and even get in additional prep work for 2.12, into the 2.2.0
release. The move to support 2.12 in 2.3.0 would then be a smaller change.
It isn’t strictly necessary. We could delay all of that until after 2.2.0
and get it all done between 2.2.0 and 2.3.0. But I wonder if 2.10 is legacy
enough at this stage to drop for Spark 2.2.0?
I don’t feel strongly about it but there are some reasonable arguments for
dropping it:
- 2.10 doesn’t technically support Java 8, though we do have it working
still even after requiring Java 8
- Safe to say virtually all common _2.10 libraries has a _2.11 counterpart
at this point?
- 2.10.x was “EOL” in September 2015 with the final 2.10.6 release
- For a vendor viewpoint: CDH only supports Scala 2.11 with Spark 2.x
Before I open a JIRA, just soliciting opinions.
On Tue, Oct 25, 2016 at 4:36 PM Sean Owen <so...@cloudera.com> wrote:
I'd like to gauge where people stand on the issue of dropping support for a
few things that were considered for 2.0.
First: Scala 2.10. We've seen a number of build breakages this week because
the PR builder only tests 2.11. No big deal at this stage, but, it did
cause me to wonder whether it's time to plan to drop 2.10 support,
especially with 2.12 coming soon.
Next, Java 7. It's reasonably old and out of public updates at this stage.
It's not that painful to keep supporting, to be honest. It would simplify
some bits of code, some scripts, some testing.
Hadoop versions: I think the the general argument is that most anyone would
be using, at the least, 2.6, and it would simplify some code that has to
reflect to use not-even-that-new APIs. It would remove some moderate
complexity in the build.
"When" is a tricky question. Although it's a little aggressive for minor
releases, I think these will all happen before 3.x regardless. 2.1.0 is not
out of the question, though coming soon. What about ... 2.2.0?
Although I tend to favor dropping support, I'm mostly asking for current
opinions.
Re: Straw poll: dropping support for things like Scala 2.10
Posted by Koert Kuipers <ko...@tresata.com>.
given the issues with scala 2.10 and java 8 i am in favor of dropping scala
2.10 in next release
On Sat, Feb 25, 2017 at 2:10 PM, Sean Owen <so...@cloudera.com> wrote:
> I want to bring up the issue of Scala 2.10 support again, to see how
> people feel about it. Key opinions from the previous responses, I think:
>
> Cody: only drop 2.10 support when 2.12 support is added
> Koert: we need all dependencies to support 2.12; Scala updates are pretty
> transparent to IT/ops
> Ofir: make sure to deprecate 2.10 in Spark 2.1
> Reynold: let’s maybe remove support for Scala 2.10 and Java 7 in Spark 2.2
> Matei: let’s not remove things unless they’re burdensome for the project;
> some people are still on old environments that their IT can’t easily update
>
> Scala 2.10 support was deprecated in 2.1, and we did remove Java 7 support
> for 2.2. https://issues.apache.org/jira/browse/SPARK-14220 tracks the
> work to support 2.12, and there is progress, especially in dependencies
> supporting 2.12.
>
> It looks like 2.12 support may even entail a breaking change as documented
> in https://issues.apache.org/jira/browse/SPARK-14643 and will mean
> dropping Kafka 0.8, for example. In any event it’s going to take some
> surgery and a few hacks to make one code base work across 2.11 and 2.12. I
> don’t see this happening for Spark 2.2.0 because there are just a few weeks
> left.
>
> Supporting three versions at once is probably infeasible, so dropping 2.10
> should precede 2.12 support. Right now, I would like to make progress
> towards changes that 2.12 will require but that 2.11/2.10 can support. For
> example, we have to update scalatest, breeze, chill, etc. and can do that
> before 2.12 is enabled. However I’m finding making those changes tricky or
> maybe impossible in one case while 2.10 is still supported.
>
> For 2.2.0, I’m wondering if it makes sense to go ahead and drop 2.10
> support, and even get in additional prep work for 2.12, into the 2.2.0
> release. The move to support 2.12 in 2.3.0 would then be a smaller change.
> It isn’t strictly necessary. We could delay all of that until after 2.2.0
> and get it all done between 2.2.0 and 2.3.0. But I wonder if 2.10 is legacy
> enough at this stage to drop for Spark 2.2.0?
>
> I don’t feel strongly about it but there are some reasonable arguments for
> dropping it:
>
> - 2.10 doesn’t technically support Java 8, though we do have it working
> still even after requiring Java 8
> - Safe to say virtually all common _2.10 libraries has a _2.11 counterpart
> at this point?
> - 2.10.x was “EOL” in September 2015 with the final 2.10.6 release
> - For a vendor viewpoint: CDH only supports Scala 2.11 with Spark 2.x
>
> Before I open a JIRA, just soliciting opinions.
>
>
> On Tue, Oct 25, 2016 at 4:36 PM Sean Owen <so...@cloudera.com> wrote:
>
>> I'd like to gauge where people stand on the issue of dropping support for
>> a few things that were considered for 2.0.
>>
>> First: Scala 2.10. We've seen a number of build breakages this week
>> because the PR builder only tests 2.11. No big deal at this stage, but, it
>> did cause me to wonder whether it's time to plan to drop 2.10 support,
>> especially with 2.12 coming soon.
>>
>> Next, Java 7. It's reasonably old and out of public updates at this
>> stage. It's not that painful to keep supporting, to be honest. It would
>> simplify some bits of code, some scripts, some testing.
>>
>> Hadoop versions: I think the the general argument is that most anyone
>> would be using, at the least, 2.6, and it would simplify some code that has
>> to reflect to use not-even-that-new APIs. It would remove some moderate
>> complexity in the build.
>>
>>
>> "When" is a tricky question. Although it's a little aggressive for minor
>> releases, I think these will all happen before 3.x regardless. 2.1.0 is not
>> out of the question, though coming soon. What about ... 2.2.0?
>>
>>
>> Although I tend to favor dropping support, I'm mostly asking for current
>> opinions.
>>
>