You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Koert Kuipers <ko...@tresata.com> on 2016/04/02 01:10:25 UTC
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
as long as we don't lock ourselves into supporting scala 2.10 for the
entire spark 2 lifespan it sounds reasonable to me
On Wed, Mar 30, 2016 at 3:25 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> +1 to Matei's reasoning.
>
> On Wed, Mar 30, 2016 at 9:21 AM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
>> I agree that putting it in 2.0 doesn't mean keeping Scala 2.10 for the
>> entire 2.x line. My vote is to keep Scala 2.10 in Spark 2.0, because it's
>> the default version we built with in 1.x. We want to make the transition
>> from 1.x to 2.0 as easy as possible. In 2.0, we'll have the default
>> downloads be for Scala 2.11, so people will more easily move, but we
>> shouldn't create obstacles that lead to fragmenting the community and
>> slowing down Spark 2.0's adoption. I've seen companies that stayed on an
>> old Scala version for multiple years because switching it, or mixing
>> versions, would affect the company's entire codebase.
>>
>> Matei
>>
>> On Mar 30, 2016, at 12:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>
>> oh wow, had no idea it got ripped out
>>
>> On Wed, Mar 30, 2016 at 11:50 AM, Mark Hamstra <ma...@clearstorydata.com>
>> wrote:
>>
>>> No, with 2.0 Spark really doesn't use Akka:
>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L744
>>>
>>> On Wed, Mar 30, 2016 at 9:10 AM, Koert Kuipers <ko...@tresata.com>
>>> wrote:
>>>
>>>> Spark still runs on akka. So if you want the benefits of the latest
>>>> akka (not saying we do, was just an example) then you need to drop scala
>>>> 2.10
>>>> On Mar 30, 2016 10:44 AM, "Cody Koeninger" <co...@koeninger.org> wrote:
>>>>
>>>>> I agree with Mark in that I don't see how supporting scala 2.10 for
>>>>> spark 2.0 implies supporting it for all of spark 2.x
>>>>>
>>>>> Regarding Koert's comment on akka, I thought all akka dependencies
>>>>> have been removed from spark after SPARK-7997 and the recent removal
>>>>> of external/akka
>>>>>
>>>>> On Wed, Mar 30, 2016 at 9:36 AM, Mark Hamstra <ma...@clearstorydata.com>
>>>>> wrote:
>>>>> > Dropping Scala 2.10 support has to happen at some point, so I'm not
>>>>> > fundamentally opposed to the idea; but I've got questions about how
>>>>> we go
>>>>> > about making the change and what degree of negative consequences we
>>>>> are
>>>>> > willing to accept. Until now, we have been saying that 2.10 support
>>>>> will be
>>>>> > continued in Spark 2.0.0. Switching to 2.11 will be non-trivial for
>>>>> some
>>>>> > Spark users, so abruptly dropping 2.10 support is very likely to
>>>>> delay
>>>>> > migration to Spark 2.0 for those users.
>>>>> >
>>>>> > What about continuing 2.10 support in 2.0.x, but repeatedly making an
>>>>> > obvious announcement in multiple places that such support is
>>>>> deprecated,
>>>>> > that we are not committed to maintaining it throughout 2.x, and that
>>>>> it is,
>>>>> > in fact, scheduled to be removed in 2.1.0?
>>>>> >
>>>>> > On Wed, Mar 30, 2016 at 7:45 AM, Sean Owen <so...@cloudera.com>
>>>>> wrote:
>>>>> >>
>>>>> >> (This should fork as its own thread, though it began during
>>>>> discussion
>>>>> >> of whether to continue Java 7 support in Spark 2.x.)
>>>>> >>
>>>>> >> Simply: would like to more clearly take the temperature of all
>>>>> >> interested parties about whether to support Scala 2.10 in the Spark
>>>>> >> 2.x lifecycle. Some of the arguments appear to be:
>>>>> >>
>>>>> >> Pro
>>>>> >> - Some third party dependencies do not support Scala 2.11+ yet and
>>>>> so
>>>>> >> would not be usable in a Spark app
>>>>> >>
>>>>> >> Con
>>>>> >> - Lower maintenance overhead -- no separate 2.10 build,
>>>>> >> cross-building, tests to check, esp considering support of 2.12 will
>>>>> >> be needed
>>>>> >> - Can use 2.11+ features freely
>>>>> >> - 2.10 was EOL in late 2014 and Spark 2.x lifecycle is years to come
>>>>> >>
>>>>> >> I would like to not support 2.10 for Spark 2.x, myself.
>>>>> >>
>>>>> >>
>>>>> ---------------------------------------------------------------------
>>>>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> >> For additional commands, e-mail: dev-help@spark.apache.org
>>>>> >>
>>>>> >
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>
>>>>>
>>>
>>
>>
>
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Mark Hamstra <ma...@clearstorydata.com>.
Yes, some organization do lag behind the current release by sometimes a
significant amount. That is a bug, not a feature -- and one that increases
pressure toward fragmentation of the Spark community. To date, that hasn't
been a significant problem, and I think that is mainly because the factors
motivating a decision not to upgrade in a timely fashion are almost
entirely internal to a lagging organization -- Spark itself has tried to
present minimal impediments to upgrading as soon as a new release is
available.
Changing the supported Java and Scala versions within the same quarter in
which the next version is scheduled for release would represent more than a
minimal impediment, and would increase fragmentation pressure to a degree
with which I am not entirely comfortable.
On Mon, Apr 11, 2016 at 12:10 PM, Daniel Siegmann <
daniel.siegmann@teamaol.com> wrote:
> On Wed, Apr 6, 2016 at 2:57 PM, Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>
> ... My concern is that either of those options will take more resources
>> than some Spark users will have available in the ~3 months remaining before
>> Spark 2.0.0, which will cause fragmentation into Spark 1.x and Spark 2.x
>> user communities. ...
>>
>
> It's not as if everyone is going to switch over to Spark 2.0.0 on release
> day anyway. It's not that unusual to see posts on the user list from people
> who are a version or two behind. I think a few extra months lag time will
> be OK for a major version.
>
> Besides, in my experience if you give people more time to upgrade, they're
> just going to kick the can down the road a ways and you'll eventually end
> up with the same problem. I don't see a good reason to *not* drop Java 7
> and Scala 2.10 support with Spark 2.0.0. Time to bite the bullet. If
> companies stick with Spark 1.x and find themselves missing the new features
> in the 2.x line, that will be a good motivation for them to upgrade.
>
> ~Daniel Siegmann
>
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Daniel Siegmann <da...@teamaol.com>.
On Wed, Apr 6, 2016 at 2:57 PM, Mark Hamstra <ma...@clearstorydata.com>
wrote:
... My concern is that either of those options will take more resources
> than some Spark users will have available in the ~3 months remaining before
> Spark 2.0.0, which will cause fragmentation into Spark 1.x and Spark 2.x
> user communities. ...
>
It's not as if everyone is going to switch over to Spark 2.0.0 on release
day anyway. It's not that unusual to see posts on the user list from people
who are a version or two behind. I think a few extra months lag time will
be OK for a major version.
Besides, in my experience if you give people more time to upgrade, they're
just going to kick the can down the road a ways and you'll eventually end
up with the same problem. I don't see a good reason to *not* drop Java 7
and Scala 2.10 support with Spark 2.0.0. Time to bite the bullet. If
companies stick with Spark 1.x and find themselves missing the new features
in the 2.x line, that will be a good motivation for them to upgrade.
~Daniel Siegmann
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Mark Hamstra <ma...@clearstorydata.com>.
>
> are you sure that library is being properly maintained?
>
Almost by definition the answer to that is "No; a library that hasn't been
upgraded to Scala 2.11 is not being properly maintained." That means that
a user of such a library is already facing the choice of whether to take on
the maintenance burden to bring the library up to date or to replace the
unmaintained library with an alternative that is being properly
maintained. My concern is that either of those options will take more
resources than some Spark users will have available in the ~3 months
remaining before Spark 2.0.0, which will cause fragmentation into Spark 1.x
and Spark 2.x user communities.
Which is the bigger risk and cost, maintaining Scala 2.10 support for a
while longer or fragmenting the Spark user community with the 2.0.0
release?
On Wed, Apr 6, 2016 at 10:51 AM, Dean Wampler <de...@gmail.com> wrote:
> A few other reasons to drop 2.10 support sooner rather than later.
>
> - We at Lightbend are evaluating some fundamental changes to the REPL
> to make it work better for large heaps, especially for Spark. There are
> other recent and planned enhancements. This work will be benefit notebook
> users, too. However, we won't back port these improvements to 2.10.
> - Scala 2.12 is coming out midyear. It will require Java 8, which
> means it will produce dramatically smaller code (by exploiting lambdas
> instead of custom class generation for functions) and it will offer some
> performance improvements. Hopefully Spark will will support it as an
> optional Scala version relatively quickly after availability, which means
> it would be nice to avoid supporting 3 versions of Scala.
>
> Using Scala 2.10 at this point is like using Java 1.6, seriously out of
> date. If you're using libraries that still require 2.10, are you sure that
> library is being properly maintained? Or is it a legacy dependency that
> should be eliminated before it becomes a liability? Even if you can't
> upgrade Scala versions in the next few months, you can certainly continue
> using Spark 1.X until you're ready to upgrade.
>
> So, I recommend that Spark 2.0 drop Scala 2.10 support from the beginning.
>
> dean
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Lightbend <http://lightbend.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Tue, Apr 5, 2016 at 8:54 PM, Kostas Sakellis <ko...@cloudera.com>
> wrote:
>
>> From both this and the JDK thread, I've noticed (including myself) that
>> people have different notions of compatibility guarantees between major and
>> minor versions.
>> A simple question I have is: What compatibility can we break between
>> minor vs. major releases?
>>
>> It might be worth getting on the same page wrt compatibility guarantees.
>>
>> Just a thought,
>> Kostas
>>
>> On Tue, Apr 5, 2016 at 4:39 PM, Holden Karau <ho...@pigscanfly.ca>
>> wrote:
>>
>>> One minor downside to having both 2.10 and 2.11 (and eventually 2.12) is
>>> deprecation warnings in our builds that we can't fix without introducing a
>>> wrapper/ scala version specific code. This isn't a big deal, and if we drop
>>> 2.10 in the 3-6 month time frame talked about we can cleanup those warnings
>>> once we get there.
>>>
>>> On Fri, Apr 1, 2016 at 10:00 PM, Raymond Honderdors <
>>> Raymond.Honderdors@sizmek.com> wrote:
>>>
>>>> What about a seperate branch for scala 2.10?
>>>>
>>>>
>>>>
>>>> Sent from my Samsung Galaxy smartphone.
>>>>
>>>>
>>>> -------- Original message --------
>>>> From: Koert Kuipers <ko...@tresata.com>
>>>> Date: 4/2/2016 02:10 (GMT+02:00)
>>>> To: Michael Armbrust <mi...@databricks.com>
>>>> Cc: Matei Zaharia <ma...@gmail.com>, Mark Hamstra <
>>>> mark@clearstorydata.com>, Cody Koeninger <co...@koeninger.org>, Sean
>>>> Owen <so...@cloudera.com>, dev@spark.apache.org
>>>> Subject: Re: Discuss: commit to Scala 2.10 support for Spark 2.x
>>>> lifecycle
>>>>
>>>> as long as we don't lock ourselves into supporting scala 2.10 for the
>>>> entire spark 2 lifespan it sounds reasonable to me
>>>>
>>>> On Wed, Mar 30, 2016 at 3:25 PM, Michael Armbrust <
>>>> michael@databricks.com> wrote:
>>>>
>>>>> +1 to Matei's reasoning.
>>>>>
>>>>> On Wed, Mar 30, 2016 at 9:21 AM, Matei Zaharia <
>>>>> matei.zaharia@gmail.com> wrote:
>>>>>
>>>>>> I agree that putting it in 2.0 doesn't mean keeping Scala 2.10 for
>>>>>> the entire 2.x line. My vote is to keep Scala 2.10 in Spark 2.0, because
>>>>>> it's the default version we built with in 1.x. We want to make the
>>>>>> transition from 1.x to 2.0 as easy as possible. In 2.0, we'll have the
>>>>>> default downloads be for Scala 2.11, so people will more easily move, but
>>>>>> we shouldn't create obstacles that lead to fragmenting the community and
>>>>>> slowing down Spark 2.0's adoption. I've seen companies that stayed on an
>>>>>> old Scala version for multiple years because switching it, or mixing
>>>>>> versions, would affect the company's entire codebase.
>>>>>>
>>>>>> Matei
>>>>>>
>>>>>> On Mar 30, 2016, at 12:08 PM, Koert Kuipers <ko...@tresata.com>
>>>>>> wrote:
>>>>>>
>>>>>> oh wow, had no idea it got ripped out
>>>>>>
>>>>>> On Wed, Mar 30, 2016 at 11:50 AM, Mark Hamstra <
>>>>>> mark@clearstorydata.com> wrote:
>>>>>>
>>>>>>> No, with 2.0 Spark really doesn't use Akka:
>>>>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L744
>>>>>>>
>>>>>>> On Wed, Mar 30, 2016 at 9:10 AM, Koert Kuipers <ko...@tresata.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Spark still runs on akka. So if you want the benefits of the latest
>>>>>>>> akka (not saying we do, was just an example) then you need to drop scala
>>>>>>>> 2.10
>>>>>>>> On Mar 30, 2016 10:44 AM, "Cody Koeninger" <co...@koeninger.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I agree with Mark in that I don't see how supporting scala 2.10 for
>>>>>>>>> spark 2.0 implies supporting it for all of spark 2.x
>>>>>>>>>
>>>>>>>>> Regarding Koert's comment on akka, I thought all akka dependencies
>>>>>>>>> have been removed from spark after SPARK-7997 and the recent
>>>>>>>>> removal
>>>>>>>>> of external/akka
>>>>>>>>>
>>>>>>>>> On Wed, Mar 30, 2016 at 9:36 AM, Mark Hamstra <
>>>>>>>>> mark@clearstorydata.com> wrote:
>>>>>>>>> > Dropping Scala 2.10 support has to happen at some point, so I'm
>>>>>>>>> not
>>>>>>>>> > fundamentally opposed to the idea; but I've got questions about
>>>>>>>>> how we go
>>>>>>>>> > about making the change and what degree of negative consequences
>>>>>>>>> we are
>>>>>>>>> > willing to accept. Until now, we have been saying that 2.10
>>>>>>>>> support will be
>>>>>>>>> > continued in Spark 2.0.0. Switching to 2.11 will be non-trivial
>>>>>>>>> for some
>>>>>>>>> > Spark users, so abruptly dropping 2.10 support is very likely to
>>>>>>>>> delay
>>>>>>>>> > migration to Spark 2.0 for those users.
>>>>>>>>> >
>>>>>>>>> > What about continuing 2.10 support in 2.0.x, but repeatedly
>>>>>>>>> making an
>>>>>>>>> > obvious announcement in multiple places that such support is
>>>>>>>>> deprecated,
>>>>>>>>> > that we are not committed to maintaining it throughout 2.x, and
>>>>>>>>> that it is,
>>>>>>>>> > in fact, scheduled to be removed in 2.1.0?
>>>>>>>>> >
>>>>>>>>> > On Wed, Mar 30, 2016 at 7:45 AM, Sean Owen <so...@cloudera.com>
>>>>>>>>> wrote:
>>>>>>>>> >>
>>>>>>>>> >> (This should fork as its own thread, though it began during
>>>>>>>>> discussion
>>>>>>>>> >> of whether to continue Java 7 support in Spark 2.x.)
>>>>>>>>> >>
>>>>>>>>> >> Simply: would like to more clearly take the temperature of all
>>>>>>>>> >> interested parties about whether to support Scala 2.10 in the
>>>>>>>>> Spark
>>>>>>>>> >> 2.x lifecycle. Some of the arguments appear to be:
>>>>>>>>> >>
>>>>>>>>> >> Pro
>>>>>>>>> >> - Some third party dependencies do not support Scala 2.11+ yet
>>>>>>>>> and so
>>>>>>>>> >> would not be usable in a Spark app
>>>>>>>>> >>
>>>>>>>>> >> Con
>>>>>>>>> >> - Lower maintenance overhead -- no separate 2.10 build,
>>>>>>>>> >> cross-building, tests to check, esp considering support of 2.12
>>>>>>>>> will
>>>>>>>>> >> be needed
>>>>>>>>> >> - Can use 2.11+ features freely
>>>>>>>>> >> - 2.10 was EOL in late 2014 and Spark 2.x lifecycle is years to
>>>>>>>>> come
>>>>>>>>> >>
>>>>>>>>> >> I would like to not support 2.10 for Spark 2.x, myself.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>> >> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Cell : 425-233-8271
>>> Twitter: https://twitter.com/holdenkarau
>>>
>>
>>
>
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Dean Wampler <de...@gmail.com>.
A few other reasons to drop 2.10 support sooner rather than later.
- We at Lightbend are evaluating some fundamental changes to the REPL to
make it work better for large heaps, especially for Spark. There are other
recent and planned enhancements. This work will be benefit notebook users,
too. However, we won't back port these improvements to 2.10.
- Scala 2.12 is coming out midyear. It will require Java 8, which means
it will produce dramatically smaller code (by exploiting lambdas instead of
custom class generation for functions) and it will offer some performance
improvements. Hopefully Spark will will support it as an optional Scala
version relatively quickly after availability, which means it would be nice
to avoid supporting 3 versions of Scala.
Using Scala 2.10 at this point is like using Java 1.6, seriously out of
date. If you're using libraries that still require 2.10, are you sure that
library is being properly maintained? Or is it a legacy dependency that
should be eliminated before it becomes a liability? Even if you can't
upgrade Scala versions in the next few months, you can certainly continue
using Spark 1.X until you're ready to upgrade.
So, I recommend that Spark 2.0 drop Scala 2.10 support from the beginning.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Lightbend <http://lightbend.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Tue, Apr 5, 2016 at 8:54 PM, Kostas Sakellis <ko...@cloudera.com> wrote:
> From both this and the JDK thread, I've noticed (including myself) that
> people have different notions of compatibility guarantees between major and
> minor versions.
> A simple question I have is: What compatibility can we break between minor
> vs. major releases?
>
> It might be worth getting on the same page wrt compatibility guarantees.
>
> Just a thought,
> Kostas
>
> On Tue, Apr 5, 2016 at 4:39 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
>
>> One minor downside to having both 2.10 and 2.11 (and eventually 2.12) is
>> deprecation warnings in our builds that we can't fix without introducing a
>> wrapper/ scala version specific code. This isn't a big deal, and if we drop
>> 2.10 in the 3-6 month time frame talked about we can cleanup those warnings
>> once we get there.
>>
>> On Fri, Apr 1, 2016 at 10:00 PM, Raymond Honderdors <
>> Raymond.Honderdors@sizmek.com> wrote:
>>
>>> What about a seperate branch for scala 2.10?
>>>
>>>
>>>
>>> Sent from my Samsung Galaxy smartphone.
>>>
>>>
>>> -------- Original message --------
>>> From: Koert Kuipers <ko...@tresata.com>
>>> Date: 4/2/2016 02:10 (GMT+02:00)
>>> To: Michael Armbrust <mi...@databricks.com>
>>> Cc: Matei Zaharia <ma...@gmail.com>, Mark Hamstra <
>>> mark@clearstorydata.com>, Cody Koeninger <co...@koeninger.org>, Sean
>>> Owen <so...@cloudera.com>, dev@spark.apache.org
>>> Subject: Re: Discuss: commit to Scala 2.10 support for Spark 2.x
>>> lifecycle
>>>
>>> as long as we don't lock ourselves into supporting scala 2.10 for the
>>> entire spark 2 lifespan it sounds reasonable to me
>>>
>>> On Wed, Mar 30, 2016 at 3:25 PM, Michael Armbrust <
>>> michael@databricks.com> wrote:
>>>
>>>> +1 to Matei's reasoning.
>>>>
>>>> On Wed, Mar 30, 2016 at 9:21 AM, Matei Zaharia <matei.zaharia@gmail.com
>>>> > wrote:
>>>>
>>>>> I agree that putting it in 2.0 doesn't mean keeping Scala 2.10 for the
>>>>> entire 2.x line. My vote is to keep Scala 2.10 in Spark 2.0, because it's
>>>>> the default version we built with in 1.x. We want to make the transition
>>>>> from 1.x to 2.0 as easy as possible. In 2.0, we'll have the default
>>>>> downloads be for Scala 2.11, so people will more easily move, but we
>>>>> shouldn't create obstacles that lead to fragmenting the community and
>>>>> slowing down Spark 2.0's adoption. I've seen companies that stayed on an
>>>>> old Scala version for multiple years because switching it, or mixing
>>>>> versions, would affect the company's entire codebase.
>>>>>
>>>>> Matei
>>>>>
>>>>> On Mar 30, 2016, at 12:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>>>>
>>>>> oh wow, had no idea it got ripped out
>>>>>
>>>>> On Wed, Mar 30, 2016 at 11:50 AM, Mark Hamstra <
>>>>> mark@clearstorydata.com> wrote:
>>>>>
>>>>>> No, with 2.0 Spark really doesn't use Akka:
>>>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L744
>>>>>>
>>>>>> On Wed, Mar 30, 2016 at 9:10 AM, Koert Kuipers <ko...@tresata.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Spark still runs on akka. So if you want the benefits of the latest
>>>>>>> akka (not saying we do, was just an example) then you need to drop scala
>>>>>>> 2.10
>>>>>>> On Mar 30, 2016 10:44 AM, "Cody Koeninger" <co...@koeninger.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I agree with Mark in that I don't see how supporting scala 2.10 for
>>>>>>>> spark 2.0 implies supporting it for all of spark 2.x
>>>>>>>>
>>>>>>>> Regarding Koert's comment on akka, I thought all akka dependencies
>>>>>>>> have been removed from spark after SPARK-7997 and the recent removal
>>>>>>>> of external/akka
>>>>>>>>
>>>>>>>> On Wed, Mar 30, 2016 at 9:36 AM, Mark Hamstra <
>>>>>>>> mark@clearstorydata.com> wrote:
>>>>>>>> > Dropping Scala 2.10 support has to happen at some point, so I'm
>>>>>>>> not
>>>>>>>> > fundamentally opposed to the idea; but I've got questions about
>>>>>>>> how we go
>>>>>>>> > about making the change and what degree of negative consequences
>>>>>>>> we are
>>>>>>>> > willing to accept. Until now, we have been saying that 2.10
>>>>>>>> support will be
>>>>>>>> > continued in Spark 2.0.0. Switching to 2.11 will be non-trivial
>>>>>>>> for some
>>>>>>>> > Spark users, so abruptly dropping 2.10 support is very likely to
>>>>>>>> delay
>>>>>>>> > migration to Spark 2.0 for those users.
>>>>>>>> >
>>>>>>>> > What about continuing 2.10 support in 2.0.x, but repeatedly
>>>>>>>> making an
>>>>>>>> > obvious announcement in multiple places that such support is
>>>>>>>> deprecated,
>>>>>>>> > that we are not committed to maintaining it throughout 2.x, and
>>>>>>>> that it is,
>>>>>>>> > in fact, scheduled to be removed in 2.1.0?
>>>>>>>> >
>>>>>>>> > On Wed, Mar 30, 2016 at 7:45 AM, Sean Owen <so...@cloudera.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> (This should fork as its own thread, though it began during
>>>>>>>> discussion
>>>>>>>> >> of whether to continue Java 7 support in Spark 2.x.)
>>>>>>>> >>
>>>>>>>> >> Simply: would like to more clearly take the temperature of all
>>>>>>>> >> interested parties about whether to support Scala 2.10 in the
>>>>>>>> Spark
>>>>>>>> >> 2.x lifecycle. Some of the arguments appear to be:
>>>>>>>> >>
>>>>>>>> >> Pro
>>>>>>>> >> - Some third party dependencies do not support Scala 2.11+ yet
>>>>>>>> and so
>>>>>>>> >> would not be usable in a Spark app
>>>>>>>> >>
>>>>>>>> >> Con
>>>>>>>> >> - Lower maintenance overhead -- no separate 2.10 build,
>>>>>>>> >> cross-building, tests to check, esp considering support of 2.12
>>>>>>>> will
>>>>>>>> >> be needed
>>>>>>>> >> - Can use 2.11+ features freely
>>>>>>>> >> - 2.10 was EOL in late 2014 and Spark 2.x lifecycle is years to
>>>>>>>> come
>>>>>>>> >>
>>>>>>>> >> I would like to not support 2.10 for Spark 2.x, myself.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> >> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>> >>
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>>
>
>
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Mark Hamstra <ma...@clearstorydata.com>.
I agree with your general logic and understanding of semver. That is why
if we are going to violate the strictures of semver, I'd only be happy
doing so if support for Java 7 and/or Scala 2.10 were clearly understood to
be deprecated already in the 2.0.0 release -- i.e. from the outset not to
be understood to be part of the semver guarantees implied in 2.x.
On Wed, Apr 6, 2016 at 11:04 AM, Sean Owen <so...@cloudera.com> wrote:
> Answering for myself: I assume everyone is following
> http://semver.org/ semantic versioning. If not, would be good to hear
> an alternative theory.
>
> For semver, strictly speaking, minor releases should be
> backwards-compatible for callers. Are things like stopping support for
> Java 8 or Scala 2.10 backwards-incompatible? In the end, yes,
> non-trivially, on both counts. This is why it seems like these changes
> must go with 2.0 or wait until 3.0.
>
> Rules are made to be broken and few software projects do this right. I
> hear the legitimate concern that getting the latest features and fixes
> into users hands is really important, and it is.
>
> Really, that argues that it's important to keep maintaining the 1.x
> branch with features and fixes. However, it seems obvious there will
> never be a 1.7, and probably not 1.6.2, for lack of bandwidth. And
> indeed it's way too much work given the scope of this project compared
> to hands to do the work.
>
> But this is what's forcing conflation of backwards-compatibility
> concerns onto a version boundary that doesn't inherently have one.
> It's a reason I personally root for breaking as many promises and
> encumbrances that make sense to break all at once at this boundary.
>
> Anyway, hope that explains my general logic.
>
>
> On Wed, Apr 6, 2016 at 2:54 AM, Kostas Sakellis <ko...@cloudera.com>
> wrote:
> > From both this and the JDK thread, I've noticed (including myself) that
> > people have different notions of compatibility guarantees between major
> and
> > minor versions.
> > A simple question I have is: What compatibility can we break between
> minor
> > vs. major releases?
> >
> > It might be worth getting on the same page wrt compatibility guarantees.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Mridul Muralidharan <mr...@gmail.com>.
In general, I agree - it is preferable to break backward compatibility
(where unavoidable) only at major versions.
Unfortunately, this usually is planned better - with earlier versions
announcing intent of the change - deprecation across multiple
releases, defaults changed, etc.
>From the thread, since scala 2.10 was default till last release, it
has been mentioned as a large disruption to drop support in 2.0.
The same applies for jdk 7 too unfortunately - since backward
compatibility is stronger guarantee in jvm compared to scala, it might
work better there ?
Regards,
Mridul
On Wed, Apr 6, 2016 at 11:04 AM, Sean Owen <so...@cloudera.com> wrote:
> Answering for myself: I assume everyone is following
> http://semver.org/ semantic versioning. If not, would be good to hear
> an alternative theory.
>
> For semver, strictly speaking, minor releases should be
> backwards-compatible for callers. Are things like stopping support for
> Java 8 or Scala 2.10 backwards-incompatible? In the end, yes,
> non-trivially, on both counts. This is why it seems like these changes
> must go with 2.0 or wait until 3.0.
>
> Rules are made to be broken and few software projects do this right. I
> hear the legitimate concern that getting the latest features and fixes
> into users hands is really important, and it is.
>
> Really, that argues that it's important to keep maintaining the 1.x
> branch with features and fixes. However, it seems obvious there will
> never be a 1.7, and probably not 1.6.2, for lack of bandwidth. And
> indeed it's way too much work given the scope of this project compared
> to hands to do the work.
>
> But this is what's forcing conflation of backwards-compatibility
> concerns onto a version boundary that doesn't inherently have one.
> It's a reason I personally root for breaking as many promises and
> encumbrances that make sense to break all at once at this boundary.
>
> Anyway, hope that explains my general logic.
>
>
> On Wed, Apr 6, 2016 at 2:54 AM, Kostas Sakellis <ko...@cloudera.com> wrote:
>> From both this and the JDK thread, I've noticed (including myself) that
>> people have different notions of compatibility guarantees between major and
>> minor versions.
>> A simple question I have is: What compatibility can we break between minor
>> vs. major releases?
>>
>> It might be worth getting on the same page wrt compatibility guarantees.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Sean Owen <so...@cloudera.com>.
Answering for myself: I assume everyone is following
http://semver.org/ semantic versioning. If not, would be good to hear
an alternative theory.
For semver, strictly speaking, minor releases should be
backwards-compatible for callers. Are things like stopping support for
Java 8 or Scala 2.10 backwards-incompatible? In the end, yes,
non-trivially, on both counts. This is why it seems like these changes
must go with 2.0 or wait until 3.0.
Rules are made to be broken and few software projects do this right. I
hear the legitimate concern that getting the latest features and fixes
into users hands is really important, and it is.
Really, that argues that it's important to keep maintaining the 1.x
branch with features and fixes. However, it seems obvious there will
never be a 1.7, and probably not 1.6.2, for lack of bandwidth. And
indeed it's way too much work given the scope of this project compared
to hands to do the work.
But this is what's forcing conflation of backwards-compatibility
concerns onto a version boundary that doesn't inherently have one.
It's a reason I personally root for breaking as many promises and
encumbrances that make sense to break all at once at this boundary.
Anyway, hope that explains my general logic.
On Wed, Apr 6, 2016 at 2:54 AM, Kostas Sakellis <ko...@cloudera.com> wrote:
> From both this and the JDK thread, I've noticed (including myself) that
> people have different notions of compatibility guarantees between major and
> minor versions.
> A simple question I have is: What compatibility can we break between minor
> vs. major releases?
>
> It might be worth getting on the same page wrt compatibility guarantees.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Kostas Sakellis <ko...@cloudera.com>.
>From both this and the JDK thread, I've noticed (including myself) that
people have different notions of compatibility guarantees between major and
minor versions.
A simple question I have is: What compatibility can we break between minor
vs. major releases?
It might be worth getting on the same page wrt compatibility guarantees.
Just a thought,
Kostas
On Tue, Apr 5, 2016 at 4:39 PM, Holden Karau <ho...@pigscanfly.ca> wrote:
> One minor downside to having both 2.10 and 2.11 (and eventually 2.12) is
> deprecation warnings in our builds that we can't fix without introducing a
> wrapper/ scala version specific code. This isn't a big deal, and if we drop
> 2.10 in the 3-6 month time frame talked about we can cleanup those warnings
> once we get there.
>
> On Fri, Apr 1, 2016 at 10:00 PM, Raymond Honderdors <
> Raymond.Honderdors@sizmek.com> wrote:
>
>> What about a seperate branch for scala 2.10?
>>
>>
>>
>> Sent from my Samsung Galaxy smartphone.
>>
>>
>> -------- Original message --------
>> From: Koert Kuipers <ko...@tresata.com>
>> Date: 4/2/2016 02:10 (GMT+02:00)
>> To: Michael Armbrust <mi...@databricks.com>
>> Cc: Matei Zaharia <ma...@gmail.com>, Mark Hamstra <
>> mark@clearstorydata.com>, Cody Koeninger <co...@koeninger.org>, Sean Owen
>> <so...@cloudera.com>, dev@spark.apache.org
>> Subject: Re: Discuss: commit to Scala 2.10 support for Spark 2.x
>> lifecycle
>>
>> as long as we don't lock ourselves into supporting scala 2.10 for the
>> entire spark 2 lifespan it sounds reasonable to me
>>
>> On Wed, Mar 30, 2016 at 3:25 PM, Michael Armbrust <michael@databricks.com
>> > wrote:
>>
>>> +1 to Matei's reasoning.
>>>
>>> On Wed, Mar 30, 2016 at 9:21 AM, Matei Zaharia <ma...@gmail.com>
>>> wrote:
>>>
>>>> I agree that putting it in 2.0 doesn't mean keeping Scala 2.10 for the
>>>> entire 2.x line. My vote is to keep Scala 2.10 in Spark 2.0, because it's
>>>> the default version we built with in 1.x. We want to make the transition
>>>> from 1.x to 2.0 as easy as possible. In 2.0, we'll have the default
>>>> downloads be for Scala 2.11, so people will more easily move, but we
>>>> shouldn't create obstacles that lead to fragmenting the community and
>>>> slowing down Spark 2.0's adoption. I've seen companies that stayed on an
>>>> old Scala version for multiple years because switching it, or mixing
>>>> versions, would affect the company's entire codebase.
>>>>
>>>> Matei
>>>>
>>>> On Mar 30, 2016, at 12:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>>>
>>>> oh wow, had no idea it got ripped out
>>>>
>>>> On Wed, Mar 30, 2016 at 11:50 AM, Mark Hamstra <mark@clearstorydata.com
>>>> > wrote:
>>>>
>>>>> No, with 2.0 Spark really doesn't use Akka:
>>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L744
>>>>>
>>>>> On Wed, Mar 30, 2016 at 9:10 AM, Koert Kuipers <ko...@tresata.com>
>>>>> wrote:
>>>>>
>>>>>> Spark still runs on akka. So if you want the benefits of the latest
>>>>>> akka (not saying we do, was just an example) then you need to drop scala
>>>>>> 2.10
>>>>>> On Mar 30, 2016 10:44 AM, "Cody Koeninger" <co...@koeninger.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I agree with Mark in that I don't see how supporting scala 2.10 for
>>>>>>> spark 2.0 implies supporting it for all of spark 2.x
>>>>>>>
>>>>>>> Regarding Koert's comment on akka, I thought all akka dependencies
>>>>>>> have been removed from spark after SPARK-7997 and the recent removal
>>>>>>> of external/akka
>>>>>>>
>>>>>>> On Wed, Mar 30, 2016 at 9:36 AM, Mark Hamstra <
>>>>>>> mark@clearstorydata.com> wrote:
>>>>>>> > Dropping Scala 2.10 support has to happen at some point, so I'm not
>>>>>>> > fundamentally opposed to the idea; but I've got questions about
>>>>>>> how we go
>>>>>>> > about making the change and what degree of negative consequences
>>>>>>> we are
>>>>>>> > willing to accept. Until now, we have been saying that 2.10
>>>>>>> support will be
>>>>>>> > continued in Spark 2.0.0. Switching to 2.11 will be non-trivial
>>>>>>> for some
>>>>>>> > Spark users, so abruptly dropping 2.10 support is very likely to
>>>>>>> delay
>>>>>>> > migration to Spark 2.0 for those users.
>>>>>>> >
>>>>>>> > What about continuing 2.10 support in 2.0.x, but repeatedly making
>>>>>>> an
>>>>>>> > obvious announcement in multiple places that such support is
>>>>>>> deprecated,
>>>>>>> > that we are not committed to maintaining it throughout 2.x, and
>>>>>>> that it is,
>>>>>>> > in fact, scheduled to be removed in 2.1.0?
>>>>>>> >
>>>>>>> > On Wed, Mar 30, 2016 at 7:45 AM, Sean Owen <so...@cloudera.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> (This should fork as its own thread, though it began during
>>>>>>> discussion
>>>>>>> >> of whether to continue Java 7 support in Spark 2.x.)
>>>>>>> >>
>>>>>>> >> Simply: would like to more clearly take the temperature of all
>>>>>>> >> interested parties about whether to support Scala 2.10 in the
>>>>>>> Spark
>>>>>>> >> 2.x lifecycle. Some of the arguments appear to be:
>>>>>>> >>
>>>>>>> >> Pro
>>>>>>> >> - Some third party dependencies do not support Scala 2.11+ yet
>>>>>>> and so
>>>>>>> >> would not be usable in a Spark app
>>>>>>> >>
>>>>>>> >> Con
>>>>>>> >> - Lower maintenance overhead -- no separate 2.10 build,
>>>>>>> >> cross-building, tests to check, esp considering support of 2.12
>>>>>>> will
>>>>>>> >> be needed
>>>>>>> >> - Can use 2.11+ features freely
>>>>>>> >> - 2.10 was EOL in late 2014 and Spark 2.x lifecycle is years to
>>>>>>> come
>>>>>>> >>
>>>>>>> >> I would like to not support 2.10 for Spark 2.x, myself.
>>>>>>> >>
>>>>>>> >>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> >> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Cell : 425-233-8271
> Twitter: https://twitter.com/holdenkarau
>
Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Holden Karau <ho...@pigscanfly.ca>.
One minor downside to having both 2.10 and 2.11 (and eventually 2.12) is
deprecation warnings in our builds that we can't fix without introducing a
wrapper/ scala version specific code. This isn't a big deal, and if we drop
2.10 in the 3-6 month time frame talked about we can cleanup those warnings
once we get there.
On Fri, Apr 1, 2016 at 10:00 PM, Raymond Honderdors <
Raymond.Honderdors@sizmek.com> wrote:
> What about a seperate branch for scala 2.10?
>
>
>
> Sent from my Samsung Galaxy smartphone.
>
>
> -------- Original message --------
> From: Koert Kuipers <ko...@tresata.com>
> Date: 4/2/2016 02:10 (GMT+02:00)
> To: Michael Armbrust <mi...@databricks.com>
> Cc: Matei Zaharia <ma...@gmail.com>, Mark Hamstra <
> mark@clearstorydata.com>, Cody Koeninger <co...@koeninger.org>, Sean Owen <
> sowen@cloudera.com>, dev@spark.apache.org
> Subject: Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
>
> as long as we don't lock ourselves into supporting scala 2.10 for the
> entire spark 2 lifespan it sounds reasonable to me
>
> On Wed, Mar 30, 2016 at 3:25 PM, Michael Armbrust <mi...@databricks.com>
> wrote:
>
>> +1 to Matei's reasoning.
>>
>> On Wed, Mar 30, 2016 at 9:21 AM, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>
>>> I agree that putting it in 2.0 doesn't mean keeping Scala 2.10 for the
>>> entire 2.x line. My vote is to keep Scala 2.10 in Spark 2.0, because it's
>>> the default version we built with in 1.x. We want to make the transition
>>> from 1.x to 2.0 as easy as possible. In 2.0, we'll have the default
>>> downloads be for Scala 2.11, so people will more easily move, but we
>>> shouldn't create obstacles that lead to fragmenting the community and
>>> slowing down Spark 2.0's adoption. I've seen companies that stayed on an
>>> old Scala version for multiple years because switching it, or mixing
>>> versions, would affect the company's entire codebase.
>>>
>>> Matei
>>>
>>> On Mar 30, 2016, at 12:08 PM, Koert Kuipers <ko...@tresata.com> wrote:
>>>
>>> oh wow, had no idea it got ripped out
>>>
>>> On Wed, Mar 30, 2016 at 11:50 AM, Mark Hamstra <ma...@clearstorydata.com>
>>> wrote:
>>>
>>>> No, with 2.0 Spark really doesn't use Akka:
>>>> https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L744
>>>>
>>>> On Wed, Mar 30, 2016 at 9:10 AM, Koert Kuipers <ko...@tresata.com>
>>>> wrote:
>>>>
>>>>> Spark still runs on akka. So if you want the benefits of the latest
>>>>> akka (not saying we do, was just an example) then you need to drop scala
>>>>> 2.10
>>>>> On Mar 30, 2016 10:44 AM, "Cody Koeninger" <co...@koeninger.org> wrote:
>>>>>
>>>>>> I agree with Mark in that I don't see how supporting scala 2.10 for
>>>>>> spark 2.0 implies supporting it for all of spark 2.x
>>>>>>
>>>>>> Regarding Koert's comment on akka, I thought all akka dependencies
>>>>>> have been removed from spark after SPARK-7997 and the recent removal
>>>>>> of external/akka
>>>>>>
>>>>>> On Wed, Mar 30, 2016 at 9:36 AM, Mark Hamstra <
>>>>>> mark@clearstorydata.com> wrote:
>>>>>> > Dropping Scala 2.10 support has to happen at some point, so I'm not
>>>>>> > fundamentally opposed to the idea; but I've got questions about how
>>>>>> we go
>>>>>> > about making the change and what degree of negative consequences we
>>>>>> are
>>>>>> > willing to accept. Until now, we have been saying that 2.10
>>>>>> support will be
>>>>>> > continued in Spark 2.0.0. Switching to 2.11 will be non-trivial
>>>>>> for some
>>>>>> > Spark users, so abruptly dropping 2.10 support is very likely to
>>>>>> delay
>>>>>> > migration to Spark 2.0 for those users.
>>>>>> >
>>>>>> > What about continuing 2.10 support in 2.0.x, but repeatedly making
>>>>>> an
>>>>>> > obvious announcement in multiple places that such support is
>>>>>> deprecated,
>>>>>> > that we are not committed to maintaining it throughout 2.x, and
>>>>>> that it is,
>>>>>> > in fact, scheduled to be removed in 2.1.0?
>>>>>> >
>>>>>> > On Wed, Mar 30, 2016 at 7:45 AM, Sean Owen <so...@cloudera.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> (This should fork as its own thread, though it began during
>>>>>> discussion
>>>>>> >> of whether to continue Java 7 support in Spark 2.x.)
>>>>>> >>
>>>>>> >> Simply: would like to more clearly take the temperature of all
>>>>>> >> interested parties about whether to support Scala 2.10 in the Spark
>>>>>> >> 2.x lifecycle. Some of the arguments appear to be:
>>>>>> >>
>>>>>> >> Pro
>>>>>> >> - Some third party dependencies do not support Scala 2.11+ yet and
>>>>>> so
>>>>>> >> would not be usable in a Spark app
>>>>>> >>
>>>>>> >> Con
>>>>>> >> - Lower maintenance overhead -- no separate 2.10 build,
>>>>>> >> cross-building, tests to check, esp considering support of 2.12
>>>>>> will
>>>>>> >> be needed
>>>>>> >> - Can use 2.11+ features freely
>>>>>> >> - 2.10 was EOL in late 2014 and Spark 2.x lifecycle is years to
>>>>>> come
>>>>>> >>
>>>>>> >> I would like to not support 2.10 for Spark 2.x, myself.
>>>>>> >>
>>>>>> >>
>>>>>> ---------------------------------------------------------------------
>>>>>> >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>> >> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>> >>
>>>>>> >
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>
>>>>>>
>>>>
>>>
>>>
>>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
RE: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
Posted by Raymond Honderdors <Ra...@sizmek.com>.
What about a seperate branch for scala 2.10?
Sent from my Samsung Galaxy smartphone.
-------- Original message --------
From: Koert Kuipers <ko...@tresata.com>
Date: 4/2/2016 02:10 (GMT+02:00)
To: Michael Armbrust <mi...@databricks.com>
Cc: Matei Zaharia <ma...@gmail.com>, Mark Hamstra <ma...@clearstorydata.com>, Cody Koeninger <co...@koeninger.org>, Sean Owen <so...@cloudera.com>, dev@spark.apache.org
Subject: Re: Discuss: commit to Scala 2.10 support for Spark 2.x lifecycle
as long as we don't lock ourselves into supporting scala 2.10 for the entire spark 2 lifespan it sounds reasonable to me
On Wed, Mar 30, 2016 at 3:25 PM, Michael Armbrust <mi...@databricks.com>> wrote:
+1 to Matei's reasoning.
On Wed, Mar 30, 2016 at 9:21 AM, Matei Zaharia <ma...@gmail.com>> wrote:
I agree that putting it in 2.0 doesn't mean keeping Scala 2.10 for the entire 2.x line. My vote is to keep Scala 2.10 in Spark 2.0, because it's the default version we built with in 1.x. We want to make the transition from 1.x to 2.0 as easy as possible. In 2.0, we'll have the default downloads be for Scala 2.11, so people will more easily move, but we shouldn't create obstacles that lead to fragmenting the community and slowing down Spark 2.0's adoption. I've seen companies that stayed on an old Scala version for multiple years because switching it, or mixing versions, would affect the company's entire codebase.
Matei
On Mar 30, 2016, at 12:08 PM, Koert Kuipers <ko...@tresata.com>> wrote:
oh wow, had no idea it got ripped out
On Wed, Mar 30, 2016 at 11:50 AM, Mark Hamstra <ma...@clearstorydata.com>> wrote:
No, with 2.0 Spark really doesn't use Akka: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkConf.scala#L744
On Wed, Mar 30, 2016 at 9:10 AM, Koert Kuipers <ko...@tresata.com>> wrote:
Spark still runs on akka. So if you want the benefits of the latest akka (not saying we do, was just an example) then you need to drop scala 2.10
On Mar 30, 2016 10:44 AM, "Cody Koeninger" <co...@koeninger.org>> wrote:
I agree with Mark in that I don't see how supporting scala 2.10 for
spark 2.0 implies supporting it for all of spark 2.x
Regarding Koert's comment on akka, I thought all akka dependencies
have been removed from spark after SPARK-7997 and the recent removal
of external/akka
On Wed, Mar 30, 2016 at 9:36 AM, Mark Hamstra <ma...@clearstorydata.com>> wrote:
> Dropping Scala 2.10 support has to happen at some point, so I'm not
> fundamentally opposed to the idea; but I've got questions about how we go
> about making the change and what degree of negative consequences we are
> willing to accept. Until now, we have been saying that 2.10 support will be
> continued in Spark 2.0.0. Switching to 2.11 will be non-trivial for some
> Spark users, so abruptly dropping 2.10 support is very likely to delay
> migration to Spark 2.0 for those users.
>
> What about continuing 2.10 support in 2.0.x, but repeatedly making an
> obvious announcement in multiple places that such support is deprecated,
> that we are not committed to maintaining it throughout 2.x, and that it is,
> in fact, scheduled to be removed in 2.1.0?
>
> On Wed, Mar 30, 2016 at 7:45 AM, Sean Owen <so...@cloudera.com>> wrote:
>>
>> (This should fork as its own thread, though it began during discussion
>> of whether to continue Java 7 support in Spark 2.x.)
>>
>> Simply: would like to more clearly take the temperature of all
>> interested parties about whether to support Scala 2.10 in the Spark
>> 2.x lifecycle. Some of the arguments appear to be:
>>
>> Pro
>> - Some third party dependencies do not support Scala 2.11+ yet and so
>> would not be usable in a Spark app
>>
>> Con
>> - Lower maintenance overhead -- no separate 2.10 build,
>> cross-building, tests to check, esp considering support of 2.12 will
>> be needed
>> - Can use 2.11+ features freely
>> - 2.10 was EOL in late 2014 and Spark 2.x lifecycle is years to come
>>
>> I would like to not support 2.10 for Spark 2.x, myself.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
>> For additional commands, e-mail: dev-help@spark.apache.org<ma...@spark.apache.org>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
For additional commands, e-mail: dev-help@spark.apache.org<ma...@spark.apache.org>