You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@crunch.apache.org by Josh Wills <jw...@cloudera.com> on 2014/04/25 22:45:54 UTC
spark/scala version changes for 0.10.0/0.8.3
Hey all,
I've been working on doing a new release candidate, and I think that we're
going to need to upgrade to Spark 0.9.1 in order for Crunch-on-Spark to
work in Spark standalone/Spark-on-YARN on top of Hadoop, as opposed to the
local mode that works now. There are a couple of implications to this, but
the big one is that Spark 0.9.1 is only developed against Scala 2.10, which
would mean that we would need to switch over to 2.10 for at least
crunch-spark, and it would seem much easier to me for us to upgrade Scrunch
to 2.10 as well.
Couple of options here off the top of my head:
1) Do the upgrade to Scala 2.10/Spark 0.9.1 in 0.10.0, but not 0.8.3.
2) Do the upgrade in both 0.8.3 and 0.10.0.
3) Only upgrade Spark to 2.10 (in either 0.8.3 or 0.10.0), but still have
Scrunch build against 2.9.3.
I'm in favor of #2 myself, but am open to suggestion here, and would
especially like to know if anyone feels strongly about staying on 2.9.3 for
any reason.
Thanks!
Josh
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: spark/scala version changes for 0.10.0/0.8.3
Posted by Josh Wills <jw...@cloudera.com>.
Thanks guys; patch is up here:
https://issues.apache.org/jira/browse/CRUNCH-384
Micah, do you want to take a pass through this one?
J
On Mon, Apr 28, 2014 at 10:30 PM, Som Satpathy <so...@gmail.com>wrote:
> +1 for option 2.
>
> Thanks,
> Som
>
>
> On Fri, Apr 25, 2014 at 1:45 PM, Josh Wills <jw...@cloudera.com> wrote:
>
>> Hey all,
>>
>> I've been working on doing a new release candidate, and I think that
>> we're going to need to upgrade to Spark 0.9.1 in order for Crunch-on-Spark
>> to work in Spark standalone/Spark-on-YARN on top of Hadoop, as opposed to
>> the local mode that works now. There are a couple of implications to this,
>> but the big one is that Spark 0.9.1 is only developed against Scala 2.10,
>> which would mean that we would need to switch over to 2.10 for at least
>> crunch-spark, and it would seem much easier to me for us to upgrade Scrunch
>> to 2.10 as well.
>>
>> Couple of options here off the top of my head:
>> 1) Do the upgrade to Scala 2.10/Spark 0.9.1 in 0.10.0, but not 0.8.3.
>> 2) Do the upgrade in both 0.8.3 and 0.10.0.
>> 3) Only upgrade Spark to 2.10 (in either 0.8.3 or 0.10.0), but still have
>> Scrunch build against 2.9.3.
>>
>> I'm in favor of #2 myself, but am open to suggestion here, and would
>> especially like to know if anyone feels strongly about staying on 2.9.3 for
>> any reason.
>>
>> Thanks!
>> Josh
>>
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>
>
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: spark/scala version changes for 0.10.0/0.8.3
Posted by Josh Wills <jw...@cloudera.com>.
Thanks guys; patch is up here:
https://issues.apache.org/jira/browse/CRUNCH-384
Micah, do you want to take a pass through this one?
J
On Mon, Apr 28, 2014 at 10:30 PM, Som Satpathy <so...@gmail.com>wrote:
> +1 for option 2.
>
> Thanks,
> Som
>
>
> On Fri, Apr 25, 2014 at 1:45 PM, Josh Wills <jw...@cloudera.com> wrote:
>
>> Hey all,
>>
>> I've been working on doing a new release candidate, and I think that
>> we're going to need to upgrade to Spark 0.9.1 in order for Crunch-on-Spark
>> to work in Spark standalone/Spark-on-YARN on top of Hadoop, as opposed to
>> the local mode that works now. There are a couple of implications to this,
>> but the big one is that Spark 0.9.1 is only developed against Scala 2.10,
>> which would mean that we would need to switch over to 2.10 for at least
>> crunch-spark, and it would seem much easier to me for us to upgrade Scrunch
>> to 2.10 as well.
>>
>> Couple of options here off the top of my head:
>> 1) Do the upgrade to Scala 2.10/Spark 0.9.1 in 0.10.0, but not 0.8.3.
>> 2) Do the upgrade in both 0.8.3 and 0.10.0.
>> 3) Only upgrade Spark to 2.10 (in either 0.8.3 or 0.10.0), but still have
>> Scrunch build against 2.9.3.
>>
>> I'm in favor of #2 myself, but am open to suggestion here, and would
>> especially like to know if anyone feels strongly about staying on 2.9.3 for
>> any reason.
>>
>> Thanks!
>> Josh
>>
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>
>
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: spark/scala version changes for 0.10.0/0.8.3
Posted by Som Satpathy <so...@gmail.com>.
+1 for option 2.
Thanks,
Som
On Fri, Apr 25, 2014 at 1:45 PM, Josh Wills <jw...@cloudera.com> wrote:
> Hey all,
>
> I've been working on doing a new release candidate, and I think that we're
> going to need to upgrade to Spark 0.9.1 in order for Crunch-on-Spark to
> work in Spark standalone/Spark-on-YARN on top of Hadoop, as opposed to the
> local mode that works now. There are a couple of implications to this, but
> the big one is that Spark 0.9.1 is only developed against Scala 2.10, which
> would mean that we would need to switch over to 2.10 for at least
> crunch-spark, and it would seem much easier to me for us to upgrade Scrunch
> to 2.10 as well.
>
> Couple of options here off the top of my head:
> 1) Do the upgrade to Scala 2.10/Spark 0.9.1 in 0.10.0, but not 0.8.3.
> 2) Do the upgrade in both 0.8.3 and 0.10.0.
> 3) Only upgrade Spark to 2.10 (in either 0.8.3 or 0.10.0), but still have
> Scrunch build against 2.9.3.
>
> I'm in favor of #2 myself, but am open to suggestion here, and would
> especially like to know if anyone feels strongly about staying on 2.9.3 for
> any reason.
>
> Thanks!
> Josh
>
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
Re: spark/scala version changes for 0.10.0/0.8.3
Posted by Micah Whitacre <mk...@gmail.com>.
+1 to option #2.
On Fri, Apr 25, 2014 at 3:45 PM, Josh Wills <jw...@cloudera.com> wrote:
> Hey all,
>
> I've been working on doing a new release candidate, and I think that we're
> going to need to upgrade to Spark 0.9.1 in order for Crunch-on-Spark to
> work in Spark standalone/Spark-on-YARN on top of Hadoop, as opposed to the
> local mode that works now. There are a couple of implications to this, but
> the big one is that Spark 0.9.1 is only developed against Scala 2.10, which
> would mean that we would need to switch over to 2.10 for at least
> crunch-spark, and it would seem much easier to me for us to upgrade Scrunch
> to 2.10 as well.
>
> Couple of options here off the top of my head:
> 1) Do the upgrade to Scala 2.10/Spark 0.9.1 in 0.10.0, but not 0.8.3.
> 2) Do the upgrade in both 0.8.3 and 0.10.0.
> 3) Only upgrade Spark to 2.10 (in either 0.8.3 or 0.10.0), but still have
> Scrunch build against 2.9.3.
>
> I'm in favor of #2 myself, but am open to suggestion here, and would
> especially like to know if anyone feels strongly about staying on 2.9.3 for
> any reason.
>
> Thanks!
> Josh
>
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>