State of spark on scala 2.10

[Aureliano Buendia <bu...@gmail.com>]

Hi,

I was going to give https://github.com/scala/pickling a try on spark to see
how it would compare with kryo. Unfortunately, it only works with scala
2.10.3.

- Is there a time line for spark to work with scala 2.10?

- Is the 2.10 branch as stable as 2.9?

- What's blocking spark to work with 2.10?

Re: State of spark on scala 2.10

[Prashant Sharma <sc...@gmail.com>]

sbt assembly/assembly

or if you are devloping spark i.e. working on spark code base itself not
rolling a distribution.
sbt assemble-deps

also

there is a convenience script called make-distribution.sh

HTH


On Sun, Jan 5, 2014 at 7:52 PM, Aureliano Buendia <bu...@gmail.com>wrote:

>
>
>
> On Sun, Jan 5, 2014 at 5:42 AM, Patrick Wendell <pw...@gmail.com>wrote:
>
>> I usually just use the existing launch scripts to create a correctly
>> sized cluster then just:
>>
>> 1. Copy spark/conf/* to /tmp
>> 2. rm -rf spark/
>> 3. Checkout and build new spark from github (you'll need to rename
>> incubator-spark to spark)
>>
>
> I called 'sbt assembly' at this point, and it took ages to build the
> examples. What would be the minimal build command, just to build enough
> stuff for production?
>
>
>
>> 4. Copy conf files back
>> 5. /root/spark-ec2/copy-dir --delete /root/spark
>>
>>
>> - Patrick
>>
>> On Sat, Jan 4, 2014 at 9:14 PM, Aureliano Buendia <bu...@gmail.com>
>> wrote:
>> > So I should just lanuch an AMI from one of
>> > https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the
>> > development version on it? Is it just a simple matter of copying the
>> target
>> > to a directory, or does it need a lot of configurations?
>> >
>> >
>> >
>> > On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <pw...@gmail.com>
>> wrote:
>> >>
>> >> I meant you'll need to build your own version of Spark. Typically we
>> >> do this by launching an existing AMI and then just building a new
>> >> version of spark and copying it to the slaves....
>> >>
>> >> - Patrick
>> >>
>> >> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <pw...@gmail.com>
>> >> wrote:
>> >> > You'll have to build your own. Also there are some packaging
>> >> > differences in master (some bin/ scripts moved to sbin/) just to give
>> >> > you a heads up.
>> >> >
>> >> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <
>> buendia360@gmail.com>
>> >> > wrote:
>> >> >> Good to know the next release will be on scala 2.10.
>> >> >>
>> >> >> Meanwhile, does using the master branch mean that I will have to
>> build
>> >> >> my
>> >> >> own AMI when launching an ec2 cluster? Also, is there a nightly
>> binary
>> >> >> build
>> >> >> maven repo available for spark?
>> >> >>
>> >> >>
>> >> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <il...@gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Scala 2.10.3 support was recently merged into master (#259). The
>> >> >>> branch is
>> >> >>> probably not as stable as 0.8.1, but things "should" work.
>> >> >>>
>> >> >>> The 2.10 branch should be deleted, the only issue is there are some
>> >> >>> outstanding PRs against that branch that haven't been moved to
>> master.
>> >> >>>
>> >> >>>
>> >> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia
>> >> >>> <bu...@gmail.com>
>> >> >>> wrote:
>> >> >>>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> I was going to give https://github.com/scala/pickling a try on
>> spark
>> >> >>>> to
>> >> >>>> see how it would compare with kryo. Unfortunately, it only works
>> with
>> >> >>>> scala
>> >> >>>> 2.10.3.
>> >> >>>>
>> >> >>>> - Is there a time line for spark to work with scala 2.10?
>> >> >>>>
>> >> >>>> - Is the 2.10 branch as stable as 2.9?
>> >> >>>>
>> >> >>>> - What's blocking spark to work with 2.10?
>> >> >>>
>> >> >>>
>> >> >>
>> >
>> >
>>
>
>


-- 
Prashant

Re: State of spark on scala 2.10

[Aureliano Buendia <bu...@gmail.com>]

On Sun, Jan 5, 2014 at 5:42 AM, Patrick Wendell <pw...@gmail.com> wrote:

> I usually just use the existing launch scripts to create a correctly
> sized cluster then just:
>
> 1. Copy spark/conf/* to /tmp
> 2. rm -rf spark/
> 3. Checkout and build new spark from github (you'll need to rename
> incubator-spark to spark)
>

I called 'sbt assembly' at this point, and it took ages to build the
examples. What would be the minimal build command, just to build enough
stuff for production?



> 4. Copy conf files back
> 5. /root/spark-ec2/copy-dir --delete /root/spark
>
>
> - Patrick
>
> On Sat, Jan 4, 2014 at 9:14 PM, Aureliano Buendia <bu...@gmail.com>
> wrote:
> > So I should just lanuch an AMI from one of
> > https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the
> > development version on it? Is it just a simple matter of copying the
> target
> > to a directory, or does it need a lot of configurations?
> >
> >
> >
> > On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <pw...@gmail.com>
> wrote:
> >>
> >> I meant you'll need to build your own version of Spark. Typically we
> >> do this by launching an existing AMI and then just building a new
> >> version of spark and copying it to the slaves....
> >>
> >> - Patrick
> >>
> >> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <pw...@gmail.com>
> >> wrote:
> >> > You'll have to build your own. Also there are some packaging
> >> > differences in master (some bin/ scripts moved to sbin/) just to give
> >> > you a heads up.
> >> >
> >> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <
> buendia360@gmail.com>
> >> > wrote:
> >> >> Good to know the next release will be on scala 2.10.
> >> >>
> >> >> Meanwhile, does using the master branch mean that I will have to
> build
> >> >> my
> >> >> own AMI when launching an ec2 cluster? Also, is there a nightly
> binary
> >> >> build
> >> >> maven repo available for spark?
> >> >>
> >> >>
> >> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <il...@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Scala 2.10.3 support was recently merged into master (#259). The
> >> >>> branch is
> >> >>> probably not as stable as 0.8.1, but things "should" work.
> >> >>>
> >> >>> The 2.10 branch should be deleted, the only issue is there are some
> >> >>> outstanding PRs against that branch that haven't been moved to
> master.
> >> >>>
> >> >>>
> >> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia
> >> >>> <bu...@gmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> I was going to give https://github.com/scala/pickling a try on
> spark
> >> >>>> to
> >> >>>> see how it would compare with kryo. Unfortunately, it only works
> with
> >> >>>> scala
> >> >>>> 2.10.3.
> >> >>>>
> >> >>>> - Is there a time line for spark to work with scala 2.10?
> >> >>>>
> >> >>>> - Is the 2.10 branch as stable as 2.9?
> >> >>>>
> >> >>>> - What's blocking spark to work with 2.10?
> >> >>>
> >> >>>
> >> >>
> >
> >
>

Re: State of spark on scala 2.10

[Patrick Wendell <pw...@gmail.com>]

I usually just use the existing launch scripts to create a correctly
sized cluster then just:

1. Copy spark/conf/* to /tmp
2. rm -rf spark/
3. Checkout and build new spark from github (you'll need to rename
incubator-spark to spark)
4. Copy conf files back
5. /root/spark-ec2/copy-dir --delete /root/spark


- Patrick

On Sat, Jan 4, 2014 at 9:14 PM, Aureliano Buendia <bu...@gmail.com> wrote:
> So I should just lanuch an AMI from one of
> https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the
> development version on it? Is it just a simple matter of copying the target
> to a directory, or does it need a lot of configurations?
>
>
>
> On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <pw...@gmail.com> wrote:
>>
>> I meant you'll need to build your own version of Spark. Typically we
>> do this by launching an existing AMI and then just building a new
>> version of spark and copying it to the slaves....
>>
>> - Patrick
>>
>> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <pw...@gmail.com>
>> wrote:
>> > You'll have to build your own. Also there are some packaging
>> > differences in master (some bin/ scripts moved to sbin/) just to give
>> > you a heads up.
>> >
>> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <bu...@gmail.com>
>> > wrote:
>> >> Good to know the next release will be on scala 2.10.
>> >>
>> >> Meanwhile, does using the master branch mean that I will have to build
>> >> my
>> >> own AMI when launching an ec2 cluster? Also, is there a nightly binary
>> >> build
>> >> maven repo available for spark?
>> >>
>> >>
>> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <il...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Scala 2.10.3 support was recently merged into master (#259). The
>> >>> branch is
>> >>> probably not as stable as 0.8.1, but things "should" work.
>> >>>
>> >>> The 2.10 branch should be deleted, the only issue is there are some
>> >>> outstanding PRs against that branch that haven't been moved to master.
>> >>>
>> >>>
>> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia
>> >>> <bu...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> I was going to give https://github.com/scala/pickling a try on spark
>> >>>> to
>> >>>> see how it would compare with kryo. Unfortunately, it only works with
>> >>>> scala
>> >>>> 2.10.3.
>> >>>>
>> >>>> - Is there a time line for spark to work with scala 2.10?
>> >>>>
>> >>>> - Is the 2.10 branch as stable as 2.9?
>> >>>>
>> >>>> - What's blocking spark to work with 2.10?
>> >>>
>> >>>
>> >>
>
>

Re: State of spark on scala 2.10

[Aureliano Buendia <bu...@gmail.com>]

So I should just lanuch an AMI from one of
https://github.com/mesos/spark-ec2/tree/v2/ami-list and build the
development version on it? Is it just a simple matter of copying the target
to a directory, or does it need a lot of configurations?


On Sun, Jan 5, 2014 at 5:03 AM, Patrick Wendell <pw...@gmail.com> wrote:

> I meant you'll need to build your own version of Spark. Typically we
> do this by launching an existing AMI and then just building a new
> version of spark and copying it to the slaves....
>
> - Patrick
>
> On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <pw...@gmail.com>
> wrote:
> > You'll have to build your own. Also there are some packaging
> > differences in master (some bin/ scripts moved to sbin/) just to give
> > you a heads up.
> >
> > On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <bu...@gmail.com>
> wrote:
> >> Good to know the next release will be on scala 2.10.
> >>
> >> Meanwhile, does using the master branch mean that I will have to build
> my
> >> own AMI when launching an ec2 cluster? Also, is there a nightly binary
> build
> >> maven repo available for spark?
> >>
> >>
> >> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <il...@gmail.com>
> wrote:
> >>>
> >>> Scala 2.10.3 support was recently merged into master (#259). The
> branch is
> >>> probably not as stable as 0.8.1, but things "should" work.
> >>>
> >>> The 2.10 branch should be deleted, the only issue is there are some
> >>> outstanding PRs against that branch that haven't been moved to master.
> >>>
> >>>
> >>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <
> buendia360@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I was going to give https://github.com/scala/pickling a try on spark
> to
> >>>> see how it would compare with kryo. Unfortunately, it only works with
> scala
> >>>> 2.10.3.
> >>>>
> >>>> - Is there a time line for spark to work with scala 2.10?
> >>>>
> >>>> - Is the 2.10 branch as stable as 2.9?
> >>>>
> >>>> - What's blocking spark to work with 2.10?
> >>>
> >>>
> >>
>

Re: State of spark on scala 2.10

[Patrick Wendell <pw...@gmail.com>]

I meant you'll need to build your own version of Spark. Typically we
do this by launching an existing AMI and then just building a new
version of spark and copying it to the slaves....

- Patrick

On Sat, Jan 4, 2014 at 8:44 PM, Patrick Wendell <pw...@gmail.com> wrote:
> You'll have to build your own. Also there are some packaging
> differences in master (some bin/ scripts moved to sbin/) just to give
> you a heads up.
>
> On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <bu...@gmail.com> wrote:
>> Good to know the next release will be on scala 2.10.
>>
>> Meanwhile, does using the master branch mean that I will have to build my
>> own AMI when launching an ec2 cluster? Also, is there a nightly binary build
>> maven repo available for spark?
>>
>>
>> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <il...@gmail.com> wrote:
>>>
>>> Scala 2.10.3 support was recently merged into master (#259). The branch is
>>> probably not as stable as 0.8.1, but things "should" work.
>>>
>>> The 2.10 branch should be deleted, the only issue is there are some
>>> outstanding PRs against that branch that haven't been moved to master.
>>>
>>>
>>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <bu...@gmail.com>
>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I was going to give https://github.com/scala/pickling a try on spark to
>>>> see how it would compare with kryo. Unfortunately, it only works with scala
>>>> 2.10.3.
>>>>
>>>> - Is there a time line for spark to work with scala 2.10?
>>>>
>>>> - Is the 2.10 branch as stable as 2.9?
>>>>
>>>> - What's blocking spark to work with 2.10?
>>>
>>>
>>

Re: State of spark on scala 2.10

[Patrick Wendell <pw...@gmail.com>]

You'll have to build your own. Also there are some packaging
differences in master (some bin/ scripts moved to sbin/) just to give
you a heads up.

On Sat, Jan 4, 2014 at 8:14 PM, Aureliano Buendia <bu...@gmail.com> wrote:
> Good to know the next release will be on scala 2.10.
>
> Meanwhile, does using the master branch mean that I will have to build my
> own AMI when launching an ec2 cluster? Also, is there a nightly binary build
> maven repo available for spark?
>
>
> On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <il...@gmail.com> wrote:
>>
>> Scala 2.10.3 support was recently merged into master (#259). The branch is
>> probably not as stable as 0.8.1, but things "should" work.
>>
>> The 2.10 branch should be deleted, the only issue is there are some
>> outstanding PRs against that branch that haven't been moved to master.
>>
>>
>> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <bu...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I was going to give https://github.com/scala/pickling a try on spark to
>>> see how it would compare with kryo. Unfortunately, it only works with scala
>>> 2.10.3.
>>>
>>> - Is there a time line for spark to work with scala 2.10?
>>>
>>> - Is the 2.10 branch as stable as 2.9?
>>>
>>> - What's blocking spark to work with 2.10?
>>
>>
>

Re: State of spark on scala 2.10

[Aureliano Buendia <bu...@gmail.com>]

Good to know the next release will be on scala 2.10.

Meanwhile, does using the master branch mean that I will have to build my
own AMI when launching an ec2 cluster? Also, is there a nightly binary
build maven repo available for spark?


On Sun, Jan 5, 2014 at 3:56 AM, Aaron Davidson <il...@gmail.com> wrote:

> Scala 2.10.3 support was recently merged into master (#259<https://github.com/apache/incubator-spark/pull/259>).
> The branch is probably not as stable as 0.8.1, but things "should" work.
>
> The 2.10 branch should be deleted, the only issue is there are some
> outstanding PRs against that branch that haven't been moved to master.
>
>
> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <bu...@gmail.com>wrote:
>
>> Hi,
>>
>> I was going to give https://github.com/scala/pickling a try on spark to
>> see how it would compare with kryo. Unfortunately, it only works with scala
>> 2.10.3.
>>
>> - Is there a time line for spark to work with scala 2.10?
>>
>> - Is the 2.10 branch as stable as 2.9?
>>
>> - What's blocking spark to work with 2.10?
>>
>
>

Re: State of spark on scala 2.10

[Tathagata Das <ta...@gmail.com>]

Thumbs up on your plans of trying pickling. I was at the Scala Days 2013
presentation of pickling and it was very cool. Would be fun to see how does
compile-time generation of serialization code perform against Kryo.

TD


On Sat, Jan 4, 2014 at 7:56 PM, Aaron Davidson <il...@gmail.com> wrote:

> Scala 2.10.3 support was recently merged into master (#259<https://github.com/apache/incubator-spark/pull/259>).
> The branch is probably not as stable as 0.8.1, but things "should" work.
>
> The 2.10 branch should be deleted, the only issue is there are some
> outstanding PRs against that branch that haven't been moved to master.
>
>
> On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <bu...@gmail.com>wrote:
>
>> Hi,
>>
>> I was going to give https://github.com/scala/pickling a try on spark to
>> see how it would compare with kryo. Unfortunately, it only works with scala
>> 2.10.3.
>>
>> - Is there a time line for spark to work with scala 2.10?
>>
>> - Is the 2.10 branch as stable as 2.9?
>>
>> - What's blocking spark to work with 2.10?
>>
>
>

Re: State of spark on scala 2.10

[Aaron Davidson <il...@gmail.com>]

Scala 2.10.3 support was recently merged into master
(#259<https://github.com/apache/incubator-spark/pull/259>).
The branch is probably not as stable as 0.8.1, but things "should" work.

The 2.10 branch should be deleted, the only issue is there are some
outstanding PRs against that branch that haven't been moved to master.


On Sat, Jan 4, 2014 at 7:11 PM, Aureliano Buendia <bu...@gmail.com>wrote:

> Hi,
>
> I was going to give https://github.com/scala/pickling a try on spark to
> see how it would compare with kryo. Unfortunately, it only works with scala
> 2.10.3.
>
> - Is there a time line for spark to work with scala 2.10?
>
> - Is the 2.10 branch as stable as 2.9?
>
> - What's blocking spark to work with 2.10?
>