You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Anand Avati <av...@gluster.org> on 2014/05/09 03:33:30 UTC
Spark on Scala 2.11
Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
Approximate timeline?
Thanks
Re: Spark on Scala 2.11
Posted by Koert Kuipers <ko...@tresata.com>.
i believe matei has said before that he would like to crossbuild for 2.10
and 2.11, given that the difference is not as big as between 2.9 and 2.10.
but dont know when this would happen...
On Sat, May 10, 2014 at 11:02 PM, Gary Malouf <ma...@gmail.com> wrote:
> Considering the team just bumped to 2.10 in 0.9, I would be surprised if
> this is a near term priority.
>
>
> On Thu, May 8, 2014 at 9:33 PM, Anand Avati <av...@gluster.org> wrote:
>
> > Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
> > Approximate timeline?
> >
> > Thanks
> >
>
Re: Spark on Scala 2.11
Posted by Gary Malouf <ma...@gmail.com>.
Considering the team just bumped to 2.10 in 0.9, I would be surprised if
this is a near term priority.
On Thu, May 8, 2014 at 9:33 PM, Anand Avati <av...@gluster.org> wrote:
> Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
> Approximate timeline?
>
> Thanks
>
Re: Spark on Scala 2.11
Posted by Jacek Laskowski <ja...@japila.pl>.
Thanks a lot!
Jacek
On Tue, May 13, 2014 at 1:54 AM, Matei Zaharia <ma...@gmail.com> wrote:
> Anyone can actually open a JIRA on https://issues.apache.org/jira/browse/SPARK. I’ve created one for this now: https://issues.apache.org/jira/browse/SPARK-1812.
>
> Matei
>
> On May 12, 2014, at 3:54 PM, Jacek Laskowski <ja...@japila.pl> wrote:
>
>> On Sun, May 11, 2014 at 11:08 PM, Matei Zaharia <ma...@gmail.com> wrote:
>>> We do want to support it eventually, possibly as early as Spark 1.1 (which we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it before, feel free to do so! Scala 2.11 is very close to 2.10 so I think things will mostly work, except for possibly the REPL (which has require porting over code form the Scala REPL in each version).
>>
>> Hi,
>>
>> Would that be possible to have a JIRA issue for this (so I could have
>> a branch for the cross-build in sbt and give the task a try)?
>>
>> Jacek
>>
>> --
>> Jacek Laskowski | http://blog.japila.pl
>> "Never discourage anyone who continually makes progress, no matter how
>> slow." Plato
>
--
Jacek Laskowski | http://blog.japila.pl
"Never discourage anyone who continually makes progress, no matter how
slow." Plato
Re: Spark on Scala 2.11
Posted by Matei Zaharia <ma...@gmail.com>.
Anyone can actually open a JIRA on https://issues.apache.org/jira/browse/SPARK. I’ve created one for this now: https://issues.apache.org/jira/browse/SPARK-1812.
Matei
On May 12, 2014, at 3:54 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> On Sun, May 11, 2014 at 11:08 PM, Matei Zaharia <ma...@gmail.com> wrote:
>> We do want to support it eventually, possibly as early as Spark 1.1 (which we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it before, feel free to do so! Scala 2.11 is very close to 2.10 so I think things will mostly work, except for possibly the REPL (which has require porting over code form the Scala REPL in each version).
>
> Hi,
>
> Would that be possible to have a JIRA issue for this (so I could have
> a branch for the cross-build in sbt and give the task a try)?
>
> Jacek
>
> --
> Jacek Laskowski | http://blog.japila.pl
> "Never discourage anyone who continually makes progress, no matter how
> slow." Plato
Re: Spark on Scala 2.11
Posted by Jacek Laskowski <ja...@japila.pl>.
On Sun, May 11, 2014 at 11:08 PM, Matei Zaharia <ma...@gmail.com> wrote:
> We do want to support it eventually, possibly as early as Spark 1.1 (which we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it before, feel free to do so! Scala 2.11 is very close to 2.10 so I think things will mostly work, except for possibly the REPL (which has require porting over code form the Scala REPL in each version).
Hi,
Would that be possible to have a JIRA issue for this (so I could have
a branch for the cross-build in sbt and give the task a try)?
Jacek
--
Jacek Laskowski | http://blog.japila.pl
"Never discourage anyone who continually makes progress, no matter how
slow." Plato
Re: Spark on Scala 2.11
Posted by Anand Avati <av...@gluster.org>.
On Mon, May 12, 2014 at 6:27 PM, Matei Zaharia <ma...@gmail.com>wrote:
> We can build the REPL separately for each version of Scala, or even give
> that package a different name in Scala 2.11.
>
OK.
> Scala 2.11’s REPL actually added two flags, -Yrepl-class-based and
> -Yrepl-outdir, that encompass the two modifications we made to the REPL
> (using classes instead of objects to wrap each line, and grabbing the files
> from some directory). So it might be possible to run it without
> modifications using just a simple wrapper class around it. That would
> definitely simplify things!
>
Exactly. I have been tracking those changes in 2.11 as well. We would need
a simple wrapper around ILoop, set the Yreplclassbased flag, an HttpServer
to export the repl-outdir, bind sc into repl namespace and maybe little
more initialization (customizing repl init code in 2.11 is not as trivial
as the <=2.10 versions, but still doable.) Still, even with all this, it
should be much simpler than pulling in and refactoring everything like
today. I already have a prototype for this in my working tree, still needs
integration testing.
> BTW did the non-REPL parts run fine on 2.11?
>
Currently fighting to get all the dependencies in 2.11. Quick pointer where
I can get sources for akka-*-X.Y-shared-protobuf? Also, what's the smallest
set of dependencies to build the smallest testable subset of the project?
Thanks!
> Matei
>
> On May 12, 2014, at 2:09 PM, Anand Avati <av...@gluster.org> wrote:
>
> > Matei,
> > Thanks for confirming. I was looking specifically at the REPL part and
> how
> > it can be significantly simplified with 2.11 Scala, without having to
> > inherit a full copy of a refactored repl inside Spark. I am happy to
> > investigate/contribute a simpler 2.11 based REPL if this is were seen as
> a
> > priority (1.1 does not seem "too far" away.) However a 2.10 compatible
> > cross build would still require a separate (existing) REPL code for the
> > 2.10 build, no?
> >
> > Thanks.
> >
> > On Sun, May 11, 2014 at 2:08 PM, Matei Zaharia <matei.zaharia@gmail.com
> >wrote:
> >
> >> We do want to support it eventually, possibly as early as Spark 1.1
> (which
> >> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it
> >> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think
> >> things will mostly work, except for possibly the REPL (which has require
> >> porting over code form the Scala REPL in each version).
> >>
> >> Matei
> >>
> >> On May 8, 2014, at 6:33 PM, Anand Avati <av...@gluster.org> wrote:
> >>
> >>> Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
> >>> Approximate timeline?
> >>>
> >>> Thank
> >>
>
>
Re: Spark on Scala 2.11
Posted by Matei Zaharia <ma...@gmail.com>.
We can build the REPL separately for each version of Scala, or even give that package a different name in Scala 2.11.
Scala 2.11’s REPL actually added two flags, -Yrepl-class-based and -Yrepl-outdir, that encompass the two modifications we made to the REPL (using classes instead of objects to wrap each line, and grabbing the files from some directory). So it might be possible to run it without modifications using just a simple wrapper class around it. That would definitely simplify things!
BTW did the non-REPL parts run fine on 2.11?
Matei
On May 12, 2014, at 2:09 PM, Anand Avati <av...@gluster.org> wrote:
> Matei,
> Thanks for confirming. I was looking specifically at the REPL part and how
> it can be significantly simplified with 2.11 Scala, without having to
> inherit a full copy of a refactored repl inside Spark. I am happy to
> investigate/contribute a simpler 2.11 based REPL if this is were seen as a
> priority (1.1 does not seem "too far" away.) However a 2.10 compatible
> cross build would still require a separate (existing) REPL code for the
> 2.10 build, no?
>
> Thanks.
>
> On Sun, May 11, 2014 at 2:08 PM, Matei Zaharia <ma...@gmail.com>wrote:
>
>> We do want to support it eventually, possibly as early as Spark 1.1 (which
>> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it
>> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think
>> things will mostly work, except for possibly the REPL (which has require
>> porting over code form the Scala REPL in each version).
>>
>> Matei
>>
>> On May 8, 2014, at 6:33 PM, Anand Avati <av...@gluster.org> wrote:
>>
>>> Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
>>> Approximate timeline?
>>>
>>> Thank
>>
Re: Spark on Scala 2.11
Posted by Anand Avati <av...@gluster.org>.
Matei,
Thanks for confirming. I was looking specifically at the REPL part and how
it can be significantly simplified with 2.11 Scala, without having to
inherit a full copy of a refactored repl inside Spark. I am happy to
investigate/contribute a simpler 2.11 based REPL if this is were seen as a
priority (1.1 does not seem "too far" away.) However a 2.10 compatible
cross build would still require a separate (existing) REPL code for the
2.10 build, no?
Thanks.
On Sun, May 11, 2014 at 2:08 PM, Matei Zaharia <ma...@gmail.com>wrote:
> We do want to support it eventually, possibly as early as Spark 1.1 (which
> we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it
> before, feel free to do so! Scala 2.11 is very close to 2.10 so I think
> things will mostly work, except for possibly the REPL (which has require
> porting over code form the Scala REPL in each version).
>
> Matei
>
> On May 8, 2014, at 6:33 PM, Anand Avati <av...@gluster.org> wrote:
>
> > Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
> > Approximate timeline?
> >
> > Thank
>
Re: Spark on Scala 2.11
Posted by Matei Zaharia <ma...@gmail.com>.
We do want to support it eventually, possibly as early as Spark 1.1 (which we’d cross-build on Scala 2.10 and 2.11). If someone wants to look at it before, feel free to do so! Scala 2.11 is very close to 2.10 so I think things will mostly work, except for possibly the REPL (which has require porting over code form the Scala REPL in each version).
Matei
On May 8, 2014, at 6:33 PM, Anand Avati <av...@gluster.org> wrote:
> Is there an ongoing effort (or intent) to support Spark on Scala 2.11?
> Approximate timeline?
>
> Thanks