You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Reynold Xin <rx...@databricks.com> on 2016/05/25 06:44:36 UTC
[ANNOUNCE] Apache Spark 2.0.0-preview release
In the past the Spark community have created preview packages (not official
releases) and used those as opportunities to ask community members to test
the upcoming versions of Apache Spark. Several people in the Apache
community have suggested we conduct votes for these preview packages and
turn them into formal releases by the Apache foundation's standard. This is
a result of that.
Note that this preview release should contain almost all the new features
that will be in Apache Spark 2.0.0. However, it is not meant to be
functional, i.e. the preview release contain critical bugs and
documentation errors. To download, please see the bottom of this web page:
http://spark.apache.org/downloads.html
For the list of known issues, please see
https://issues.apache.org/jira/browse/SPARK-15520?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%2Fs%22%20%3D%202.0.0
Note 1: The current download link goes directly to dist.apache.org. Once
all the files are propagated to all mirrors, I will update the link to link
to the mirror selector instead.
Note 2: This is the first time we are publishing official, voted preview
releases. Would love to hear feedback.
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Marcin Tustin <mt...@handybook.com>.
The use case of docker images in general is that you can deploy and develop
with exactly the same binary environment - same java 8, same scala, same
spark. This makes things repeatable.
On Wed, May 25, 2016 at 8:38 PM, Matei Zaharia <ma...@gmail.com>
wrote:
> Just wondering, what is the main use case for the Docker images -- to
> develop apps locally or to deploy a cluster? If the image is really just a
> script to download a certain package name from a mirror, it may be okay to
> create an official one, though it does seem tricky to make it properly use
> the right mirror.
>
> Matei
>
> On May 25, 2016, at 6:05 PM, Luciano Resende <lu...@gmail.com> wrote:
>
>
>
> On Wed, May 25, 2016 at 2:34 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> I don't think the project would bless anything but the standard
>> release artifacts since only those are voted on. People are free to
>> maintain whatever they like and even share it, as long as it's clear
>> it's not from the Apache project.
>>
>>
> +1
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
>
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Gurvinder Singh <gu...@uninett.no>.
On 05/26/2016 02:38 AM, Matei Zaharia wrote:
> Just wondering, what is the main use case for the Docker images -- to
> develop apps locally or to deploy a cluster?
I use docker images for both development and deploying on production
cluster. As it makes sure I have the correct version of Java and Spark.
If the image is really just
> a script to download a certain package name from a mirror, it may be
> okay to create an official one, though it does seem tricky to make it
> properly use the right mirror.
I don't think that's an issue, as you will publish a docker image which
will already have spark baked from which ever mirror you choose. The
mirror issue will be only when people want to build their own image from
published Dockerfile, then they can change the mirror if they prefer.
Here is the link to current Spark Dockerfile
(https://gist.github.com/gurvindersingh/8308d46995a58303b90e4bc2fc46e343)
I use as base, then I can start master and worker from it as I like.
- Gurvinder
>
> Matei
>
>> On May 25, 2016, at 6:05 PM, Luciano Resende <luckbr1975@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>>
>>
>> On Wed, May 25, 2016 at 2:34 PM, Sean Owen <sowen@cloudera.com
>> <ma...@cloudera.com>> wrote:
>>
>> I don't think the project would bless anything but the standard
>> release artifacts since only those are voted on. People are free to
>> maintain whatever they like and even share it, as long as it's clear
>> it's not from the Apache project.
>>
>>
>> +1
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Matei Zaharia <ma...@gmail.com>.
Just wondering, what is the main use case for the Docker images -- to develop apps locally or to deploy a cluster? If the image is really just a script to download a certain package name from a mirror, it may be okay to create an official one, though it does seem tricky to make it properly use the right mirror.
Matei
> On May 25, 2016, at 6:05 PM, Luciano Resende <lu...@gmail.com> wrote:
>
>
>
> On Wed, May 25, 2016 at 2:34 PM, Sean Owen <sowen@cloudera.com <ma...@cloudera.com>> wrote:
> I don't think the project would bless anything but the standard
> release artifacts since only those are voted on. People are free to
> maintain whatever they like and even share it, as long as it's clear
> it's not from the Apache project.
>
>
> +1
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975 <http://twitter.com/lresende1975>
> http://lresende.blogspot.com/ <http://lresende.blogspot.com/>
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Luciano Resende <lu...@gmail.com>.
On Wed, May 25, 2016 at 2:34 PM, Sean Owen <so...@cloudera.com> wrote:
> I don't think the project would bless anything but the standard
> release artifacts since only those are voted on. People are free to
> maintain whatever they like and even share it, as long as it's clear
> it's not from the Apache project.
>
>
+1
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Sean Owen <so...@cloudera.com>.
I don't think the project would bless anything but the standard
release artifacts since only those are voted on. People are free to
maintain whatever they like and even share it, as long as it's clear
it's not from the Apache project.
On Wed, May 25, 2016 at 3:41 PM, Marcin Tustin <mt...@handybook.com> wrote:
> Ah very nice. Would it be possible to have this blessed into an official
> image?
>
> On Wed, May 25, 2016 at 4:12 PM, Luciano Resende <lu...@gmail.com>
> wrote:
>>
>>
>>
>> On Wed, May 25, 2016 at 6:53 AM, Marcin Tustin <mt...@handybook.com>
>> wrote:
>>>
>>> Would it be useful to start baking docker images? Would anyone find that
>>> a boon to their testing?
>>>
>>
>> +1, I had done one (still based on 1.6) for some SystemML experiments, I
>> could easily get it based on a nightly build.
>>
>> https://github.com/lresende/docker-spark
>>
>> One question tough, how often should the image be updated ? every night ?
>> every week ? I could see if I can automate the build + publish in a CI job
>> at one of our Jenkins servers (Apache or something)...
>>
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>
>
>
> Want to work at Handy? Check out our culture deck and open roles
> Latest news at Handy
> Handy just raised $50m led by Fidelity
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Marcin Tustin <mt...@handybook.com>.
Ah very nice. Would it be possible to have this blessed into an official
image?
On Wed, May 25, 2016 at 4:12 PM, Luciano Resende <lu...@gmail.com>
wrote:
>
>
> On Wed, May 25, 2016 at 6:53 AM, Marcin Tustin <mt...@handybook.com>
> wrote:
>
>> Would it be useful to start baking docker images? Would anyone find that
>> a boon to their testing?
>>
>>
> +1, I had done one (still based on 1.6) for some SystemML experiments, I
> could easily get it based on a nightly build.
>
> https://github.com/lresende/docker-spark
>
> One question tough, how often should the image be updated ? every night ?
> every week ? I could see if I can automate the build + publish in a CI job
> at one of our Jenkins servers (Apache or something)...
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Luciano Resende <lu...@gmail.com>.
On Wed, May 25, 2016 at 6:53 AM, Marcin Tustin <mt...@handybook.com>
wrote:
> Would it be useful to start baking docker images? Would anyone find that a
> boon to their testing?
>
>
+1, I had done one (still based on 1.6) for some SystemML experiments, I
could easily get it based on a nightly build.
https://github.com/lresende/docker-spark
One question tough, how often should the image be updated ? every night ?
every week ? I could see if I can automate the build + publish in a CI job
at one of our Jenkins servers (Apache or something)...
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Marcin Tustin <mt...@handybook.com>.
Would it be useful to start baking docker images? Would anyone find that a
boon to their testing?
On Wed, May 25, 2016 at 2:44 AM, Reynold Xin <rx...@databricks.com> wrote:
> In the past the Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the
> Apache community have suggested we conduct votes for these preview packages
> and turn them into formal releases by the Apache foundation's standard.
> This is a result of that.
>
> Note that this preview release should contain almost all the new features
> that will be in Apache Spark 2.0.0. However, it is not meant to be
> functional, i.e. the preview release contain critical bugs and
> documentation errors. To download, please see the bottom of this web
> page: http://spark.apache.org/downloads.html
>
> For the list of known issues, please see
> https://issues.apache.org/jira/browse/SPARK-15520?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%2Fs%22%20%3D%202.0.0
>
>
> Note 1: The current download link goes directly to dist.apache.org. Once
> all the files are propagated to all mirrors, I will update the link to link
> to the mirror selector instead.
>
> Note 2: This is the first time we are publishing official, voted preview
> releases. Would love to hear feedback.
>
>
>
--
Want to work at Handy? Check out our culture deck and open roles
<http://www.handy.com/careers>
Latest news <http://www.handy.com/press> at Handy
Handy just raised $50m
<http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/> led
by Fidelity
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Sean Owen <so...@cloudera.com>.
I still don't see any artifacts in maven -- did it publish?
http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.spark%22
On Wed, May 25, 2016 at 10:30 AM, Reynold Xin <rx...@databricks.com> wrote:
> Yup I have published it to maven. Will post the link in a bit.
>
> One thing is that for developers, it might be better to use the nightly
> snapshot because that one probably has fewer bugs than the preview one.
>
>
> On Wednesday, May 25, 2016, Daniel Darabos
> <da...@lynxanalytics.com> wrote:
>>
>> Awesome, thanks! It's very helpful for preparing for the migration. Do you
>> plan to push 2.0.0-preview to Maven too? (I for one would appreciate the
>> convenience.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Reynold Xin <rx...@databricks.com>.
The maven artifacts can be found at
https://repository.apache.org/content/repositories/orgapachespark-1182/
But really for people on this list, it might be better to go straight to
the nightly snapshots.
https://repository.apache.org/content/groups/snapshots/org/apache/spark/
On Wed, May 25, 2016 at 8:30 AM, Reynold Xin <rx...@databricks.com> wrote:
> Yup I have published it to maven. Will post the link in a bit.
>
> One thing is that for developers, it might be better to use the nightly
> snapshot because that one probably has fewer bugs than the preview one.
>
>
> On Wednesday, May 25, 2016, Daniel Darabos <
> daniel.darabos@lynxanalytics.com> wrote:
>
>> Awesome, thanks! It's very helpful for preparing for the migration. Do
>> you plan to push 2.0.0-preview to Maven too? (I for one would appreciate
>> the convenience.)
>>
>> On Wed, May 25, 2016 at 8:44 AM, Reynold Xin <rx...@databricks.com> wrote:
>>
>>> In the past the Spark community have created preview packages (not
>>> official releases) and used those as opportunities to ask community members
>>> to test the upcoming versions of Apache Spark. Several people in the
>>> Apache community have suggested we conduct votes for these preview packages
>>> and turn them into formal releases by the Apache foundation's standard.
>>> This is a result of that.
>>>
>>> Note that this preview release should contain almost all the new
>>> features that will be in Apache Spark 2.0.0. However, it is not meant
>>> to be functional, i.e. the preview release contain critical bugs and
>>> documentation errors. To download, please see the bottom of this web
>>> page: http://spark.apache.org/downloads.html
>>>
>>> For the list of known issues, please see
>>> https://issues.apache.org/jira/browse/SPARK-15520?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%2Fs%22%20%3D%202.0.0
>>>
>>>
>>> Note 1: The current download link goes directly to dist.apache.org.
>>> Once all the files are propagated to all mirrors, I will update the link to
>>> link to the mirror selector instead.
>>>
>>> Note 2: This is the first time we are publishing official, voted preview
>>> releases. Would love to hear feedback.
>>>
>>>
>>>
>>
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Reynold Xin <rx...@databricks.com>.
Yup I have published it to maven. Will post the link in a bit.
One thing is that for developers, it might be better to use the nightly
snapshot because that one probably has fewer bugs than the preview one.
On Wednesday, May 25, 2016, Daniel Darabos <da...@lynxanalytics.com>
wrote:
> Awesome, thanks! It's very helpful for preparing for the migration. Do you
> plan to push 2.0.0-preview to Maven too? (I for one would appreciate the
> convenience.)
>
> On Wed, May 25, 2016 at 8:44 AM, Reynold Xin <rxin@databricks.com
> <javascript:_e(%7B%7D,'cvml','rxin@databricks.com');>> wrote:
>
>> In the past the Spark community have created preview packages (not
>> official releases) and used those as opportunities to ask community members
>> to test the upcoming versions of Apache Spark. Several people in the
>> Apache community have suggested we conduct votes for these preview packages
>> and turn them into formal releases by the Apache foundation's standard.
>> This is a result of that.
>>
>> Note that this preview release should contain almost all the new features
>> that will be in Apache Spark 2.0.0. However, it is not meant to be
>> functional, i.e. the preview release contain critical bugs and
>> documentation errors. To download, please see the bottom of this web
>> page: http://spark.apache.org/downloads.html
>>
>> For the list of known issues, please see
>> https://issues.apache.org/jira/browse/SPARK-15520?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%2Fs%22%20%3D%202.0.0
>>
>>
>> Note 1: The current download link goes directly to dist.apache.org. Once
>> all the files are propagated to all mirrors, I will update the link to link
>> to the mirror selector instead.
>>
>> Note 2: This is the first time we are publishing official, voted preview
>> releases. Would love to hear feedback.
>>
>>
>>
>
Re: [ANNOUNCE] Apache Spark 2.0.0-preview release
Posted by Daniel Darabos <da...@lynxanalytics.com>.
Awesome, thanks! It's very helpful for preparing for the migration. Do you
plan to push 2.0.0-preview to Maven too? (I for one would appreciate the
convenience.)
On Wed, May 25, 2016 at 8:44 AM, Reynold Xin <rx...@databricks.com> wrote:
> In the past the Spark community have created preview packages (not
> official releases) and used those as opportunities to ask community members
> to test the upcoming versions of Apache Spark. Several people in the
> Apache community have suggested we conduct votes for these preview packages
> and turn them into formal releases by the Apache foundation's standard.
> This is a result of that.
>
> Note that this preview release should contain almost all the new features
> that will be in Apache Spark 2.0.0. However, it is not meant to be
> functional, i.e. the preview release contain critical bugs and
> documentation errors. To download, please see the bottom of this web
> page: http://spark.apache.org/downloads.html
>
> For the list of known issues, please see
> https://issues.apache.org/jira/browse/SPARK-15520?jql=project%20%3D%20SPARK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%2Fs%22%20%3D%202.0.0
>
>
> Note 1: The current download link goes directly to dist.apache.org. Once
> all the files are propagated to all mirrors, I will update the link to link
> to the mirror selector instead.
>
> Note 2: This is the first time we are publishing official, voted preview
> releases. Would love to hear feedback.
>
>
>