You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Reynold Xin <rx...@databricks.com> on 2015/11/04 00:22:28 UTC
[VOTE] Release Apache Spark 1.5.2 (RC2)
Please vote on releasing the following candidate as Apache Spark version
1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
majority of at least 3 +1 PMC votes are cast.
[ ] +1 Release this package as Apache Spark 1.5.2
[ ] -1 Do not release this package because ...
The release fixes 59 known issues in Spark 1.5.1, listed here:
http://s.apache.org/spark-1.5.2
The tag to be voted on is v1.5.2-rc2:
https://github.com/apache/spark/releases/tag/v1.5.2-rc2
The release files, including signatures, digests, etc. can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc
The staging repository for this release can be found at:
- as version 1.5.2-rc2:
https://repository.apache.org/content/repositories/orgapachespark-1153
- as version 1.5.2:
https://repository.apache.org/content/repositories/orgapachespark-1152
The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
=======================================
How can I help test this release?
=======================================
If you are a Spark user, you can help us test this release by taking an
existing Spark workload and running on this release candidate, then
reporting any regressions.
================================================
What justifies a -1 vote for this release?
================================================
-1 vote should occur for regressions from Spark 1.5.1. Bugs already present
in 1.5.1 will not block this release.
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Reynold Xin <rx...@databricks.com>.
Thanks everybody for voting. I'm going to close the vote now. The vote
passes with 14 +1 votes and no -1 vote. I will work on packaging this asap.
+1:
Jean-Baptiste Onofré
Egor Pahomov
Luc Bourlier
Tom Graves*
Chester Chen
Michael Armbrust*
Krishna Sankar
Robin East
Reynold Xin*
Joseph Bradley
Mark Hamstra
Denny Lee
Ted Yu
Sean McNamara
-0:
Nicholas Chammas (spark-ec2 issue)
-1:
none
On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> =======================================
> How can I help test this release?
> =======================================
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> ================================================
> What justifies a -1 vote for this release?
> ================================================
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
> present in 1.5.1 will not block this release.
>
>
>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Tom Graves <tg...@yahoo.com.INVALID>.
Its either --num-workers or --num-executors when using the spark-class interface directly. If you use spark-submit with --num-executors it ends up setting spark.executor.instances which works around the issue.
Tom
On Friday, November 6, 2015 2:14 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
The way I read Tom's report, it just affects a long-deprecated command
line option (--num-workers). I wouldn't block the release for it.
On Fri, Nov 6, 2015 at 12:10 PM, Sean Owen <so...@cloudera.com> wrote:
> Hm, if I read that right, looks like --num-executors doesn't work at
> all on YARN unless dynamic allocation is on? the fix is easy, but
> sounds like it could be a Blocker.
>
> On Fri, Nov 6, 2015 at 2:51 PM, Tom Graves <tg...@yahoo.com> wrote:
>> While running our regression tests I found
>> https://issues.apache.org/jira/browse/SPARK-11555. It is a break in
>> backwards compatibility but its using the old spark-class and --num-workers
>> interface which I hope no one is still using.
>>
>> I'm a +0 as it doesn't seem super critical but I hate to break backwards
>> compatibility unless we explicitly decide to.
>>
>>
>> Tom
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Marcelo Vanzin <va...@cloudera.com>.
The way I read Tom's report, it just affects a long-deprecated command
line option (--num-workers). I wouldn't block the release for it.
On Fri, Nov 6, 2015 at 12:10 PM, Sean Owen <so...@cloudera.com> wrote:
> Hm, if I read that right, looks like --num-executors doesn't work at
> all on YARN unless dynamic allocation is on? the fix is easy, but
> sounds like it could be a Blocker.
>
> On Fri, Nov 6, 2015 at 2:51 PM, Tom Graves <tg...@yahoo.com> wrote:
>> While running our regression tests I found
>> https://issues.apache.org/jira/browse/SPARK-11555. It is a break in
>> backwards compatibility but its using the old spark-class and --num-workers
>> interface which I hope no one is still using.
>>
>> I'm a +0 as it doesn't seem super critical but I hate to break backwards
>> compatibility unless we explicitly decide to.
>>
>>
>> Tom
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Sean Owen <so...@cloudera.com>.
Hm, if I read that right, looks like --num-executors doesn't work at
all on YARN unless dynamic allocation is on? the fix is easy, but
sounds like it could be a Blocker.
On Fri, Nov 6, 2015 at 2:51 PM, Tom Graves <tg...@yahoo.com> wrote:
> While running our regression tests I found
> https://issues.apache.org/jira/browse/SPARK-11555. It is a break in
> backwards compatibility but its using the old spark-class and --num-workers
> interface which I hope no one is still using.
>
> I'm a +0 as it doesn't seem super critical but I hate to break backwards
> compatibility unless we explicitly decide to.
>
>
> Tom
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Tom Graves <tg...@yahoo.com.INVALID>.
While running our regression tests I found https://issues.apache.org/jira/browse/SPARK-11555. It is a break in backwards compatibility but its using the old spark-class and --num-workers interface which I hope no one is still using.
I'm a +0 as it doesn't seem super critical but I hate to break backwards compatibility unless we explicitly decide to.
Tom
On Thursday, November 5, 2015 2:02 PM, Nicholas Chammas <ni...@gmail.com> wrote:
-0
The spark-ec2 version is still set to 1.5.1.
Nick
On Wed, Nov 4, 2015 at 8:20 PM Egor Pahomov <pa...@gmail.com> wrote:
+1
Things, which our infrastructure use and I checked:
Dynamic allocationSpark ODBC serverReading jsonWriting parquetSQL quires (hive context)Running on CDH
2015-11-04 9:03 GMT-08:00 Sean Owen <so...@cloudera.com>:
As usual the signatures and licenses and so on look fine. I continue
to get the same test failures on Ubuntu in Java 7/8:
- Unpersisting HttpBroadcast on executors only in distributed mode ***
FAILED ***
But I continue to assume that's specific to tests and/or Ubuntu and/or
the build profile, since I don't see any evidence of this in other
builds on Jenkins. It's not a change from previous behavior, though it
doesn't always happen either.
On Tue, Nov 3, 2015 at 11:22 PM, Reynold Xin <rx...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> =======================================
> How can I help test this release?
> =======================================
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> ================================================
> What justifies a -1 vote for this release?
> ================================================
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already present
> in 1.5.1 will not block this release.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
--
Sincerely yours
Egor Pakhomov, AnchorFree
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Luc Bourlier <lu...@typesafe.com>.
+1 (non binding)
Tested the integration with Mesos in the different configurations.
Luc
Le jeu. 5 nov. 2015 à 21:02, Nicholas Chammas <ni...@gmail.com>
a écrit :
> -0
>
> The spark-ec2 version is still set to 1.5.1
> <https://github.com/apache/spark/blob/v1.5.2-rc2/ec2/spark_ec2.py#L54>.
>
> Nick
>
> On Wed, Nov 4, 2015 at 8:20 PM Egor Pahomov <pa...@gmail.com>
> wrote:
>
>> +1
>>
>> Things, which our infrastructure use and I checked:
>>
>> Dynamic allocation
>> Spark ODBC server
>> Reading json
>> Writing parquet
>> SQL quires (hive context)
>> Running on CDH
>>
>>
>> 2015-11-04 9:03 GMT-08:00 Sean Owen <so...@cloudera.com>:
>>
>>> As usual the signatures and licenses and so on look fine. I continue
>>> to get the same test failures on Ubuntu in Java 7/8:
>>>
>>> - Unpersisting HttpBroadcast on executors only in distributed mode ***
>>> FAILED ***
>>>
>>> But I continue to assume that's specific to tests and/or Ubuntu and/or
>>> the build profile, since I don't see any evidence of this in other
>>> builds on Jenkins. It's not a change from previous behavior, though it
>>> doesn't always happen either.
>>>
>>> On Tue, Nov 3, 2015 at 11:22 PM, Reynold Xin <rx...@databricks.com>
>>> wrote:
>>> > Please vote on releasing the following candidate as Apache Spark
>>> version
>>> > 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes
>>> if a
>>> > majority of at least 3 +1 PMC votes are cast.
>>> >
>>> > [ ] +1 Release this package as Apache Spark 1.5.2
>>> > [ ] -1 Do not release this package because ...
>>> >
>>> >
>>> > The release fixes 59 known issues in Spark 1.5.1, listed here:
>>> > http://s.apache.org/spark-1.5.2
>>> >
>>> > The tag to be voted on is v1.5.2-rc2:
>>> > https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>> >
>>> > The release files, including signatures, digests, etc. can be found at:
>>> > http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>> >
>>> > Release artifacts are signed with the following key:
>>> > https://people.apache.org/keys/committer/pwendell.asc
>>> >
>>> > The staging repository for this release can be found at:
>>> > - as version 1.5.2-rc2:
>>> > https://repository.apache.org/content/repositories/orgapachespark-1153
>>> > - as version 1.5.2:
>>> > https://repository.apache.org/content/repositories/orgapachespark-1152
>>> >
>>> > The documentation corresponding to this release can be found at:
>>> >
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>> >
>>> >
>>> > =======================================
>>> > How can I help test this release?
>>> > =======================================
>>> > If you are a Spark user, you can help us test this release by taking an
>>> > existing Spark workload and running on this release candidate, then
>>> > reporting any regressions.
>>> >
>>> > ================================================
>>> > What justifies a -1 vote for this release?
>>> > ================================================
>>> > -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>>> present
>>> > in 1.5.1 will not block this release.
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>
>>
>> --
>>
>> *Sincerely yoursEgor Pakhomov, *
>>
>> *AnchorFree*
>>
>>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Nicholas Chammas <ni...@gmail.com>.
-0
The spark-ec2 version is still set to 1.5.1
<https://github.com/apache/spark/blob/v1.5.2-rc2/ec2/spark_ec2.py#L54>.
Nick
On Wed, Nov 4, 2015 at 8:20 PM Egor Pahomov <pa...@gmail.com> wrote:
> +1
>
> Things, which our infrastructure use and I checked:
>
> Dynamic allocation
> Spark ODBC server
> Reading json
> Writing parquet
> SQL quires (hive context)
> Running on CDH
>
>
> 2015-11-04 9:03 GMT-08:00 Sean Owen <so...@cloudera.com>:
>
>> As usual the signatures and licenses and so on look fine. I continue
>> to get the same test failures on Ubuntu in Java 7/8:
>>
>> - Unpersisting HttpBroadcast on executors only in distributed mode ***
>> FAILED ***
>>
>> But I continue to assume that's specific to tests and/or Ubuntu and/or
>> the build profile, since I don't see any evidence of this in other
>> builds on Jenkins. It's not a change from previous behavior, though it
>> doesn't always happen either.
>>
>> On Tue, Nov 3, 2015 at 11:22 PM, Reynold Xin <rx...@databricks.com> wrote:
>> > Please vote on releasing the following candidate as Apache Spark version
>> > 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes
>> if a
>> > majority of at least 3 +1 PMC votes are cast.
>> >
>> > [ ] +1 Release this package as Apache Spark 1.5.2
>> > [ ] -1 Do not release this package because ...
>> >
>> >
>> > The release fixes 59 known issues in Spark 1.5.1, listed here:
>> > http://s.apache.org/spark-1.5.2
>> >
>> > The tag to be voted on is v1.5.2-rc2:
>> > https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>> >
>> > The release files, including signatures, digests, etc. can be found at:
>> > http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>> >
>> > Release artifacts are signed with the following key:
>> > https://people.apache.org/keys/committer/pwendell.asc
>> >
>> > The staging repository for this release can be found at:
>> > - as version 1.5.2-rc2:
>> > https://repository.apache.org/content/repositories/orgapachespark-1153
>> > - as version 1.5.2:
>> > https://repository.apache.org/content/repositories/orgapachespark-1152
>> >
>> > The documentation corresponding to this release can be found at:
>> > http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>> >
>> >
>> > =======================================
>> > How can I help test this release?
>> > =======================================
>> > If you are a Spark user, you can help us test this release by taking an
>> > existing Spark workload and running on this release candidate, then
>> > reporting any regressions.
>> >
>> > ================================================
>> > What justifies a -1 vote for this release?
>> > ================================================
>> > -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>> present
>> > in 1.5.1 will not block this release.
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
>
> --
>
> *Sincerely yoursEgor Pakhomov, *
>
> *AnchorFree*
>
>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Egor Pahomov <pa...@gmail.com>.
+1
Things, which our infrastructure use and I checked:
Dynamic allocation
Spark ODBC server
Reading json
Writing parquet
SQL quires (hive context)
Running on CDH
2015-11-04 9:03 GMT-08:00 Sean Owen <so...@cloudera.com>:
> As usual the signatures and licenses and so on look fine. I continue
> to get the same test failures on Ubuntu in Java 7/8:
>
> - Unpersisting HttpBroadcast on executors only in distributed mode ***
> FAILED ***
>
> But I continue to assume that's specific to tests and/or Ubuntu and/or
> the build profile, since I don't see any evidence of this in other
> builds on Jenkins. It's not a change from previous behavior, though it
> doesn't always happen either.
>
> On Tue, Nov 3, 2015 at 11:22 PM, Reynold Xin <rx...@databricks.com> wrote:
> > Please vote on releasing the following candidate as Apache Spark version
> > 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if
> a
> > majority of at least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Spark 1.5.2
> > [ ] -1 Do not release this package because ...
> >
> >
> > The release fixes 59 known issues in Spark 1.5.1, listed here:
> > http://s.apache.org/spark-1.5.2
> >
> > The tag to be voted on is v1.5.2-rc2:
> > https://github.com/apache/spark/releases/tag/v1.5.2-rc2
> >
> > The release files, including signatures, digests, etc. can be found at:
> > http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/pwendell.asc
> >
> > The staging repository for this release can be found at:
> > - as version 1.5.2-rc2:
> > https://repository.apache.org/content/repositories/orgapachespark-1153
> > - as version 1.5.2:
> > https://repository.apache.org/content/repositories/orgapachespark-1152
> >
> > The documentation corresponding to this release can be found at:
> > http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
> >
> >
> > =======================================
> > How can I help test this release?
> > =======================================
> > If you are a Spark user, you can help us test this release by taking an
> > existing Spark workload and running on this release candidate, then
> > reporting any regressions.
> >
> > ================================================
> > What justifies a -1 vote for this release?
> > ================================================
> > -1 vote should occur for regressions from Spark 1.5.1. Bugs already
> present
> > in 1.5.1 will not block this release.
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
--
*Sincerely yoursEgor Pakhomov, *
*AnchorFree*
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Sean Owen <so...@cloudera.com>.
As usual the signatures and licenses and so on look fine. I continue
to get the same test failures on Ubuntu in Java 7/8:
- Unpersisting HttpBroadcast on executors only in distributed mode ***
FAILED ***
But I continue to assume that's specific to tests and/or Ubuntu and/or
the build profile, since I don't see any evidence of this in other
builds on Jenkins. It's not a change from previous behavior, though it
doesn't always happen either.
On Tue, Nov 3, 2015 at 11:22 PM, Reynold Xin <rx...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> =======================================
> How can I help test this release?
> =======================================
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> ================================================
> What justifies a -1 vote for this release?
> ================================================
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already present
> in 1.5.1 will not block this release.
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Krishna Sankar <ks...@gmail.com>.
+1 (non-binding, of course) (Hope I made it in time. ~T-20 !)
1. Compiled OSX 10.10 (Yosemite) OK Total time: 25:52 min
mvn clean package -Pyarn -Phadoop-2.6 -DskipTests
2. Tested pyspark, mllib (iPython 4.0, FYI, notebook install is separate
“conda install ipython” and then “conda install jupyter”)
2.0 Spark version is 1.5.2.
2.1. statistics (min,max,mean,Pearson,Spearman) OK
2.2. Linear/Ridge/Laso Regression OK
2.3. Decision Tree, Naive Bayes OK
2.4. KMeans OK
Center And Scale OK
2.5. RDD operations OK
State of the Union Texts - MapReduce, Filter,sortByKey (word count)
2.6. Recommendation (Movielens medium dataset ~1 M ratings) OK
Model evaluation/optimization (rank, numIter, lambda) with itertools
OK
3. Scala - MLlib
3.1. statistics (min,max,mean,Pearson,Spearman) OK
3.2. LinearRegressionWithSGD OK
3.3. Decision Tree OK
3.4. KMeans OK
3.5. Recommendation (Movielens medium dataset ~1 M ratings) OK
3.6. saveAsParquetFile OK
3.7. Read and verify the 4.3 save(above) - sqlContext.parquetFile,
registerTempTable, sql OK
3.8. result = sqlContext.sql("SELECT
OrderDetails.OrderID,ShipCountry,UnitPrice,Qty,Discount FROM Orders INNER
JOIN OrderDetails ON Orders.OrderID = OrderDetails.OrderID") OK
4.0. Spark SQL from Python OK
4.1. result = sqlContext.sql("SELECT * from people WHERE State = 'WA'") OK
5.0. Packages
5.1. com.databricks.spark.csv - read/write OK (--packages
com.databricks:spark-csv_2.10:1.2.0)
6.0. DataFrames
6.1. cast,dtypes OK
6.2. groupBy,avg,crosstab,corr,isNull,na.drop OK
6.3. All joins,sql,set operations,udf OK
Cheers
<k/>
On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> =======================================
> How can I help test this release?
> =======================================
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> ================================================
> What justifies a -1 vote for this release?
> ================================================
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
> present in 1.5.1 will not block this release.
>
>
>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Ted Yu <yu...@gmail.com>.
+1
On Sat, Nov 7, 2015 at 4:35 PM, Denny Lee <de...@gmail.com> wrote:
> +1
>
>
> On Sat, Nov 7, 2015 at 12:01 PM Mark Hamstra <ma...@clearstorydata.com>
> wrote:
>
>> +1
>>
>> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>>> majority of at least 3 +1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 1.5.2
>>> [ ] -1 Do not release this package because ...
>>>
>>>
>>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>>> http://s.apache.org/spark-1.5.2
>>>
>>> The tag to be voted on is v1.5.2-rc2:
>>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The staging repository for this release can be found at:
>>> - as version 1.5.2-rc2:
>>> https://repository.apache.org/content/repositories/orgapachespark-1153
>>> - as version 1.5.2:
>>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>>
>>>
>>> =======================================
>>> How can I help test this release?
>>> =======================================
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> ================================================
>>> What justifies a -1 vote for this release?
>>> ================================================
>>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>>> present in 1.5.1 will not block this release.
>>>
>>>
>>>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Denny Lee <de...@gmail.com>.
+1
On Sat, Nov 7, 2015 at 12:01 PM Mark Hamstra <ma...@clearstorydata.com>
wrote:
> +1
>
> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 1.5.2
>> [ ] -1 Do not release this package because ...
>>
>>
>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>> http://s.apache.org/spark-1.5.2
>>
>> The tag to be voted on is v1.5.2-rc2:
>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The staging repository for this release can be found at:
>> - as version 1.5.2-rc2:
>> https://repository.apache.org/content/repositories/orgapachespark-1153
>> - as version 1.5.2:
>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>
>>
>> =======================================
>> How can I help test this release?
>> =======================================
>> If you are a Spark user, you can help us test this release by taking an
>> existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> ================================================
>> What justifies a -1 vote for this release?
>> ================================================
>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>> present in 1.5.1 will not block this release.
>>
>>
>>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Mark Hamstra <ma...@clearstorydata.com>.
+1
On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> =======================================
> How can I help test this release?
> =======================================
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> ================================================
> What justifies a -1 vote for this release?
> ================================================
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
> present in 1.5.1 will not block this release.
>
>
>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 (non binding)
Just tested with some snippets on my side.
Regards
JB
On 11/04/2015 12:22 AM, Reynold Xin wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if
> a majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> =======================================
> How can I help test this release?
> =======================================
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> ================================================
> What justifies a -1 vote for this release?
> ================================================
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
> present in 1.5.1 will not block this release.
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Sean McNamara <Se...@Webtrends.com>.
+1
Sean
On Nov 3, 2015, at 4:28 PM, Reynold Xin <rx...@databricks.com>> wrote:
Please vote on releasing the following candidate as Apache Spark version 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a majority of at least 3 +1 PMC votes are cast.
[ ] +1 Release this package as Apache Spark 1.5.2
[ ] -1 Do not release this package because ...
The release fixes 59 known issues in Spark 1.5.1, listed here:
http://s.apache.org/spark-1.5.2
The tag to be voted on is v1.5.2-rc2:
https://github.com/apache/spark/releases/tag/v1.5.2-rc2
The release files, including signatures, digests, etc. can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc
The staging repository for this release can be found at:
- as version 1.5.2-rc2: https://repository.apache.org/content/repositories/orgapachespark-1153
- as version 1.5.2: https://repository.apache.org/content/repositories/orgapachespark-1152
The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
=======================================
How can I help test this release?
=======================================
If you are a Spark user, you can help us test this release by taking an existing Spark workload and running on this release candidate, then reporting any regressions.
================================================
What justifies a -1 vote for this release?
================================================
-1 vote should occur for regressions from Spark 1.5.1. Bugs already present in 1.5.1 will not block this release.
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Michael Armbrust <mi...@databricks.com>.
+1
On Fri, Nov 6, 2015 at 9:27 AM, Chester Chen <ch...@alpinenow.com> wrote:
> +1
> Test against CDH5.4.2 with hadoop 2.6.0 version using yesterday's code,
> build locally.
>
> Regression running in Yarn Cluster mode against few internal ML ( logistic
> regression, linear regression, random forest and statistic summary) as well
> Mlib KMeans. all seems to work fine.
>
> Chester
>
>
> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 1.5.2
>> [ ] -1 Do not release this package because ...
>>
>>
>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>> http://s.apache.org/spark-1.5.2
>>
>> The tag to be voted on is v1.5.2-rc2:
>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The staging repository for this release can be found at:
>> - as version 1.5.2-rc2:
>> https://repository.apache.org/content/repositories/orgapachespark-1153
>> - as version 1.5.2:
>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>
>>
>> =======================================
>> How can I help test this release?
>> =======================================
>> If you are a Spark user, you can help us test this release by taking an
>> existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> ================================================
>> What justifies a -1 vote for this release?
>> ================================================
>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>> present in 1.5.1 will not block this release.
>>
>>
>>
>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Joseph Bradley <jo...@databricks.com>.
+1 tested on OS X
On Sat, Nov 7, 2015 at 10:25 AM, Reynold Xin <rx...@databricks.com> wrote:
> +1 myself too
>
> On Sat, Nov 7, 2015 at 12:01 AM, Robin East <ro...@xense.co.uk>
> wrote:
>
>> +1
>> Mac OS X 10.10.5 Yosemite
>>
>> mvn clean package -DskipTests (13min)
>>
>> Basic graph tests
>> Load graph using edgeListFile...SUCCESS
>> Run PageRank...SUCCESS
>> Connected Components tests
>> Kaggle social circles competition...SUCCESS
>> Minimum Spanning Tree Algorithm
>> Run basic Minimum Spanning Tree algorithmSUCCESS
>> Run Minimum Spanning Tree taxonomy creation...SUCCESS
>>
>>
>> -------------------------------------------------------------------------------
>> Robin East
>> *Spark GraphX in Action* Michael Malak and Robin East
>> Manning Publications Co.
>> http://www.manning.com/books/spark-graphx-in-action
>>
>>
>>
>>
>>
>> On 6 Nov 2015, at 17:27, Chester Chen <ch...@alpinenow.com> wrote:
>>
>> +1
>> Test against CDH5.4.2 with hadoop 2.6.0 version using yesterday's code,
>> build locally.
>>
>> Regression running in Yarn Cluster mode against few internal ML (
>> logistic regression, linear regression, random forest and statistic
>> summary) as well Mlib KMeans. all seems to work fine.
>>
>> Chester
>>
>>
>> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark version
>>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>>> majority of at least 3 +1 PMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 1.5.2
>>> [ ] -1 Do not release this package because ...
>>>
>>>
>>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>>> http://s.apache.org/spark-1.5.2
>>>
>>> The tag to be voted on is v1.5.2-rc2:
>>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>>
>>> The release files, including signatures, digests, etc. can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The staging repository for this release can be found at:
>>> - as version 1.5.2-rc2:
>>> https://repository.apache.org/content/repositories/orgapachespark-1153
>>> - as version 1.5.2:
>>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>>
>>>
>>> =======================================
>>> How can I help test this release?
>>> =======================================
>>> If you are a Spark user, you can help us test this release by taking an
>>> existing Spark workload and running on this release candidate, then
>>> reporting any regressions.
>>>
>>> ================================================
>>> What justifies a -1 vote for this release?
>>> ================================================
>>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>>> present in 1.5.1 will not block this release.
>>>
>>>
>>>
>>
>>
>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Reynold Xin <rx...@databricks.com>.
+1 myself too
On Sat, Nov 7, 2015 at 12:01 AM, Robin East <ro...@xense.co.uk> wrote:
> +1
> Mac OS X 10.10.5 Yosemite
>
> mvn clean package -DskipTests (13min)
>
> Basic graph tests
> Load graph using edgeListFile...SUCCESS
> Run PageRank...SUCCESS
> Connected Components tests
> Kaggle social circles competition...SUCCESS
> Minimum Spanning Tree Algorithm
> Run basic Minimum Spanning Tree algorithmSUCCESS
> Run Minimum Spanning Tree taxonomy creation...SUCCESS
>
>
> -------------------------------------------------------------------------------
> Robin East
> *Spark GraphX in Action* Michael Malak and Robin East
> Manning Publications Co.
> http://www.manning.com/books/spark-graphx-in-action
>
>
>
>
>
> On 6 Nov 2015, at 17:27, Chester Chen <ch...@alpinenow.com> wrote:
>
> +1
> Test against CDH5.4.2 with hadoop 2.6.0 version using yesterday's code,
> build locally.
>
> Regression running in Yarn Cluster mode against few internal ML ( logistic
> regression, linear regression, random forest and statistic summary) as well
> Mlib KMeans. all seems to work fine.
>
> Chester
>
>
> On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
>
>> Please vote on releasing the following candidate as Apache Spark version
>> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Spark 1.5.2
>> [ ] -1 Do not release this package because ...
>>
>>
>> The release fixes 59 known issues in Spark 1.5.1, listed here:
>> http://s.apache.org/spark-1.5.2
>>
>> The tag to be voted on is v1.5.2-rc2:
>> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>>
>> The release files, including signatures, digests, etc. can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>>
>> The staging repository for this release can be found at:
>> - as version 1.5.2-rc2:
>> https://repository.apache.org/content/repositories/orgapachespark-1153
>> - as version 1.5.2:
>> https://repository.apache.org/content/repositories/orgapachespark-1152
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>>
>>
>> =======================================
>> How can I help test this release?
>> =======================================
>> If you are a Spark user, you can help us test this release by taking an
>> existing Spark workload and running on this release candidate, then
>> reporting any regressions.
>>
>> ================================================
>> What justifies a -1 vote for this release?
>> ================================================
>> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
>> present in 1.5.1 will not block this release.
>>
>>
>>
>
>
Re: [VOTE] Release Apache Spark 1.5.2 (RC2)
Posted by Chester Chen <ch...@alpinenow.com>.
+1
Test against CDH5.4.2 with hadoop 2.6.0 version using yesterday's code,
build locally.
Regression running in Yarn Cluster mode against few internal ML ( logistic
regression, linear regression, random forest and statistic summary) as well
Mlib KMeans. all seems to work fine.
Chester
On Tue, Nov 3, 2015 at 3:22 PM, Reynold Xin <rx...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.5.2. The vote is open until Sat Nov 7, 2015 at 00:00 UTC and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.5.2
> [ ] -1 Do not release this package because ...
>
>
> The release fixes 59 known issues in Spark 1.5.1, listed here:
> http://s.apache.org/spark-1.5.2
>
> The tag to be voted on is v1.5.2-rc2:
> https://github.com/apache/spark/releases/tag/v1.5.2-rc2
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-bin/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> - as version 1.5.2-rc2:
> https://repository.apache.org/content/repositories/orgapachespark-1153
> - as version 1.5.2:
> https://repository.apache.org/content/repositories/orgapachespark-1152
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-releases/spark-1.5.2-rc2-docs/
>
>
> =======================================
> How can I help test this release?
> =======================================
> If you are a Spark user, you can help us test this release by taking an
> existing Spark workload and running on this release candidate, then
> reporting any regressions.
>
> ================================================
> What justifies a -1 vote for this release?
> ================================================
> -1 vote should occur for regressions from Spark 1.5.1. Bugs already
> present in 1.5.1 will not block this release.
>
>
>