You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Andrew Or <an...@databricks.com> on 2014/11/13 05:34:47 UTC
[VOTE] Release Apache Spark 1.1.1 (RC1)
Please vote on releasing the following candidate as Apache Spark version 1
.1.1.
This release fixes a number of bugs in Spark 1.1.0. Some of the notable
ones are
- [SPARK-3426] Sort-based shuffle compression settings are incompatible
- [SPARK-3948] Stream corruption issues in sort-based shuffle
- [SPARK-4107] Incorrect handling of Channel.read() led to data truncation
The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached.
The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
http://s.apache.org/cZC
The release files, including signatures, digests, etc can be found at:
http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/andrewor14.asc
The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1034/
The documentation corresponding to this release can be found at:
http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
Please vote on releasing this package as Apache Spark 1.1.1!
The vote is open until Sunday, November 16, at 04:30 UTC and passes if
a majority of at least 3 +1 PMC votes are cast.
[ ] +1 Release this package as Apache Spark 1.1.1
[ ] -1 Do not release this package because ...
To learn more about Apache Spark, please see
http://spark.apache.org/
Cheers,
Andrew
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Kevin,
If you are upgrading from 1.0.X to 1.1.X checkout the upgrade notes
here [1] - it could be that default changes caused a regression for
your workload. Do you still see a regression if you restore the
configuration changes?
It's great to hear specifically about issues like this, so please fork
a new thread and describe your workload if you see a regression. The
main focus of a patch release vote like this is to test regressions
against the previous release on the same line (e.g. 1.1.1 vs 1.1.0)
though of course we still want to be cognizant of 1.0-to-1.1
regressions and make sure we can address them down the road.
[1] https://spark.apache.org/releases/spark-release-1-1-0.html
On Mon, Nov 17, 2014 at 2:04 PM, Kevin Markey <ke...@oracle.com> wrote:
> +0 (non-binding)
>
> Compiled Spark, recompiled and ran application with 1.1.1 RC1 with Yarn,
> plain-vanilla Hadoop 2.3.0. No regressions.
>
> However, 12% to 22% increase in run time relative to 1.0.0 release. (No
> other environment or configuration changes.) Would have recommended +1 were
> it not for added latency.
>
> Not sure if added latency a function of 1.0 vs 1.1 or 1.0 vs 1.1.1 changes,
> as we've never tested with 1.1.0. But thought I'd share the results. (This
> is somewhat disappointing.)
>
> Kevin Markey
>
>
> On 11/17/2014 11:42 AM, Debasish Das wrote:
>>
>> Andrew,
>>
>> I put up 1.1.1 branch and I am getting shuffle failures while doing
>> flatMap
>> followed by groupBy...My cluster memory is less than the memory I need and
>> therefore flatMap does around 400 GB of shuffle...memory is around 120
>> GB...
>>
>> 14/11/13 23:10:49 WARN TaskSetManager: Lost task 22.1 in stage 191.0 (TID
>> 4084, istgbd020.hadoop.istg.verizon.com): FetchFailed(null, shuffleId=4,
>> mapId=-1, reduceId=22)
>>
>> I searched on user-list and this issue has been found over there:
>>
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/Issues-with-partitionBy-FetchFailed-td14760.html
>>
>> I wanted to make sure whether 1.1.1 does not have the same bug...-1 from
>> me
>> till we figure out the root cause...
>>
>> Thanks.
>>
>> Deb
>>
>> On Mon, Nov 17, 2014 at 10:33 AM, Andrew Or <an...@databricks.com> wrote:
>>
>>> This seems like a legitimate blocker. We will cut another RC to include
>>> the
>>> revert.
>>>
>>> 2014-11-16 17:29 GMT-08:00 Kousuke Saruta <sa...@oss.nttdata.co.jp>:
>>>
>>>> Now I've finished to revert for SPARK-4434 and opened PR.
>>>>
>>>>
>>>> (2014/11/16 17:08), Josh Rosen wrote:
>>>>
>>>>> -1
>>>>>
>>>>> I found a potential regression in 1.1.1 related to spark-submit and
>>>>> cluster
>>>>> deploy mode: https://issues.apache.org/jira/browse/SPARK-4434
>>>>>
>>>>> I think that this is worth fixing.
>>>>>
>>>>> On Fri, Nov 14, 2014 at 7:28 PM, Cheng Lian <li...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> +1
>>>>>>
>>>>>>
>>>>>> Tested HiveThriftServer2 against Hive 0.12.0 on Mac OS X. Known issues
>>>>>> are
>>>>>> fixed. Hive version inspection works as expected.
>>>>>>
>>>>>>
>>>>>> On 11/15/14 8:25 AM, Zach Fry wrote:
>>>>>>
>>>>>> +0
>>>>>>>
>>>>>>>
>>>>>>> I expect to start testing on Monday but won't have enough results to
>>>>>>> change
>>>>>>> my vote from +0
>>>>>>> until Monday night or Tuesday morning.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Zach
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context: http://apache-spark-
>>>>>>> developers-list.1001551.n3.nabble.com/VOTE-Release-
>>>>>>> Apache-Spark-1-1-1-RC1-tp9311p9370.html
>>>>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>>>>> Nabble.com.
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>
>>>>
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Kevin Markey <ke...@oracle.com>.
+0 (non-binding)
Compiled Spark, recompiled and ran application with 1.1.1 RC1 with Yarn,
plain-vanilla Hadoop 2.3.0. No regressions.
However, 12% to 22% increase in run time relative to 1.0.0 release. (No
other environment or configuration changes.) Would have recommended +1
were it not for added latency.
Not sure if added latency a function of 1.0 vs 1.1 or 1.0 vs 1.1.1
changes, as we've never tested with 1.1.0. But thought I'd share the
results. (This is somewhat disappointing.)
Kevin Markey
On 11/17/2014 11:42 AM, Debasish Das wrote:
> Andrew,
>
> I put up 1.1.1 branch and I am getting shuffle failures while doing flatMap
> followed by groupBy...My cluster memory is less than the memory I need and
> therefore flatMap does around 400 GB of shuffle...memory is around 120 GB...
>
> 14/11/13 23:10:49 WARN TaskSetManager: Lost task 22.1 in stage 191.0 (TID
> 4084, istgbd020.hadoop.istg.verizon.com): FetchFailed(null, shuffleId=4,
> mapId=-1, reduceId=22)
>
> I searched on user-list and this issue has been found over there:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Issues-with-partitionBy-FetchFailed-td14760.html
>
> I wanted to make sure whether 1.1.1 does not have the same bug...-1 from me
> till we figure out the root cause...
>
> Thanks.
>
> Deb
>
> On Mon, Nov 17, 2014 at 10:33 AM, Andrew Or <an...@databricks.com> wrote:
>
>> This seems like a legitimate blocker. We will cut another RC to include the
>> revert.
>>
>> 2014-11-16 17:29 GMT-08:00 Kousuke Saruta <sa...@oss.nttdata.co.jp>:
>>
>>> Now I've finished to revert for SPARK-4434 and opened PR.
>>>
>>>
>>> (2014/11/16 17:08), Josh Rosen wrote:
>>>
>>>> -1
>>>>
>>>> I found a potential regression in 1.1.1 related to spark-submit and
>>>> cluster
>>>> deploy mode: https://issues.apache.org/jira/browse/SPARK-4434
>>>>
>>>> I think that this is worth fixing.
>>>>
>>>> On Fri, Nov 14, 2014 at 7:28 PM, Cheng Lian <li...@gmail.com>
>>>> wrote:
>>>>
>>>> +1
>>>>>
>>>>> Tested HiveThriftServer2 against Hive 0.12.0 on Mac OS X. Known issues
>>>>> are
>>>>> fixed. Hive version inspection works as expected.
>>>>>
>>>>>
>>>>> On 11/15/14 8:25 AM, Zach Fry wrote:
>>>>>
>>>>> +0
>>>>>>
>>>>>> I expect to start testing on Monday but won't have enough results to
>>>>>> change
>>>>>> my vote from +0
>>>>>> until Monday night or Tuesday morning.
>>>>>>
>>>>>> Thanks,
>>>>>> Zach
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context: http://apache-spark-
>>>>>> developers-list.1001551.n3.nabble.com/VOTE-Release-
>>>>>> Apache-Spark-1-1-1-RC1-tp9311p9370.html
>>>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>>>> Nabble.com.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Debasish Das <de...@gmail.com>.
Andrew,
I put up 1.1.1 branch and I am getting shuffle failures while doing flatMap
followed by groupBy...My cluster memory is less than the memory I need and
therefore flatMap does around 400 GB of shuffle...memory is around 120 GB...
14/11/13 23:10:49 WARN TaskSetManager: Lost task 22.1 in stage 191.0 (TID
4084, istgbd020.hadoop.istg.verizon.com): FetchFailed(null, shuffleId=4,
mapId=-1, reduceId=22)
I searched on user-list and this issue has been found over there:
http://apache-spark-user-list.1001560.n3.nabble.com/Issues-with-partitionBy-FetchFailed-td14760.html
I wanted to make sure whether 1.1.1 does not have the same bug...-1 from me
till we figure out the root cause...
Thanks.
Deb
On Mon, Nov 17, 2014 at 10:33 AM, Andrew Or <an...@databricks.com> wrote:
> This seems like a legitimate blocker. We will cut another RC to include the
> revert.
>
> 2014-11-16 17:29 GMT-08:00 Kousuke Saruta <sa...@oss.nttdata.co.jp>:
>
> > Now I've finished to revert for SPARK-4434 and opened PR.
> >
> >
> > (2014/11/16 17:08), Josh Rosen wrote:
> >
> >> -1
> >>
> >> I found a potential regression in 1.1.1 related to spark-submit and
> >> cluster
> >> deploy mode: https://issues.apache.org/jira/browse/SPARK-4434
> >>
> >> I think that this is worth fixing.
> >>
> >> On Fri, Nov 14, 2014 at 7:28 PM, Cheng Lian <li...@gmail.com>
> >> wrote:
> >>
> >> +1
> >>>
> >>> Tested HiveThriftServer2 against Hive 0.12.0 on Mac OS X. Known issues
> >>> are
> >>> fixed. Hive version inspection works as expected.
> >>>
> >>>
> >>> On 11/15/14 8:25 AM, Zach Fry wrote:
> >>>
> >>> +0
> >>>>
> >>>> I expect to start testing on Monday but won't have enough results to
> >>>> change
> >>>> my vote from +0
> >>>> until Monday night or Tuesday morning.
> >>>>
> >>>> Thanks,
> >>>> Zach
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> View this message in context: http://apache-spark-
> >>>> developers-list.1001551.n3.nabble.com/VOTE-Release-
> >>>> Apache-Spark-1-1-1-RC1-tp9311p9370.html
> >>>> Sent from the Apache Spark Developers List mailing list archive at
> >>>> Nabble.com.
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>
> >>>
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
> >
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Andrew Or <an...@databricks.com>.
This seems like a legitimate blocker. We will cut another RC to include the
revert.
2014-11-16 17:29 GMT-08:00 Kousuke Saruta <sa...@oss.nttdata.co.jp>:
> Now I've finished to revert for SPARK-4434 and opened PR.
>
>
> (2014/11/16 17:08), Josh Rosen wrote:
>
>> -1
>>
>> I found a potential regression in 1.1.1 related to spark-submit and
>> cluster
>> deploy mode: https://issues.apache.org/jira/browse/SPARK-4434
>>
>> I think that this is worth fixing.
>>
>> On Fri, Nov 14, 2014 at 7:28 PM, Cheng Lian <li...@gmail.com>
>> wrote:
>>
>> +1
>>>
>>> Tested HiveThriftServer2 against Hive 0.12.0 on Mac OS X. Known issues
>>> are
>>> fixed. Hive version inspection works as expected.
>>>
>>>
>>> On 11/15/14 8:25 AM, Zach Fry wrote:
>>>
>>> +0
>>>>
>>>> I expect to start testing on Monday but won't have enough results to
>>>> change
>>>> my vote from +0
>>>> until Monday night or Tuesday morning.
>>>>
>>>> Thanks,
>>>> Zach
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-
>>>> developers-list.1001551.n3.nabble.com/VOTE-Release-
>>>> Apache-Spark-1-1-1-RC1-tp9311p9370.html
>>>> Sent from the Apache Spark Developers List mailing list archive at
>>>> Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Kousuke Saruta <sa...@oss.nttdata.co.jp>.
Now I've finished to revert for SPARK-4434 and opened PR.
(2014/11/16 17:08), Josh Rosen wrote:
> -1
>
> I found a potential regression in 1.1.1 related to spark-submit and cluster
> deploy mode: https://issues.apache.org/jira/browse/SPARK-4434
>
> I think that this is worth fixing.
>
> On Fri, Nov 14, 2014 at 7:28 PM, Cheng Lian <li...@gmail.com> wrote:
>
>> +1
>>
>> Tested HiveThriftServer2 against Hive 0.12.0 on Mac OS X. Known issues are
>> fixed. Hive version inspection works as expected.
>>
>>
>> On 11/15/14 8:25 AM, Zach Fry wrote:
>>
>>> +0
>>>
>>> I expect to start testing on Monday but won't have enough results to
>>> change
>>> my vote from +0
>>> until Monday night or Tuesday morning.
>>>
>>> Thanks,
>>> Zach
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-spark-
>>> developers-list.1001551.n3.nabble.com/VOTE-Release-
>>> Apache-Spark-1-1-1-RC1-tp9311p9370.html
>>> Sent from the Apache Spark Developers List mailing list archive at
>>> Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Josh Rosen <ro...@gmail.com>.
-1
I found a potential regression in 1.1.1 related to spark-submit and cluster
deploy mode: https://issues.apache.org/jira/browse/SPARK-4434
I think that this is worth fixing.
On Fri, Nov 14, 2014 at 7:28 PM, Cheng Lian <li...@gmail.com> wrote:
> +1
>
> Tested HiveThriftServer2 against Hive 0.12.0 on Mac OS X. Known issues are
> fixed. Hive version inspection works as expected.
>
>
> On 11/15/14 8:25 AM, Zach Fry wrote:
>
>> +0
>>
>> I expect to start testing on Monday but won't have enough results to
>> change
>> my vote from +0
>> until Monday night or Tuesday morning.
>>
>> Thanks,
>> Zach
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-
>> developers-list.1001551.n3.nabble.com/VOTE-Release-
>> Apache-Spark-1-1-1-RC1-tp9311p9370.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Cheng Lian <li...@gmail.com>.
+1
Tested HiveThriftServer2 against Hive 0.12.0 on Mac OS X. Known issues
are fixed. Hive version inspection works as expected.
On 11/15/14 8:25 AM, Zach Fry wrote:
> +0
>
> I expect to start testing on Monday but won't have enough results to change
> my vote from +0
> until Monday night or Tuesday morning.
>
> Thanks,
> Zach
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-1-1-RC1-tp9311p9370.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Zach Fry <zf...@palantir.com>.
+0
I expect to start testing on Monday but won't have enough results to change
my vote from +0
until Monday night or Tuesday morning.
Thanks,
Zach
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-1-1-RC1-tp9311p9370.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Matei Zaharia <ma...@gmail.com>.
+1
Tested on Mac OS X, and verified that sort-based shuffle bug is fixed.
Matei
> On Nov 14, 2014, at 10:45 AM, Andrew Or <an...@databricks.com> wrote:
>
> Hi all, since the vote ends on a Sunday, please let me know if you would
> like to extend the deadline to allow more time for testing.
>
> 2014-11-13 12:10 GMT-08:00 Sean Owen <so...@cloudera.com>:
>
>> Ah right. This is because I'm running Java 8. This was fixed in
>> SPARK-3329 (
>> https://github.com/apache/spark/commit/2b7ab814f9bde65ebc57ebd04386e56c97f06f4a#diff-7bfd8d7c8cbb02aa0023e4c3497ee832
>> ).
>> Consider back-porting it if other reasons arise, but this is specific
>> to tests and to Java 8.
>>
>> On Thu, Nov 13, 2014 at 8:01 PM, Andrew Or <an...@databricks.com> wrote:
>>> Yeah, this seems to be somewhat environment specific too. The same test
>> has
>>> been passing here for a while:
>>>
>> https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.1-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/lastBuild/consoleFull
>>>
>>> 2014-11-13 11:26 GMT-08:00 Michael Armbrust <mi...@databricks.com>:
>>>
>>>> Hey Sean,
>>>>
>>>> Thanks for pointing this out. Looks like a bad test where we should be
>>>> doing Set comparison instead of Array.
>>>>
>>>> Michael
>>>>
>>>> On Thu, Nov 13, 2014 at 2:05 AM, Sean Owen <so...@cloudera.com> wrote:
>>>>>
>>>>> LICENSE and NOTICE are fine. Signature and checksum is fine. I
>>>>> unzipped and built the plain source distribution, which built.
>>>>>
>>>>> However I am seeing a consistent test failure with "mvn -DskipTests
>>>>> clean package; mvn test". In the Hive module:
>>>>>
>>>>> - SET commands semantics for a HiveContext *** FAILED ***
>>>>> Expected Array("spark.sql.key.usedfortestonly=test.val.0",
>>>>>
>>>>>
>> "spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0"),
>>>>> but got
>>>>>
>> Array("spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0",
>>>>> "spark.sql.key.usedfortestonly=test.val.0") (HiveQuerySuite.scala:544)
>>>>>
>>>>> Anyone else seeing this?
>>>>>
>>>>>
>>>>> On Thu, Nov 13, 2014 at 8:18 AM, Krishna Sankar <ks...@gmail.com>
>>>>> wrote:
>>>>>> +1
>>>>>> 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
>>>>>> -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
>>>>>> 2. Tested pyspark, mlib
>>>>>> 2.1. statistics OK
>>>>>> 2.2. Linear/Ridge/Laso Regression OK
>>>>>> 2.3. Decision Tree, Naive Bayes OK
>>>>>> 2.4. KMeans OK
>>>>>> 2.5. rdd operations OK
>>>>>> 2.6. recommendation OK
>>>>>> 2.7. Good work ! In 1.1.0, there was an error and my program used to
>>>>>> hang
>>>>>> (over memory allocation) consistently running validation using
>>>>>> itertools,
>>>>>> compute optimum rank, lambda,numofiterations/rmse; data - movielens
>>>>>> medium
>>>>>> dataset (1 million records) . It works well in 1.1.1 !
>>>>>> Cheers
>>>>>> <k/>
>>>>>> P.S: Missed Reply all, first time
>>>>>>
>>>>>> On Wed, Nov 12, 2014 at 8:35 PM, Andrew Or <an...@databricks.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I will start the vote with a +1
>>>>>>>
>>>>>>> 2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
>>>>>>>
>>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>>> version
>>>>>>> 1
>>>>>>>> .1.1.
>>>>>>>>
>>>>>>>> This release fixes a number of bugs in Spark 1.1.0. Some of the
>>>>>>>> notable
>>>>>>>> ones are
>>>>>>>> - [SPARK-3426] Sort-based shuffle compression settings are
>>>>>>>> incompatible
>>>>>>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>>>>>>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
>>>>>>> truncation
>>>>>>>> The full list is at http://s.apache.org/z9h and in the
>> CHANGES.txt
>>>>>>>> attached.
>>>>>>>>
>>>>>>>> The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
>>>>>>>> http://s.apache.org/cZC
>>>>>>>>
>>>>>>>> The release files, including signatures, digests, etc can be found
>>>>>>>> at:
>>>>>>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
>>>>>>>>
>>>>>>>> Release artifacts are signed with the following key:
>>>>>>>> https://people.apache.org/keys/committer/andrewor14.asc
>>>>>>>>
>>>>>>>> The staging repository for this release can be found at:
>>>>>>>>
>>>>>>>>
>> https://repository.apache.org/content/repositories/orgapachespark-1034/
>>>>>>>>
>>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
>>>>>>>>
>>>>>>>> Please vote on releasing this package as Apache Spark 1.1.1!
>>>>>>>>
>>>>>>>> The vote is open until Sunday, November 16, at 04:30 UTC and
>> passes
>>>>>>>> if
>>>>>>>> a majority of at least 3 +1 PMC votes are cast.
>>>>>>>> [ ] +1 Release this package as Apache Spark 1.1.1
>>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>>
>>>>>>>> To learn more about Apache Spark, please see
>>>>>>>> http://spark.apache.org/
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Andrew
>>>>>>>>
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>>>
>>>>
>>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Andrew Or <an...@databricks.com>.
Hi all, since the vote ends on a Sunday, please let me know if you would
like to extend the deadline to allow more time for testing.
2014-11-13 12:10 GMT-08:00 Sean Owen <so...@cloudera.com>:
> Ah right. This is because I'm running Java 8. This was fixed in
> SPARK-3329 (
> https://github.com/apache/spark/commit/2b7ab814f9bde65ebc57ebd04386e56c97f06f4a#diff-7bfd8d7c8cbb02aa0023e4c3497ee832
> ).
> Consider back-porting it if other reasons arise, but this is specific
> to tests and to Java 8.
>
> On Thu, Nov 13, 2014 at 8:01 PM, Andrew Or <an...@databricks.com> wrote:
> > Yeah, this seems to be somewhat environment specific too. The same test
> has
> > been passing here for a while:
> >
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.1-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/lastBuild/consoleFull
> >
> > 2014-11-13 11:26 GMT-08:00 Michael Armbrust <mi...@databricks.com>:
> >
> >> Hey Sean,
> >>
> >> Thanks for pointing this out. Looks like a bad test where we should be
> >> doing Set comparison instead of Array.
> >>
> >> Michael
> >>
> >> On Thu, Nov 13, 2014 at 2:05 AM, Sean Owen <so...@cloudera.com> wrote:
> >>>
> >>> LICENSE and NOTICE are fine. Signature and checksum is fine. I
> >>> unzipped and built the plain source distribution, which built.
> >>>
> >>> However I am seeing a consistent test failure with "mvn -DskipTests
> >>> clean package; mvn test". In the Hive module:
> >>>
> >>> - SET commands semantics for a HiveContext *** FAILED ***
> >>> Expected Array("spark.sql.key.usedfortestonly=test.val.0",
> >>>
> >>>
> "spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0"),
> >>> but got
> >>>
> Array("spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0",
> >>> "spark.sql.key.usedfortestonly=test.val.0") (HiveQuerySuite.scala:544)
> >>>
> >>> Anyone else seeing this?
> >>>
> >>>
> >>> On Thu, Nov 13, 2014 at 8:18 AM, Krishna Sankar <ks...@gmail.com>
> >>> wrote:
> >>> > +1
> >>> > 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
> >>> > -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
> >>> > 2. Tested pyspark, mlib
> >>> > 2.1. statistics OK
> >>> > 2.2. Linear/Ridge/Laso Regression OK
> >>> > 2.3. Decision Tree, Naive Bayes OK
> >>> > 2.4. KMeans OK
> >>> > 2.5. rdd operations OK
> >>> > 2.6. recommendation OK
> >>> > 2.7. Good work ! In 1.1.0, there was an error and my program used to
> >>> > hang
> >>> > (over memory allocation) consistently running validation using
> >>> > itertools,
> >>> > compute optimum rank, lambda,numofiterations/rmse; data - movielens
> >>> > medium
> >>> > dataset (1 million records) . It works well in 1.1.1 !
> >>> > Cheers
> >>> > <k/>
> >>> > P.S: Missed Reply all, first time
> >>> >
> >>> > On Wed, Nov 12, 2014 at 8:35 PM, Andrew Or <an...@databricks.com>
> >>> > wrote:
> >>> >
> >>> >> I will start the vote with a +1
> >>> >>
> >>> >> 2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
> >>> >>
> >>> >> > Please vote on releasing the following candidate as Apache Spark
> >>> >> > version
> >>> >> 1
> >>> >> > .1.1.
> >>> >> >
> >>> >> > This release fixes a number of bugs in Spark 1.1.0. Some of the
> >>> >> > notable
> >>> >> > ones are
> >>> >> > - [SPARK-3426] Sort-based shuffle compression settings are
> >>> >> > incompatible
> >>> >> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
> >>> >> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
> >>> >> truncation
> >>> >> > The full list is at http://s.apache.org/z9h and in the
> CHANGES.txt
> >>> >> > attached.
> >>> >> >
> >>> >> > The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
> >>> >> > http://s.apache.org/cZC
> >>> >> >
> >>> >> > The release files, including signatures, digests, etc can be found
> >>> >> > at:
> >>> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
> >>> >> >
> >>> >> > Release artifacts are signed with the following key:
> >>> >> > https://people.apache.org/keys/committer/andrewor14.asc
> >>> >> >
> >>> >> > The staging repository for this release can be found at:
> >>> >> >
> >>> >> >
> https://repository.apache.org/content/repositories/orgapachespark-1034/
> >>> >> >
> >>> >> > The documentation corresponding to this release can be found at:
> >>> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
> >>> >> >
> >>> >> > Please vote on releasing this package as Apache Spark 1.1.1!
> >>> >> >
> >>> >> > The vote is open until Sunday, November 16, at 04:30 UTC and
> passes
> >>> >> > if
> >>> >> > a majority of at least 3 +1 PMC votes are cast.
> >>> >> > [ ] +1 Release this package as Apache Spark 1.1.1
> >>> >> > [ ] -1 Do not release this package because ...
> >>> >> >
> >>> >> > To learn more about Apache Spark, please see
> >>> >> > http://spark.apache.org/
> >>> >> >
> >>> >> > Cheers,
> >>> >> > Andrew
> >>> >> >
> >>> >>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>
> >>
> >
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Sean Owen <so...@cloudera.com>.
Ah right. This is because I'm running Java 8. This was fixed in
SPARK-3329 (https://github.com/apache/spark/commit/2b7ab814f9bde65ebc57ebd04386e56c97f06f4a#diff-7bfd8d7c8cbb02aa0023e4c3497ee832).
Consider back-porting it if other reasons arise, but this is specific
to tests and to Java 8.
On Thu, Nov 13, 2014 at 8:01 PM, Andrew Or <an...@databricks.com> wrote:
> Yeah, this seems to be somewhat environment specific too. The same test has
> been passing here for a while:
> https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.1-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/lastBuild/consoleFull
>
> 2014-11-13 11:26 GMT-08:00 Michael Armbrust <mi...@databricks.com>:
>
>> Hey Sean,
>>
>> Thanks for pointing this out. Looks like a bad test where we should be
>> doing Set comparison instead of Array.
>>
>> Michael
>>
>> On Thu, Nov 13, 2014 at 2:05 AM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>> LICENSE and NOTICE are fine. Signature and checksum is fine. I
>>> unzipped and built the plain source distribution, which built.
>>>
>>> However I am seeing a consistent test failure with "mvn -DskipTests
>>> clean package; mvn test". In the Hive module:
>>>
>>> - SET commands semantics for a HiveContext *** FAILED ***
>>> Expected Array("spark.sql.key.usedfortestonly=test.val.0",
>>>
>>> "spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0"),
>>> but got
>>> Array("spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0",
>>> "spark.sql.key.usedfortestonly=test.val.0") (HiveQuerySuite.scala:544)
>>>
>>> Anyone else seeing this?
>>>
>>>
>>> On Thu, Nov 13, 2014 at 8:18 AM, Krishna Sankar <ks...@gmail.com>
>>> wrote:
>>> > +1
>>> > 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
>>> > -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
>>> > 2. Tested pyspark, mlib
>>> > 2.1. statistics OK
>>> > 2.2. Linear/Ridge/Laso Regression OK
>>> > 2.3. Decision Tree, Naive Bayes OK
>>> > 2.4. KMeans OK
>>> > 2.5. rdd operations OK
>>> > 2.6. recommendation OK
>>> > 2.7. Good work ! In 1.1.0, there was an error and my program used to
>>> > hang
>>> > (over memory allocation) consistently running validation using
>>> > itertools,
>>> > compute optimum rank, lambda,numofiterations/rmse; data - movielens
>>> > medium
>>> > dataset (1 million records) . It works well in 1.1.1 !
>>> > Cheers
>>> > <k/>
>>> > P.S: Missed Reply all, first time
>>> >
>>> > On Wed, Nov 12, 2014 at 8:35 PM, Andrew Or <an...@databricks.com>
>>> > wrote:
>>> >
>>> >> I will start the vote with a +1
>>> >>
>>> >> 2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
>>> >>
>>> >> > Please vote on releasing the following candidate as Apache Spark
>>> >> > version
>>> >> 1
>>> >> > .1.1.
>>> >> >
>>> >> > This release fixes a number of bugs in Spark 1.1.0. Some of the
>>> >> > notable
>>> >> > ones are
>>> >> > - [SPARK-3426] Sort-based shuffle compression settings are
>>> >> > incompatible
>>> >> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
>>> >> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
>>> >> truncation
>>> >> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>>> >> > attached.
>>> >> >
>>> >> > The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
>>> >> > http://s.apache.org/cZC
>>> >> >
>>> >> > The release files, including signatures, digests, etc can be found
>>> >> > at:
>>> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
>>> >> >
>>> >> > Release artifacts are signed with the following key:
>>> >> > https://people.apache.org/keys/committer/andrewor14.asc
>>> >> >
>>> >> > The staging repository for this release can be found at:
>>> >> >
>>> >> > https://repository.apache.org/content/repositories/orgapachespark-1034/
>>> >> >
>>> >> > The documentation corresponding to this release can be found at:
>>> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
>>> >> >
>>> >> > Please vote on releasing this package as Apache Spark 1.1.1!
>>> >> >
>>> >> > The vote is open until Sunday, November 16, at 04:30 UTC and passes
>>> >> > if
>>> >> > a majority of at least 3 +1 PMC votes are cast.
>>> >> > [ ] +1 Release this package as Apache Spark 1.1.1
>>> >> > [ ] -1 Do not release this package because ...
>>> >> >
>>> >> > To learn more about Apache Spark, please see
>>> >> > http://spark.apache.org/
>>> >> >
>>> >> > Cheers,
>>> >> > Andrew
>>> >> >
>>> >>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Andrew Or <an...@databricks.com>.
Yeah, this seems to be somewhat environment specific too. The same test has
been passing here for a while:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.1-Maven-pre-YARN/hadoop.version=1.0.4,label=centos/lastBuild/consoleFull
2014-11-13 11:26 GMT-08:00 Michael Armbrust <mi...@databricks.com>:
> Hey Sean,
>
> Thanks for pointing this out. Looks like a bad test where we should be
> doing Set comparison instead of Array.
>
> Michael
>
> On Thu, Nov 13, 2014 at 2:05 AM, Sean Owen <so...@cloudera.com> wrote:
>
>> LICENSE and NOTICE are fine. Signature and checksum is fine. I
>> unzipped and built the plain source distribution, which built.
>>
>> However I am seeing a consistent test failure with "mvn -DskipTests
>> clean package; mvn test". In the Hive module:
>>
>> - SET commands semantics for a HiveContext *** FAILED ***
>> Expected Array("spark.sql.key.usedfortestonly=test.val.0",
>>
>> "spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0"),
>> but got
>> Array("spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0",
>> "spark.sql.key.usedfortestonly=test.val.0") (HiveQuerySuite.scala:544)
>>
>> Anyone else seeing this?
>>
>>
>> On Thu, Nov 13, 2014 at 8:18 AM, Krishna Sankar <ks...@gmail.com>
>> wrote:
>> > +1
>> > 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
>> > -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
>> > 2. Tested pyspark, mlib
>> > 2.1. statistics OK
>> > 2.2. Linear/Ridge/Laso Regression OK
>> > 2.3. Decision Tree, Naive Bayes OK
>> > 2.4. KMeans OK
>> > 2.5. rdd operations OK
>> > 2.6. recommendation OK
>> > 2.7. Good work ! In 1.1.0, there was an error and my program used to
>> hang
>> > (over memory allocation) consistently running validation using
>> itertools,
>> > compute optimum rank, lambda,numofiterations/rmse; data - movielens
>> medium
>> > dataset (1 million records) . It works well in 1.1.1 !
>> > Cheers
>> > <k/>
>> > P.S: Missed Reply all, first time
>> >
>> > On Wed, Nov 12, 2014 at 8:35 PM, Andrew Or <an...@databricks.com>
>> wrote:
>> >
>> >> I will start the vote with a +1
>> >>
>> >> 2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
>> >>
>> >> > Please vote on releasing the following candidate as Apache Spark
>> version
>> >> 1
>> >> > .1.1.
>> >> >
>> >> > This release fixes a number of bugs in Spark 1.1.0. Some of the
>> notable
>> >> > ones are
>> >> > - [SPARK-3426] Sort-based shuffle compression settings are
>> incompatible
>> >> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> >> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
>> >> truncation
>> >> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>> >> > attached.
>> >> >
>> >> > The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
>> >> > http://s.apache.org/cZC
>> >> >
>> >> > The release files, including signatures, digests, etc can be found
>> at:
>> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
>> >> >
>> >> > Release artifacts are signed with the following key:
>> >> > https://people.apache.org/keys/committer/andrewor14.asc
>> >> >
>> >> > The staging repository for this release can be found at:
>> >> >
>> https://repository.apache.org/content/repositories/orgapachespark-1034/
>> >> >
>> >> > The documentation corresponding to this release can be found at:
>> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
>> >> >
>> >> > Please vote on releasing this package as Apache Spark 1.1.1!
>> >> >
>> >> > The vote is open until Sunday, November 16, at 04:30 UTC and passes
>> if
>> >> > a majority of at least 3 +1 PMC votes are cast.
>> >> > [ ] +1 Release this package as Apache Spark 1.1.1
>> >> > [ ] -1 Do not release this package because ...
>> >> >
>> >> > To learn more about Apache Spark, please see
>> >> > http://spark.apache.org/
>> >> >
>> >> > Cheers,
>> >> > Andrew
>> >> >
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Michael Armbrust <mi...@databricks.com>.
Hey Sean,
Thanks for pointing this out. Looks like a bad test where we should be
doing Set comparison instead of Array.
Michael
On Thu, Nov 13, 2014 at 2:05 AM, Sean Owen <so...@cloudera.com> wrote:
> LICENSE and NOTICE are fine. Signature and checksum is fine. I
> unzipped and built the plain source distribution, which built.
>
> However I am seeing a consistent test failure with "mvn -DskipTests
> clean package; mvn test". In the Hive module:
>
> - SET commands semantics for a HiveContext *** FAILED ***
> Expected Array("spark.sql.key.usedfortestonly=test.val.0",
>
> "spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0"),
> but got
> Array("spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0",
> "spark.sql.key.usedfortestonly=test.val.0") (HiveQuerySuite.scala:544)
>
> Anyone else seeing this?
>
>
> On Thu, Nov 13, 2014 at 8:18 AM, Krishna Sankar <ks...@gmail.com>
> wrote:
> > +1
> > 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
> > -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
> > 2. Tested pyspark, mlib
> > 2.1. statistics OK
> > 2.2. Linear/Ridge/Laso Regression OK
> > 2.3. Decision Tree, Naive Bayes OK
> > 2.4. KMeans OK
> > 2.5. rdd operations OK
> > 2.6. recommendation OK
> > 2.7. Good work ! In 1.1.0, there was an error and my program used to hang
> > (over memory allocation) consistently running validation using itertools,
> > compute optimum rank, lambda,numofiterations/rmse; data - movielens
> medium
> > dataset (1 million records) . It works well in 1.1.1 !
> > Cheers
> > <k/>
> > P.S: Missed Reply all, first time
> >
> > On Wed, Nov 12, 2014 at 8:35 PM, Andrew Or <an...@databricks.com>
> wrote:
> >
> >> I will start the vote with a +1
> >>
> >> 2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
> >>
> >> > Please vote on releasing the following candidate as Apache Spark
> version
> >> 1
> >> > .1.1.
> >> >
> >> > This release fixes a number of bugs in Spark 1.1.0. Some of the
> notable
> >> > ones are
> >> > - [SPARK-3426] Sort-based shuffle compression settings are
> incompatible
> >> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
> >> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
> >> truncation
> >> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> >> > attached.
> >> >
> >> > The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
> >> > http://s.apache.org/cZC
> >> >
> >> > The release files, including signatures, digests, etc can be found at:
> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
> >> >
> >> > Release artifacts are signed with the following key:
> >> > https://people.apache.org/keys/committer/andrewor14.asc
> >> >
> >> > The staging repository for this release can be found at:
> >> >
> https://repository.apache.org/content/repositories/orgapachespark-1034/
> >> >
> >> > The documentation corresponding to this release can be found at:
> >> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
> >> >
> >> > Please vote on releasing this package as Apache Spark 1.1.1!
> >> >
> >> > The vote is open until Sunday, November 16, at 04:30 UTC and passes if
> >> > a majority of at least 3 +1 PMC votes are cast.
> >> > [ ] +1 Release this package as Apache Spark 1.1.1
> >> > [ ] -1 Do not release this package because ...
> >> >
> >> > To learn more about Apache Spark, please see
> >> > http://spark.apache.org/
> >> >
> >> > Cheers,
> >> > Andrew
> >> >
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Sean Owen <so...@cloudera.com>.
LICENSE and NOTICE are fine. Signature and checksum is fine. I
unzipped and built the plain source distribution, which built.
However I am seeing a consistent test failure with "mvn -DskipTests
clean package; mvn test". In the Hive module:
- SET commands semantics for a HiveContext *** FAILED ***
Expected Array("spark.sql.key.usedfortestonly=test.val.0",
"spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0"),
but got Array("spark.sql.key.usedfortestonlyspark.sql.key.usedfortestonly=test.val.0test.val.0",
"spark.sql.key.usedfortestonly=test.val.0") (HiveQuerySuite.scala:544)
Anyone else seeing this?
On Thu, Nov 13, 2014 at 8:18 AM, Krishna Sankar <ks...@gmail.com> wrote:
> +1
> 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
> -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
> 2. Tested pyspark, mlib
> 2.1. statistics OK
> 2.2. Linear/Ridge/Laso Regression OK
> 2.3. Decision Tree, Naive Bayes OK
> 2.4. KMeans OK
> 2.5. rdd operations OK
> 2.6. recommendation OK
> 2.7. Good work ! In 1.1.0, there was an error and my program used to hang
> (over memory allocation) consistently running validation using itertools,
> compute optimum rank, lambda,numofiterations/rmse; data - movielens medium
> dataset (1 million records) . It works well in 1.1.1 !
> Cheers
> <k/>
> P.S: Missed Reply all, first time
>
> On Wed, Nov 12, 2014 at 8:35 PM, Andrew Or <an...@databricks.com> wrote:
>
>> I will start the vote with a +1
>>
>> 2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
>>
>> > Please vote on releasing the following candidate as Apache Spark version
>> 1
>> > .1.1.
>> >
>> > This release fixes a number of bugs in Spark 1.1.0. Some of the notable
>> > ones are
>> > - [SPARK-3426] Sort-based shuffle compression settings are incompatible
>> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
>> truncation
>> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>> > attached.
>> >
>> > The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
>> > http://s.apache.org/cZC
>> >
>> > The release files, including signatures, digests, etc can be found at:
>> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
>> >
>> > Release artifacts are signed with the following key:
>> > https://people.apache.org/keys/committer/andrewor14.asc
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1034/
>> >
>> > The documentation corresponding to this release can be found at:
>> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
>> >
>> > Please vote on releasing this package as Apache Spark 1.1.1!
>> >
>> > The vote is open until Sunday, November 16, at 04:30 UTC and passes if
>> > a majority of at least 3 +1 PMC votes are cast.
>> > [ ] +1 Release this package as Apache Spark 1.1.1
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see
>> > http://spark.apache.org/
>> >
>> > Cheers,
>> > Andrew
>> >
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Krishna Sankar <ks...@gmail.com>.
+1
1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
-Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
2. Tested pyspark, mlib
2.1. statistics OK
2.2. Linear/Ridge/Laso Regression OK
2.3. Decision Tree, Naive Bayes OK
2.4. KMeans OK
2.5. rdd operations OK
2.6. recommendation OK
2.7. Good work ! In 1.1.0, there was an error and my program used to hang
(over memory allocation) consistently running validation using itertools,
compute optimum rank, lambda,numofiterations/rmse; data - movielens medium
dataset (1 million records) . It works well in 1.1.1 !
Cheers
<k/>
P.S: Missed Reply all, first time
On Wed, Nov 12, 2014 at 8:35 PM, Andrew Or <an...@databricks.com> wrote:
> I will start the vote with a +1
>
> 2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
>
> > Please vote on releasing the following candidate as Apache Spark version
> 1
> > .1.1.
> >
> > This release fixes a number of bugs in Spark 1.1.0. Some of the notable
> > ones are
> > - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
> truncation
> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> > attached.
> >
> > The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
> > http://s.apache.org/cZC
> >
> > The release files, including signatures, digests, etc can be found at:
> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/andrewor14.asc
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1034/
> >
> > The documentation corresponding to this release can be found at:
> > http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
> >
> > Please vote on releasing this package as Apache Spark 1.1.1!
> >
> > The vote is open until Sunday, November 16, at 04:30 UTC and passes if
> > a majority of at least 3 +1 PMC votes are cast.
> > [ ] +1 Release this package as Apache Spark 1.1.1
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> > http://spark.apache.org/
> >
> > Cheers,
> > Andrew
> >
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC1)
Posted by Andrew Or <an...@databricks.com>.
I will start the vote with a +1
2014-11-12 20:34 GMT-08:00 Andrew Or <an...@databricks.com>:
> Please vote on releasing the following candidate as Apache Spark version 1
> .1.1.
>
> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
> ones are
> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> - [SPARK-3948] Stream corruption issues in sort-based shuffle
> - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation
> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> attached.
>
> The tag to be voted on is v1.1.1-rc1 (commit 72a4fdbe):
> http://s.apache.org/cZC
>
> The release files, including signatures, digests, etc can be found at:
> http://people.apache.org/~andrewor14/spark-1.1.1-rc1/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/andrewor14.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1034/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~andrewor14/spark-1.1.1-rc1-docs/
>
> Please vote on releasing this package as Apache Spark 1.1.1!
>
> The vote is open until Sunday, November 16, at 04:30 UTC and passes if
> a majority of at least 3 +1 PMC votes are cast.
> [ ] +1 Release this package as Apache Spark 1.1.1
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see
> http://spark.apache.org/
>
> Cheers,
> Andrew
>