You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Patrick Wendell <pw...@gmail.com> on 2014/05/17 09:58:09 UTC
[VOTE] Release Apache Spark 1.0.0 (rc9)
Please vote on releasing the following candidate as Apache Spark version 1.0.0!
This has one bug fix and one minor feature on top of rc8:
SPARK-1864: https://github.com/apache/spark/pull/808
SPARK-1808: https://github.com/apache/spark/pull/799
The tag to be voted on is v1.0.0-rc9 (commit 920f947):
https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
The release files, including signatures, digests, etc. can be found at:
http://people.apache.org/~pwendell/spark-1.0.0-rc9/
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc
The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1017/
The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
Please vote on releasing this package as Apache Spark 1.0.0!
The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
amajority of at least 3 +1 PMC votes are cast.
[ ] +1 Release this package as Apache Spark 1.0.0
[ ] -1 Do not release this package because ...
To learn more about Apache Spark, please see
http://spark.apache.org/
== API Changes ==
We welcome users to compile Spark applications against 1.0. There are
a few API changes in this release. Here are links to the associated
upgrade guides - user facing changes have been kept as small as
possible.
changes to ML vector specification:
http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
changes to the Java API:
http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
changes to the streaming API:
http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
changes to the GraphX API:
http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
coGroup and related functions now return Iterable[T] instead of Seq[T]
==> Call toSeq on the result to restore the old behavior
SparkContext.jarOfClass returns Option[String] instead of Seq[String]
==> Call toSeq on the result to restore old behavior
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Tom Graves <tg...@yahoo.com>.
I assume we will have an rc10 to fix the issues Matei found?
Tom
On Sunday, May 18, 2014 9:08 PM, Patrick Wendell <pw...@gmail.com> wrote:
Hey Matei - the issue you found is not related to security. This patch
a few days ago broke builds for Hadoop 1 with YARN support enabled.
The patch directly altered the way we deal with commons-lang
dependency, which is what is at the base of this stack trace.
https://github.com/apache/spark/pull/754
- Patrick
On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia <ma...@gmail.com> wrote:
> Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this?
>
> Matei
>
> On May 18, 2014, at 12:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
>
>> I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon.
>>
>> Matei
>>
>>
>> On May 17, 2014, at 11:32 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>>> +1
>>>
>>> Reran my tests from rc5:
>>>
>>> * Built the release from source.
>>> * Compiled Java and Scala apps that interact with HDFS against it.
>>> * Ran them in local mode.
>>> * Ran them against a pseudo-distributed YARN cluster in both yarn-client
>>> mode and yarn-cluster mode.
>>>
>>>
>>> On Sat, May 17, 2014 at 10:08 AM, Andrew Or <an...@databricks.com> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> 2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> I'll start the voting with a +1.
>>>>>>
>>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
>>>>>> wrote:
>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>> version
>>>>>> 1.0.0!
>>>>>>> This has one bug fix and one minor feature on top of rc8:
>>>>>>> SPARK-1864: https://github.com/apache/spark/pull/808
>>>>>>> SPARK-1808: https://github.com/apache/spark/pull/799
>>>>>>>
>>>>>>> The tag to be voted on is v1.0.0-rc9 (commit 920f947):
>>>>>>>
>>>>>>
>>>>>
>>>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc. can be found
>>>> at:
>>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>>>>>>>
>>>>>>> Release artifacts are signed with the following key:
>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>
>>>>>>> The staging repository for this release can be found at:
>>>>>>>
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1017/
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>>>>>>>
>>>>>>> Please vote on releasing this package as Apache Spark 1.0.0!
>>>>>>>
>>>>>>> The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
>>>>>>> amajority of at least 3 +1 PMC votes are cast.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 1.0.0
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>>
>>>>>>> == API Changes ==
>>>>>>> We welcome users to compile Spark applications against 1.0. There are
>>>>>>> a few API changes in this release. Here are links to the associated
>>>>>>> upgrade guides - user facing changes have been kept as small as
>>>>>>> possible.
>>>>>>>
>>>>>>> changes to ML vector specification:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>>>>>>>
>>>>>>> changes to the Java API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>>>>>>>
>>>>>>> changes to the streaming API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>>>>>>>
>>>>>>> changes to the GraphX API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>>>>>>>
>>>>>>> coGroup and related functions now return Iterable[T] instead of
>>>> Seq[T]
>>>>>>> ==> Call toSeq on the result to restore the old behavior
>>>>>>>
>>>>>>> SparkContext.jarOfClass returns Option[String] instead of Seq[String]
>>>>>>> ==> Call toSeq on the result to restore old behavior
>>>>>>
>>>>>
>>>>
>>
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Patrick Wendell <pw...@gmail.com>.
We're cancelling this RC in favor of rc10. There were two blockers: an
issue with Windows run scripts and an issue with the packaging for
Hadoop 1 when hive support is bundled.
https://issues.apache.org/jira/browse/SPARK-1875
https://issues.apache.org/jira/browse/SPARK-1876
Thanks everyone for the testing. TD will be cutting rc10, since I'm
travelling this week (thanks TD!).
- Patrick
On Mon, May 19, 2014 at 7:06 PM, Nan Zhu <zh...@gmail.com> wrote:
> just rerun my test on rc5
>
> everything works
>
> build applications with sbt and the spark-*.jar which is compiled with Hadoop 2.3
>
> +1
>
> --
> Nan Zhu
>
>
> On Sunday, May 18, 2014 at 11:07 PM, witgo wrote:
>
>> How to reproduce this bug?
>>
>>
>> ------------------ Original ------------------
>> From: "Patrick Wendell";<pwendell@gmail.com (mailto:pwendell@gmail.com)>;
>> Date: Mon, May 19, 2014 10:08 AM
>> To: "dev@spark.apache.org (mailto:dev@spark.apache.org)"<dev@spark.apache.org (mailto:dev@spark.apache.org)>;
>> Cc: "Tom Graves"<tgraves_cs@yahoo.com (mailto:tgraves_cs@yahoo.com)>;
>> Subject: Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
>>
>>
>>
>> Hey Matei - the issue you found is not related to security. This patch
>> a few days ago broke builds for Hadoop 1 with YARN support enabled.
>> The patch directly altered the way we deal with commons-lang
>> dependency, which is what is at the base of this stack trace.
>>
>> https://github.com/apache/spark/pull/754
>>
>> - Patrick
>>
>> On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia <matei.zaharia@gmail.com (mailto:matei.zaharia@gmail.com)> wrote:
>> > Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this?
>> >
>> > Matei
>> >
>> > On May 18, 2014, at 12:33 PM, Matei Zaharia <matei.zaharia@gmail.com (mailto:matei.zaharia@gmail.com)> wrote:
>> >
>> > > I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon.
>> > >
>> > > Matei
>> > >
>> > >
>> > > On May 17, 2014, at 11:32 AM, Sandy Ryza <sandy.ryza@cloudera.com (mailto:sandy.ryza@cloudera.com)> wrote:
>> > >
>> > > > +1
>> > > >
>> > > > Reran my tests from rc5:
>> > > >
>> > > > * Built the release from source.
>> > > > * Compiled Java and Scala apps that interact with HDFS against it.
>> > > > * Ran them in local mode.
>> > > > * Ran them against a pseudo-distributed YARN cluster in both yarn-client
>> > > > mode and yarn-cluster mode.
>> > > >
>> > > >
>> > > > On Sat, May 17, 2014 at 10:08 AM, Andrew Or <andrew@databricks.com (mailto:andrew@databricks.com)> wrote:
>> > > >
>> > > > > +1
>> > > > >
>> > > > >
>> > > > > 2014-05-17 8:53 GMT-07:00 Mark Hamstra <mark@clearstorydata.com (mailto:mark@clearstorydata.com)>:
>> > > > >
>> > > > > > +1
>> > > > > >
>> > > > > >
>> > > > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com (mailto:pwendell@gmail.com)
>> > > > > > > wrote:
>> > > > > >
>> > > > > >
>> > > > > > > I'll start the voting with a +1.
>> > > > > > >
>> > > > > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com (mailto:pwendell@gmail.com)>
>> > > > > > > wrote:
>> > > > > > > > Please vote on releasing the following candidate as Apache Spark
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > > > version
>> > > > > > > 1.0.0!
>> > > > > > > > This has one bug fix and one minor feature on top of rc8:
>> > > > > > > > SPARK-1864: https://github.com/apache/spark/pull/808
>> > > > > > > > SPARK-1808: https://github.com/apache/spark/pull/799
>> > > > > > > >
>> > > > > > > > The tag to be voted on is v1.0.0-rc9 (commit 920f947):
>> > > > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>> > > > > > > >
>> > > > > > > > The release files, including signatures, digests, etc. can be found
>> > > > > at:
>> > > > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>> > > > > > > >
>> > > > > > > > Release artifacts are signed with the following key:
>> > > > > > > > https://people.apache.org/keys/committer/pwendell.asc
>> > > > > > > >
>> > > > > > > > The staging repository for this release can be found at:
>> > > > > > https://repository.apache.org/content/repositories/orgapachespark-1017/
>> > > > > > > >
>> > > > > > > > The documentation corresponding to this release can be found at:
>> > > > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>> > > > > > > >
>> > > > > > > > Please vote on releasing this package as Apache Spark 1.0.0!
>> > > > > > > >
>> > > > > > > > The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
>> > > > > > > > amajority of at least 3 +1 PMC votes are cast.
>> > > > > > > >
>> > > > > > > > [ ] +1 Release this package as Apache Spark 1.0.0
>> > > > > > > > [ ] -1 Do not release this package because ...
>> > > > > > > >
>> > > > > > > > To learn more about Apache Spark, please see
>> > > > > > > > http://spark.apache.org/
>> > > > > > > >
>> > > > > > > > == API Changes ==
>> > > > > > > > We welcome users to compile Spark applications against 1.0. There are
>> > > > > > > > a few API changes in this release. Here are links to the associated
>> > > > > > > > upgrade guides - user facing changes have been kept as small as
>> > > > > > > > possible.
>> > > > > > > >
>> > > > > > > > changes to ML vector specification:
>> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>> > > > > > > >
>> > > > > > > > changes to the Java API:
>> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>> > > > > > > >
>> > > > > > > > changes to the streaming API:
>> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>> > > > > > > >
>> > > > > > > > changes to the GraphX API:
>> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>> > > > > > > >
>> > > > > > > > coGroup and related functions now return Iterable[T] instead of
>> > > > > Seq[T]
>> > > > > > > > ==> Call toSeq on the result to restore the old behavior
>> > > > > > > >
>> > > > > > > > SparkContext.jarOfClass returns Option[String] instead of Seq[String]
>> > > > > > > > ==> Call toSeq on the result to restore old behavior
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > >
>> > > >
>> > >
>> > >
>> >
>> >
>>
>>
>>
>
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Nan Zhu <zh...@gmail.com>.
just rerun my test on rc5
everything works
build applications with sbt and the spark-*.jar which is compiled with Hadoop 2.3
+1
--
Nan Zhu
On Sunday, May 18, 2014 at 11:07 PM, witgo wrote:
> How to reproduce this bug?
>
>
> ------------------ Original ------------------
> From: "Patrick Wendell";<pwendell@gmail.com (mailto:pwendell@gmail.com)>;
> Date: Mon, May 19, 2014 10:08 AM
> To: "dev@spark.apache.org (mailto:dev@spark.apache.org)"<dev@spark.apache.org (mailto:dev@spark.apache.org)>;
> Cc: "Tom Graves"<tgraves_cs@yahoo.com (mailto:tgraves_cs@yahoo.com)>;
> Subject: Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
>
>
>
> Hey Matei - the issue you found is not related to security. This patch
> a few days ago broke builds for Hadoop 1 with YARN support enabled.
> The patch directly altered the way we deal with commons-lang
> dependency, which is what is at the base of this stack trace.
>
> https://github.com/apache/spark/pull/754
>
> - Patrick
>
> On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia <matei.zaharia@gmail.com (mailto:matei.zaharia@gmail.com)> wrote:
> > Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this?
> >
> > Matei
> >
> > On May 18, 2014, at 12:33 PM, Matei Zaharia <matei.zaharia@gmail.com (mailto:matei.zaharia@gmail.com)> wrote:
> >
> > > I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon.
> > >
> > > Matei
> > >
> > >
> > > On May 17, 2014, at 11:32 AM, Sandy Ryza <sandy.ryza@cloudera.com (mailto:sandy.ryza@cloudera.com)> wrote:
> > >
> > > > +1
> > > >
> > > > Reran my tests from rc5:
> > > >
> > > > * Built the release from source.
> > > > * Compiled Java and Scala apps that interact with HDFS against it.
> > > > * Ran them in local mode.
> > > > * Ran them against a pseudo-distributed YARN cluster in both yarn-client
> > > > mode and yarn-cluster mode.
> > > >
> > > >
> > > > On Sat, May 17, 2014 at 10:08 AM, Andrew Or <andrew@databricks.com (mailto:andrew@databricks.com)> wrote:
> > > >
> > > > > +1
> > > > >
> > > > >
> > > > > 2014-05-17 8:53 GMT-07:00 Mark Hamstra <mark@clearstorydata.com (mailto:mark@clearstorydata.com)>:
> > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com (mailto:pwendell@gmail.com)
> > > > > > > wrote:
> > > > > >
> > > > > >
> > > > > > > I'll start the voting with a +1.
> > > > > > >
> > > > > > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com (mailto:pwendell@gmail.com)>
> > > > > > > wrote:
> > > > > > > > Please vote on releasing the following candidate as Apache Spark
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > version
> > > > > > > 1.0.0!
> > > > > > > > This has one bug fix and one minor feature on top of rc8:
> > > > > > > > SPARK-1864: https://github.com/apache/spark/pull/808
> > > > > > > > SPARK-1808: https://github.com/apache/spark/pull/799
> > > > > > > >
> > > > > > > > The tag to be voted on is v1.0.0-rc9 (commit 920f947):
> > > > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
> > > > > > > >
> > > > > > > > The release files, including signatures, digests, etc. can be found
> > > > > at:
> > > > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9/
> > > > > > > >
> > > > > > > > Release artifacts are signed with the following key:
> > > > > > > > https://people.apache.org/keys/committer/pwendell.asc
> > > > > > > >
> > > > > > > > The staging repository for this release can be found at:
> > > > > > https://repository.apache.org/content/repositories/orgapachespark-1017/
> > > > > > > >
> > > > > > > > The documentation corresponding to this release can be found at:
> > > > > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
> > > > > > > >
> > > > > > > > Please vote on releasing this package as Apache Spark 1.0.0!
> > > > > > > >
> > > > > > > > The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
> > > > > > > > amajority of at least 3 +1 PMC votes are cast.
> > > > > > > >
> > > > > > > > [ ] +1 Release this package as Apache Spark 1.0.0
> > > > > > > > [ ] -1 Do not release this package because ...
> > > > > > > >
> > > > > > > > To learn more about Apache Spark, please see
> > > > > > > > http://spark.apache.org/
> > > > > > > >
> > > > > > > > == API Changes ==
> > > > > > > > We welcome users to compile Spark applications against 1.0. There are
> > > > > > > > a few API changes in this release. Here are links to the associated
> > > > > > > > upgrade guides - user facing changes have been kept as small as
> > > > > > > > possible.
> > > > > > > >
> > > > > > > > changes to ML vector specification:
> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
> > > > > > > >
> > > > > > > > changes to the Java API:
> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
> > > > > > > >
> > > > > > > > changes to the streaming API:
> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
> > > > > > > >
> > > > > > > > changes to the GraphX API:
> > > > > http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
> > > > > > > >
> > > > > > > > coGroup and related functions now return Iterable[T] instead of
> > > > > Seq[T]
> > > > > > > > ==> Call toSeq on the result to restore the old behavior
> > > > > > > >
> > > > > > > > SparkContext.jarOfClass returns Option[String] instead of Seq[String]
> > > > > > > > ==> Call toSeq on the result to restore old behavior
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by witgo <wi...@qq.com>.
How to reproduce this bug?
------------------ Original ------------------
From: "Patrick Wendell";<pw...@gmail.com>;
Date: Mon, May 19, 2014 10:08 AM
To: "dev@spark.apache.org"<de...@spark.apache.org>;
Cc: "Tom Graves"<tg...@yahoo.com>;
Subject: Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Hey Matei - the issue you found is not related to security. This patch
a few days ago broke builds for Hadoop 1 with YARN support enabled.
The patch directly altered the way we deal with commons-lang
dependency, which is what is at the base of this stack trace.
https://github.com/apache/spark/pull/754
- Patrick
On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia <ma...@gmail.com> wrote:
> Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this?
>
> Matei
>
> On May 18, 2014, at 12:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
>
>> I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon.
>>
>> Matei
>>
>>
>> On May 17, 2014, at 11:32 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>>> +1
>>>
>>> Reran my tests from rc5:
>>>
>>> * Built the release from source.
>>> * Compiled Java and Scala apps that interact with HDFS against it.
>>> * Ran them in local mode.
>>> * Ran them against a pseudo-distributed YARN cluster in both yarn-client
>>> mode and yarn-cluster mode.
>>>
>>>
>>> On Sat, May 17, 2014 at 10:08 AM, Andrew Or <an...@databricks.com> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> 2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> I'll start the voting with a +1.
>>>>>>
>>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
>>>>>> wrote:
>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>> version
>>>>>> 1.0.0!
>>>>>>> This has one bug fix and one minor feature on top of rc8:
>>>>>>> SPARK-1864: https://github.com/apache/spark/pull/808
>>>>>>> SPARK-1808: https://github.com/apache/spark/pull/799
>>>>>>>
>>>>>>> The tag to be voted on is v1.0.0-rc9 (commit 920f947):
>>>>>>>
>>>>>>
>>>>>
>>>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc. can be found
>>>> at:
>>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>>>>>>>
>>>>>>> Release artifacts are signed with the following key:
>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>
>>>>>>> The staging repository for this release can be found at:
>>>>>>>
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1017/
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>>>>>>>
>>>>>>> Please vote on releasing this package as Apache Spark 1.0.0!
>>>>>>>
>>>>>>> The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
>>>>>>> amajority of at least 3 +1 PMC votes are cast.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 1.0.0
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>>
>>>>>>> == API Changes ==
>>>>>>> We welcome users to compile Spark applications against 1.0. There are
>>>>>>> a few API changes in this release. Here are links to the associated
>>>>>>> upgrade guides - user facing changes have been kept as small as
>>>>>>> possible.
>>>>>>>
>>>>>>> changes to ML vector specification:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>>>>>>>
>>>>>>> changes to the Java API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>>>>>>>
>>>>>>> changes to the streaming API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>>>>>>>
>>>>>>> changes to the GraphX API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>>>>>>>
>>>>>>> coGroup and related functions now return Iterable[T] instead of
>>>> Seq[T]
>>>>>>> ==> Call toSeq on the result to restore the old behavior
>>>>>>>
>>>>>>> SparkContext.jarOfClass returns Option[String] instead of Seq[String]
>>>>>>> ==> Call toSeq on the result to restore old behavior
>>>>>>
>>>>>
>>>>
>>
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Matei - the issue you found is not related to security. This patch
a few days ago broke builds for Hadoop 1 with YARN support enabled.
The patch directly altered the way we deal with commons-lang
dependency, which is what is at the base of this stack trace.
https://github.com/apache/spark/pull/754
- Patrick
On Sun, May 18, 2014 at 5:28 PM, Matei Zaharia <ma...@gmail.com> wrote:
> Alright, I've opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1's security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this?
>
> Matei
>
> On May 18, 2014, at 12:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
>
>> I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn't quite match the master-setting behavior of the Unix based one. I'll send a pull request to fix them soon.
>>
>> Matei
>>
>>
>> On May 17, 2014, at 11:32 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>>> +1
>>>
>>> Reran my tests from rc5:
>>>
>>> * Built the release from source.
>>> * Compiled Java and Scala apps that interact with HDFS against it.
>>> * Ran them in local mode.
>>> * Ran them against a pseudo-distributed YARN cluster in both yarn-client
>>> mode and yarn-cluster mode.
>>>
>>>
>>> On Sat, May 17, 2014 at 10:08 AM, Andrew Or <an...@databricks.com> wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> 2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
>>>>
>>>>> +1
>>>>>
>>>>>
>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
>>>>>> wrote:
>>>>>
>>>>>> I'll start the voting with a +1.
>>>>>>
>>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
>>>>>> wrote:
>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>> version
>>>>>> 1.0.0!
>>>>>>> This has one bug fix and one minor feature on top of rc8:
>>>>>>> SPARK-1864: https://github.com/apache/spark/pull/808
>>>>>>> SPARK-1808: https://github.com/apache/spark/pull/799
>>>>>>>
>>>>>>> The tag to be voted on is v1.0.0-rc9 (commit 920f947):
>>>>>>>
>>>>>>
>>>>>
>>>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc. can be found
>>>> at:
>>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>>>>>>>
>>>>>>> Release artifacts are signed with the following key:
>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>
>>>>>>> The staging repository for this release can be found at:
>>>>>>>
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1017/
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>>>>>>>
>>>>>>> Please vote on releasing this package as Apache Spark 1.0.0!
>>>>>>>
>>>>>>> The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
>>>>>>> amajority of at least 3 +1 PMC votes are cast.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 1.0.0
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.apache.org/
>>>>>>>
>>>>>>> == API Changes ==
>>>>>>> We welcome users to compile Spark applications against 1.0. There are
>>>>>>> a few API changes in this release. Here are links to the associated
>>>>>>> upgrade guides - user facing changes have been kept as small as
>>>>>>> possible.
>>>>>>>
>>>>>>> changes to ML vector specification:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>>>>>>>
>>>>>>> changes to the Java API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>>>>>>>
>>>>>>> changes to the streaming API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>>>>>>>
>>>>>>> changes to the GraphX API:
>>>>>>>
>>>>>>
>>>>>
>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>>>>>>>
>>>>>>> coGroup and related functions now return Iterable[T] instead of
>>>> Seq[T]
>>>>>>> ==> Call toSeq on the result to restore the old behavior
>>>>>>>
>>>>>>> SparkContext.jarOfClass returns Option[String] instead of Seq[String]
>>>>>>> ==> Call toSeq on the result to restore old behavior
>>>>>>
>>>>>
>>>>
>>
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Tom Graves <tg...@yahoo.com>.
no ideas off hand, I'll take a look tomorrow.
Tom
On Sunday, May 18, 2014 7:28 PM, Matei Zaharia <ma...@gmail.com> wrote:
Alright, I’ve opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1’s security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this?
Matei
On May 18, 2014, at 12:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
> I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn’t quite match the master-setting behavior of the Unix based one. I’ll send a pull request to fix them soon.
>
> Matei
>
>
> On May 17, 2014, at 11:32 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>
>> +1
>>
>> Reran my tests from rc5:
>>
>> * Built the release from source.
>> * Compiled Java and Scala apps that interact with HDFS against it.
>> * Ran them in local mode.
>> * Ran them against a pseudo-distributed YARN cluster in both yarn-client
>> mode and yarn-cluster mode.
>>
>>
>> On Sat, May 17, 2014 at 10:08 AM, Andrew Or <an...@databricks.com> wrote:
>>
>>> +1
>>>
>>>
>>> 2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
>>>
>>>> +1
>>>>
>>>>
>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
>>>>> wrote:
>>>>
>>>>> I'll start the voting with a +1.
>>>>>
>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
>>>>> wrote:
>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>> version
>>>>> 1.0.0!
>>>>>> This has one bug fix and one minor feature on top of rc8:
>>>>>> SPARK-1864: https://github.com/apache/spark/pull/808
>>>>>> SPARK-1808: https://github.com/apache/spark/pull/799
>>>>>>
>>>>>> The tag to be voted on is v1.0.0-rc9 (commit 920f947):
>>>>>>
>>>>>
>>>>
>>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found
>>> at:
>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>>>>>>
>>>>>> Release artifacts are signed with the following key:
>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>>
>>>> https://repository.apache.org/content/repositories/orgapachespark-1017/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>>>>>>
>>>>>> Please vote on releasing this package as Apache Spark 1.0.0!
>>>>>>
>>>>>> The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
>>>>>> amajority of at least 3 +1 PMC votes are cast.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 1.0.0
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see
>>>>>> http://spark.apache.org/
>>>>>>
>>>>>> == API Changes ==
>>>>>> We welcome users to compile Spark applications against 1.0. There are
>>>>>> a few API changes in this release. Here are links to the associated
>>>>>> upgrade guides - user facing changes have been kept as small as
>>>>>> possible.
>>>>>>
>>>>>> changes to ML vector specification:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>>>>>>
>>>>>> changes to the Java API:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>>>>>>
>>>>>> changes to the streaming API:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>>>>>>
>>>>>> changes to the GraphX API:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>>>>>>
>>>>>> coGroup and related functions now return Iterable[T] instead of
>>> Seq[T]
>>>>>> ==> Call toSeq on the result to restore the old behavior
>>>>>>
>>>>>> SparkContext.jarOfClass returns Option[String] instead of Seq[String]
>>>>>> ==> Call toSeq on the result to restore old behavior
>>>>>
>>>>
>>>
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Matei Zaharia <ma...@gmail.com>.
Alright, I’ve opened https://github.com/apache/spark/pull/819 with the Windows fixes. I also found one other likely bug, https://issues.apache.org/jira/browse/SPARK-1875, in the binary packages for Hadoop1 built in this RC. I think this is due to Hadoop 1’s security code depending on a different version of org.apache.commons than Hadoop 2, but it needs investigation. Tom, any thoughts on this?
Matei
On May 18, 2014, at 12:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
> I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn’t quite match the master-setting behavior of the Unix based one. I’ll send a pull request to fix them soon.
>
> Matei
>
>
> On May 17, 2014, at 11:32 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>
>> +1
>>
>> Reran my tests from rc5:
>>
>> * Built the release from source.
>> * Compiled Java and Scala apps that interact with HDFS against it.
>> * Ran them in local mode.
>> * Ran them against a pseudo-distributed YARN cluster in both yarn-client
>> mode and yarn-cluster mode.
>>
>>
>> On Sat, May 17, 2014 at 10:08 AM, Andrew Or <an...@databricks.com> wrote:
>>
>>> +1
>>>
>>>
>>> 2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
>>>
>>>> +1
>>>>
>>>>
>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
>>>>> wrote:
>>>>
>>>>> I'll start the voting with a +1.
>>>>>
>>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
>>>>> wrote:
>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>> version
>>>>> 1.0.0!
>>>>>> This has one bug fix and one minor feature on top of rc8:
>>>>>> SPARK-1864: https://github.com/apache/spark/pull/808
>>>>>> SPARK-1808: https://github.com/apache/spark/pull/799
>>>>>>
>>>>>> The tag to be voted on is v1.0.0-rc9 (commit 920f947):
>>>>>>
>>>>>
>>>>
>>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>>>>>>
>>>>>> The release files, including signatures, digests, etc. can be found
>>> at:
>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>>>>>>
>>>>>> Release artifacts are signed with the following key:
>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>>
>>>> https://repository.apache.org/content/repositories/orgapachespark-1017/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>>>>>>
>>>>>> Please vote on releasing this package as Apache Spark 1.0.0!
>>>>>>
>>>>>> The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
>>>>>> amajority of at least 3 +1 PMC votes are cast.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 1.0.0
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see
>>>>>> http://spark.apache.org/
>>>>>>
>>>>>> == API Changes ==
>>>>>> We welcome users to compile Spark applications against 1.0. There are
>>>>>> a few API changes in this release. Here are links to the associated
>>>>>> upgrade guides - user facing changes have been kept as small as
>>>>>> possible.
>>>>>>
>>>>>> changes to ML vector specification:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>>>>>>
>>>>>> changes to the Java API:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>>>>>>
>>>>>> changes to the streaming API:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>>>>>>
>>>>>> changes to the GraphX API:
>>>>>>
>>>>>
>>>>
>>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>>>>>>
>>>>>> coGroup and related functions now return Iterable[T] instead of
>>> Seq[T]
>>>>>> ==> Call toSeq on the result to restore the old behavior
>>>>>>
>>>>>> SparkContext.jarOfClass returns Option[String] instead of Seq[String]
>>>>>> ==> Call toSeq on the result to restore old behavior
>>>>>
>>>>
>>>
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Matei Zaharia <ma...@gmail.com>.
I took the always fun task of testing it on Windows, and unfortunately, I found some small problems with the prebuilt packages due to recent changes to the launch scripts: bin/spark-class2.cmd looks in ./jars instead of ./lib for the assembly JAR, and bin/run-example2.cmd doesn’t quite match the master-setting behavior of the Unix based one. I’ll send a pull request to fix them soon.
Matei
On May 17, 2014, at 11:32 AM, Sandy Ryza <sa...@cloudera.com> wrote:
> +1
>
> Reran my tests from rc5:
>
> * Built the release from source.
> * Compiled Java and Scala apps that interact with HDFS against it.
> * Ran them in local mode.
> * Ran them against a pseudo-distributed YARN cluster in both yarn-client
> mode and yarn-cluster mode.
>
>
> On Sat, May 17, 2014 at 10:08 AM, Andrew Or <an...@databricks.com> wrote:
>
>> +1
>>
>>
>> 2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
>>
>>> +1
>>>
>>>
>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
>>>> wrote:
>>>
>>>> I'll start the voting with a +1.
>>>>
>>>> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
>>>> wrote:
>>>>> Please vote on releasing the following candidate as Apache Spark
>>> version
>>>> 1.0.0!
>>>>> This has one bug fix and one minor feature on top of rc8:
>>>>> SPARK-1864: https://github.com/apache/spark/pull/808
>>>>> SPARK-1808: https://github.com/apache/spark/pull/799
>>>>>
>>>>> The tag to be voted on is v1.0.0-rc9 (commit 920f947):
>>>>>
>>>>
>>>
>> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>>>>>
>>>>> The release files, including signatures, digests, etc. can be found
>> at:
>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>>>>>
>>>>> Release artifacts are signed with the following key:
>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>
>>>>> The staging repository for this release can be found at:
>>>>>
>>> https://repository.apache.org/content/repositories/orgapachespark-1017/
>>>>>
>>>>> The documentation corresponding to this release can be found at:
>>>>> http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>>>>>
>>>>> Please vote on releasing this package as Apache Spark 1.0.0!
>>>>>
>>>>> The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
>>>>> amajority of at least 3 +1 PMC votes are cast.
>>>>>
>>>>> [ ] +1 Release this package as Apache Spark 1.0.0
>>>>> [ ] -1 Do not release this package because ...
>>>>>
>>>>> To learn more about Apache Spark, please see
>>>>> http://spark.apache.org/
>>>>>
>>>>> == API Changes ==
>>>>> We welcome users to compile Spark applications against 1.0. There are
>>>>> a few API changes in this release. Here are links to the associated
>>>>> upgrade guides - user facing changes have been kept as small as
>>>>> possible.
>>>>>
>>>>> changes to ML vector specification:
>>>>>
>>>>
>>>
>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>>>>>
>>>>> changes to the Java API:
>>>>>
>>>>
>>>
>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>>>>>
>>>>> changes to the streaming API:
>>>>>
>>>>
>>>
>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>>>>>
>>>>> changes to the GraphX API:
>>>>>
>>>>
>>>
>> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>>>>>
>>>>> coGroup and related functions now return Iterable[T] instead of
>> Seq[T]
>>>>> ==> Call toSeq on the result to restore the old behavior
>>>>>
>>>>> SparkContext.jarOfClass returns Option[String] instead of Seq[String]
>>>>> ==> Call toSeq on the result to restore old behavior
>>>>
>>>
>>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Sandy Ryza <sa...@cloudera.com>.
+1
Reran my tests from rc5:
* Built the release from source.
* Compiled Java and Scala apps that interact with HDFS against it.
* Ran them in local mode.
* Ran them against a pseudo-distributed YARN cluster in both yarn-client
mode and yarn-cluster mode.
On Sat, May 17, 2014 at 10:08 AM, Andrew Or <an...@databricks.com> wrote:
> +1
>
>
> 2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
>
> > +1
> >
> >
> > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
> > >wrote:
> >
> > > I'll start the voting with a +1.
> > >
> > > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
> > > wrote:
> > > > Please vote on releasing the following candidate as Apache Spark
> > version
> > > 1.0.0!
> > > > This has one bug fix and one minor feature on top of rc8:
> > > > SPARK-1864: https://github.com/apache/spark/pull/808
> > > > SPARK-1808: https://github.com/apache/spark/pull/799
> > > >
> > > > The tag to be voted on is v1.0.0-rc9 (commit 920f947):
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
> > > >
> > > > The release files, including signatures, digests, etc. can be found
> at:
> > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9/
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/pwendell.asc
> > > >
> > > > The staging repository for this release can be found at:
> > > >
> > https://repository.apache.org/content/repositories/orgapachespark-1017/
> > > >
> > > > The documentation corresponding to this release can be found at:
> > > > http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
> > > >
> > > > Please vote on releasing this package as Apache Spark 1.0.0!
> > > >
> > > > The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
> > > > amajority of at least 3 +1 PMC votes are cast.
> > > >
> > > > [ ] +1 Release this package as Apache Spark 1.0.0
> > > > [ ] -1 Do not release this package because ...
> > > >
> > > > To learn more about Apache Spark, please see
> > > > http://spark.apache.org/
> > > >
> > > > == API Changes ==
> > > > We welcome users to compile Spark applications against 1.0. There are
> > > > a few API changes in this release. Here are links to the associated
> > > > upgrade guides - user facing changes have been kept as small as
> > > > possible.
> > > >
> > > > changes to ML vector specification:
> > > >
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
> > > >
> > > > changes to the Java API:
> > > >
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
> > > >
> > > > changes to the streaming API:
> > > >
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
> > > >
> > > > changes to the GraphX API:
> > > >
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
> > > >
> > > > coGroup and related functions now return Iterable[T] instead of
> Seq[T]
> > > > ==> Call toSeq on the result to restore the old behavior
> > > >
> > > > SparkContext.jarOfClass returns Option[String] instead of Seq[String]
> > > > ==> Call toSeq on the result to restore old behavior
> > >
> >
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Andrew Or <an...@databricks.com>.
+1
2014-05-17 8:53 GMT-07:00 Mark Hamstra <ma...@clearstorydata.com>:
> +1
>
>
> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pwendell@gmail.com
> >wrote:
>
> > I'll start the voting with a +1.
> >
> > On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
> > wrote:
> > > Please vote on releasing the following candidate as Apache Spark
> version
> > 1.0.0!
> > > This has one bug fix and one minor feature on top of rc8:
> > > SPARK-1864: https://github.com/apache/spark/pull/808
> > > SPARK-1808: https://github.com/apache/spark/pull/799
> > >
> > > The tag to be voted on is v1.0.0-rc9 (commit 920f947):
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
> > >
> > > The release files, including signatures, digests, etc. can be found at:
> > > http://people.apache.org/~pwendell/spark-1.0.0-rc9/
> > >
> > > Release artifacts are signed with the following key:
> > > https://people.apache.org/keys/committer/pwendell.asc
> > >
> > > The staging repository for this release can be found at:
> > >
> https://repository.apache.org/content/repositories/orgapachespark-1017/
> > >
> > > The documentation corresponding to this release can be found at:
> > > http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
> > >
> > > Please vote on releasing this package as Apache Spark 1.0.0!
> > >
> > > The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
> > > amajority of at least 3 +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Spark 1.0.0
> > > [ ] -1 Do not release this package because ...
> > >
> > > To learn more about Apache Spark, please see
> > > http://spark.apache.org/
> > >
> > > == API Changes ==
> > > We welcome users to compile Spark applications against 1.0. There are
> > > a few API changes in this release. Here are links to the associated
> > > upgrade guides - user facing changes have been kept as small as
> > > possible.
> > >
> > > changes to ML vector specification:
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
> > >
> > > changes to the Java API:
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
> > >
> > > changes to the streaming API:
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
> > >
> > > changes to the GraphX API:
> > >
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
> > >
> > > coGroup and related functions now return Iterable[T] instead of Seq[T]
> > > ==> Call toSeq on the result to restore the old behavior
> > >
> > > SparkContext.jarOfClass returns Option[String] instead of Seq[String]
> > > ==> Call toSeq on the result to restore old behavior
> >
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Mark Hamstra <ma...@clearstorydata.com>.
+1
On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>wrote:
> I'll start the voting with a +1.
>
> On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com>
> wrote:
> > Please vote on releasing the following candidate as Apache Spark version
> 1.0.0!
> > This has one bug fix and one minor feature on top of rc8:
> > SPARK-1864: https://github.com/apache/spark/pull/808
> > SPARK-1808: https://github.com/apache/spark/pull/799
> >
> > The tag to be voted on is v1.0.0-rc9 (commit 920f947):
> >
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
> >
> > The release files, including signatures, digests, etc. can be found at:
> > http://people.apache.org/~pwendell/spark-1.0.0-rc9/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/pwendell.asc
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1017/
> >
> > The documentation corresponding to this release can be found at:
> > http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
> >
> > Please vote on releasing this package as Apache Spark 1.0.0!
> >
> > The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
> > amajority of at least 3 +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Spark 1.0.0
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> > http://spark.apache.org/
> >
> > == API Changes ==
> > We welcome users to compile Spark applications against 1.0. There are
> > a few API changes in this release. Here are links to the associated
> > upgrade guides - user facing changes have been kept as small as
> > possible.
> >
> > changes to ML vector specification:
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
> >
> > changes to the Java API:
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
> >
> > changes to the streaming API:
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
> >
> > changes to the GraphX API:
> >
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
> >
> > coGroup and related functions now return Iterable[T] instead of Seq[T]
> > ==> Call toSeq on the result to restore the old behavior
> >
> > SparkContext.jarOfClass returns Option[String] instead of Seq[String]
> > ==> Call toSeq on the result to restore old behavior
>
Re: [VOTE] Release Apache Spark 1.0.0 (rc9)
Posted by Patrick Wendell <pw...@gmail.com>.
I'll start the voting with a +1.
On Sat, May 17, 2014 at 12:58 AM, Patrick Wendell <pw...@gmail.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version 1.0.0!
> This has one bug fix and one minor feature on top of rc8:
> SPARK-1864: https://github.com/apache/spark/pull/808
> SPARK-1808: https://github.com/apache/spark/pull/799
>
> The tag to be voted on is v1.0.0-rc9 (commit 920f947):
> https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=920f947eb5a22a679c0c3186cf69ee75f6041c75
>
> The release files, including signatures, digests, etc. can be found at:
> http://people.apache.org/~pwendell/spark-1.0.0-rc9/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1017/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/
>
> Please vote on releasing this package as Apache Spark 1.0.0!
>
> The vote is open until Tuesday, May 20, at 08:56 UTC and passes if
> amajority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 1.0.0
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see
> http://spark.apache.org/
>
> == API Changes ==
> We welcome users to compile Spark applications against 1.0. There are
> a few API changes in this release. Here are links to the associated
> upgrade guides - user facing changes have been kept as small as
> possible.
>
> changes to ML vector specification:
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/mllib-guide.html#from-09-to-10
>
> changes to the Java API:
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/java-programming-guide.html#upgrading-from-pre-10-versions-of-spark
>
> changes to the streaming API:
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/streaming-programming-guide.html#migration-guide-from-091-or-below-to-1x
>
> changes to the GraphX API:
> http://people.apache.org/~pwendell/spark-1.0.0-rc8-docs/graphx-programming-guide.html#upgrade-guide-from-spark-091
>
> coGroup and related functions now return Iterable[T] instead of Seq[T]
> ==> Call toSeq on the result to restore the old behavior
>
> SparkContext.jarOfClass returns Option[String] instead of Seq[String]
> ==> Call toSeq on the result to restore old behavior