You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Andrew Or <an...@databricks.com> on 2014/11/19 23:51:39 UTC
[VOTE] Release Apache Spark 1.1.1 (RC2)
Please vote on releasing the following candidate as Apache Spark version 1
.1.1.
This release fixes a number of bugs in Spark 1.1.0. Some of the notable
ones are
- [SPARK-3426] Sort-based shuffle compression settings are incompatible
- [SPARK-3948] Stream corruption issues in sort-based shuffle
- [SPARK-4107] Incorrect handling of Channel.read() led to data truncation
The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached.
Additionally, this candidate fixes two blockers from the previous RC:
- [SPARK-4434] Cluster mode jar URLs are broken
- [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills
The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
http://s.apache.org/p8
The release files, including signatures, digests, etc can be found at:
http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/andrewor14.asc
The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1043/
The documentation corresponding to this release can be found at:
http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
Please vote on releasing this package as Apache Spark 1.1.1!
The vote is open until Saturday, November 22, at 23:00 UTC and passes if
a majority of at least 3 +1 PMC votes are cast.
[ ] +1 Release this package as Apache Spark 1.1.1
[ ] -1 Do not release this package because ...
To learn more about Apache Spark, please see
http://spark.apache.org/
Cheers,
Andrew
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Stephen Haberman <st...@gmail.com>.
>
> http://maven.apache.org/plugins/maven-install-plugin/
> examples/specific-local-repo.html
Hm, I didn't know about that plugin--assuming it does all of the
jar/pom/sources/etc., then, yes, that could work...
At first glance, I'm not sure it'll bring over the pom with all of the
transitive dependencies (because at least this invocation is only pointing
at a raw jar file to import), but I'll try it out tomorrow.
That is a good point though; if either you or other people have already
been solving this "import a renamed-to-rc artifact" in terribly
simple/obvious ways that I've just missed, I'm happy to be enlightened. :-)
Thanks,
Stephen
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Sean Owen <so...@cloudera.com>.
Stephen you can publish the artifact to your repo under a different
name, right? IIRC Maven will take care of the pom change along the
way. Yes you would not ever want to mess with changing an artifact
after it's published.
http://maven.apache.org/plugins/maven-install-plugin/examples/specific-local-repo.html
That's how I thought people did this, if they needed to do more than
test a tarball.
On Mon, Nov 24, 2014 at 3:54 AM, Stephen Haberman
<st...@gmail.com> wrote:
> Hi,
>
> I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but
> the"rc" releases not being tagged with "-rcX" means the pre-built artifacts
> are basically useless to me.
>
> (Pedantically, to test a release, I have to upload it into our internal
> repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact
> ends up not being final, then I'm screwed, because I would have to clear
> the local cache of any of our machines, dev/Jenkins/etc., that ever
> downloaded the "formerly known as 1.1.1 but not really" rc artifacts.)
>
> What's frustrating is that I know other Apache projects do rc releases, and
> even get them into Maven central, e.g.:
>
> http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22
>
> So, I apologize for the distraction from getting real work done, but
> perhaps you guys could find a creative way to work around the
> well-intentioned mandate on artifact voting?
>
> (E.g. perhaps have multiple votes, one for each successive rc (with -rcX
> suffix), then, once blessed, another one on the actually-final/no-rcX
> artifact (built from the last rc's tag); or publish no-rcX artifacts for
> official voting, as today, but then, at the same time, add -rcX artifacts
> to Maven central for non-binding/3rd party testing, etc.)
>
> Thanks,
> Stephen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Stephen Haberman <st...@gmail.com>.
Awesome, sounds great, guys; thanks for understanding.
Depending on how badly I need 1.1.1-rc2 (I'll check my jobs tomorrow) I'll
just build a local version for now. Should be easy, it's just been awhile.
:-)
Thanks,
Stephen
On Sun Nov 23 2014 at 11:01:09 PM Patrick Wendell <pw...@gmail.com>
wrote:
> Hey Stephen,
>
> Thanks for bringing this up. Technically when we call a release vote
> it needs to be on the exact commit that will be the final release.
> However, one thing I've thought of doing for a while would be to
> publish the maven artifacts using a version tag with $VERSION-rcX even
> if the underlying commit has $VERSION in the pom files. Some recent
> changes I've made to the way we do publishing in branch 1.2 should
> make this pretty easy - it wasn't very easy before because we used
> maven's publishing plugin which makes modifying the published version
> tricky. Our current approach is, indeed, problematic because maven
> artifacts are supposed to be immutable once they have a specific
> version identifier.
>
> I created SPARK-4568 to track this:
> https://issues.apache.org/jira/browse/SPARK-4568
>
> - Patrick
>
> On Sun, Nov 23, 2014 at 8:11 PM, Matei Zaharia <ma...@gmail.com>
> wrote:
> > Interesting, perhaps we could publish each one with two IDs, of which
> the rc one is unofficial. The problem is indeed that you have to vote on a
> hash for a potentially final artifact.
> >
> > Matei
> >
> >> On Nov 23, 2014, at 7:54 PM, Stephen Haberman <
> stephen.haberman@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but
> >> the"rc" releases not being tagged with "-rcX" means the pre-built
> artifacts
> >> are basically useless to me.
> >>
> >> (Pedantically, to test a release, I have to upload it into our internal
> >> repo, to compile jobs, start clusters, etc. Invariably when an rcX
> artifact
> >> ends up not being final, then I'm screwed, because I would have to clear
> >> the local cache of any of our machines, dev/Jenkins/etc., that ever
> >> downloaded the "formerly known as 1.1.1 but not really" rc artifacts.)
> >>
> >> What's frustrating is that I know other Apache projects do rc releases,
> and
> >> even get them into Maven central, e.g.:
> >>
> >> http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.
> apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22
> >>
> >> So, I apologize for the distraction from getting real work done, but
> >> perhaps you guys could find a creative way to work around the
> >> well-intentioned mandate on artifact voting?
> >>
> >> (E.g. perhaps have multiple votes, one for each successive rc (with -rcX
> >> suffix), then, once blessed, another one on the actually-final/no-rcX
> >> artifact (built from the last rc's tag); or publish no-rcX artifacts for
> >> official voting, as today, but then, at the same time, add -rcX
> artifacts
> >> to Maven central for non-binding/3rd party testing, etc.)
> >>
> >> Thanks,
> >> Stephen
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Patrick Wendell <pw...@gmail.com>.
Hey Stephen,
Thanks for bringing this up. Technically when we call a release vote
it needs to be on the exact commit that will be the final release.
However, one thing I've thought of doing for a while would be to
publish the maven artifacts using a version tag with $VERSION-rcX even
if the underlying commit has $VERSION in the pom files. Some recent
changes I've made to the way we do publishing in branch 1.2 should
make this pretty easy - it wasn't very easy before because we used
maven's publishing plugin which makes modifying the published version
tricky. Our current approach is, indeed, problematic because maven
artifacts are supposed to be immutable once they have a specific
version identifier.
I created SPARK-4568 to track this:
https://issues.apache.org/jira/browse/SPARK-4568
- Patrick
On Sun, Nov 23, 2014 at 8:11 PM, Matei Zaharia <ma...@gmail.com> wrote:
> Interesting, perhaps we could publish each one with two IDs, of which the rc one is unofficial. The problem is indeed that you have to vote on a hash for a potentially final artifact.
>
> Matei
>
>> On Nov 23, 2014, at 7:54 PM, Stephen Haberman <st...@gmail.com> wrote:
>>
>> Hi,
>>
>> I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but
>> the"rc" releases not being tagged with "-rcX" means the pre-built artifacts
>> are basically useless to me.
>>
>> (Pedantically, to test a release, I have to upload it into our internal
>> repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact
>> ends up not being final, then I'm screwed, because I would have to clear
>> the local cache of any of our machines, dev/Jenkins/etc., that ever
>> downloaded the "formerly known as 1.1.1 but not really" rc artifacts.)
>>
>> What's frustrating is that I know other Apache projects do rc releases, and
>> even get them into Maven central, e.g.:
>>
>> http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22
>>
>> So, I apologize for the distraction from getting real work done, but
>> perhaps you guys could find a creative way to work around the
>> well-intentioned mandate on artifact voting?
>>
>> (E.g. perhaps have multiple votes, one for each successive rc (with -rcX
>> suffix), then, once blessed, another one on the actually-final/no-rcX
>> artifact (built from the last rc's tag); or publish no-rcX artifacts for
>> official voting, as today, but then, at the same time, add -rcX artifacts
>> to Maven central for non-binding/3rd party testing, etc.)
>>
>> Thanks,
>> Stephen
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Matei Zaharia <ma...@gmail.com>.
Interesting, perhaps we could publish each one with two IDs, of which the rc one is unofficial. The problem is indeed that you have to vote on a hash for a potentially final artifact.
Matei
> On Nov 23, 2014, at 7:54 PM, Stephen Haberman <st...@gmail.com> wrote:
>
> Hi,
>
> I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but
> the"rc" releases not being tagged with "-rcX" means the pre-built artifacts
> are basically useless to me.
>
> (Pedantically, to test a release, I have to upload it into our internal
> repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact
> ends up not being final, then I'm screwed, because I would have to clear
> the local cache of any of our machines, dev/Jenkins/etc., that ever
> downloaded the "formerly known as 1.1.1 but not really" rc artifacts.)
>
> What's frustrating is that I know other Apache projects do rc releases, and
> even get them into Maven central, e.g.:
>
> http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22
>
> So, I apologize for the distraction from getting real work done, but
> perhaps you guys could find a creative way to work around the
> well-intentioned mandate on artifact voting?
>
> (E.g. perhaps have multiple votes, one for each successive rc (with -rcX
> suffix), then, once blessed, another one on the actually-final/no-rcX
> artifact (built from the last rc's tag); or publish no-rcX artifacts for
> official voting, as today, but then, at the same time, add -rcX artifacts
> to Maven central for non-binding/3rd party testing, etc.)
>
> Thanks,
> Stephen
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Stephen Haberman <st...@gmail.com>.
Hi,
I wanted to try 1.1.1-rc2 because we're running into SPARK-3633, but
the"rc" releases not being tagged with "-rcX" means the pre-built artifacts
are basically useless to me.
(Pedantically, to test a release, I have to upload it into our internal
repo, to compile jobs, start clusters, etc. Invariably when an rcX artifact
ends up not being final, then I'm screwed, because I would have to clear
the local cache of any of our machines, dev/Jenkins/etc., that ever
downloaded the "formerly known as 1.1.1 but not really" rc artifacts.)
What's frustrating is that I know other Apache projects do rc releases, and
even get them into Maven central, e.g.:
http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.tapestry%22%20AND%20a%3A%22tapestry-ioc%22
So, I apologize for the distraction from getting real work done, but
perhaps you guys could find a creative way to work around the
well-intentioned mandate on artifact voting?
(E.g. perhaps have multiple votes, one for each successive rc (with -rcX
suffix), then, once blessed, another one on the actually-final/no-rcX
artifact (built from the last rc's tag); or publish no-rcX artifacts for
official voting, as today, but then, at the same time, add -rcX artifacts
to Maven central for non-binding/3rd party testing, etc.)
Thanks,
Stephen
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by slcclimber <an...@gmail.com>.
+1
Built successfully and ran the
python examples.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-1-1-RC2-tp9439p9452.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Sean Owen <so...@cloudera.com>.
+1 (non binding)
Signatures and license looks good. I built the plain-vanilla
distribution and ran tests. While I still see the Java 8 + Hive test
failure, I think we've established this is ignorable.
On Wed, Nov 19, 2014 at 11:51 PM, Andrew Or <an...@databricks.com> wrote:
> I will start with a +1
>
> 2014-11-19 14:51 GMT-08:00 Andrew Or <an...@databricks.com>:
>
>> Please vote on releasing the following candidate as Apache Spark version 1
>> .1.1.
>>
>> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
>> ones are
>> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation
>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>> attached.
>>
>> Additionally, this candidate fixes two blockers from the previous RC:
>> - [SPARK-4434] Cluster mode jar URLs are broken
>> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
>> spills
>>
>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>> http://s.apache.org/p8
>>
>> The release files, including signatures, digests, etc can be found at:
>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/andrewor14.asc
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1043/
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>>
>> Please vote on releasing this package as Apache Spark 1.1.1!
>>
>> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
>> a majority of at least 3 +1 PMC votes are cast.
>> [ ] +1 Release this package as Apache Spark 1.1.1
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see
>> http://spark.apache.org/
>>
>> Cheers,
>> Andrew
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by vaquar khan <va...@gmail.com>.
+1 Release this package as Apache Spark 1.1.1
On 20 Nov 2014 04:22, "Andrew Or" <an...@databricks.com> wrote:
> I will start with a +1
>
> 2014-11-19 14:51 GMT-08:00 Andrew Or <an...@databricks.com>:
>
> > Please vote on releasing the following candidate as Apache Spark version
> 1
> > .1.1.
> >
> > This release fixes a number of bugs in Spark 1.1.0. Some of the notable
> > ones are
> > - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
> truncation
> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> > attached.
> >
> > Additionally, this candidate fixes two blockers from the previous RC:
> > - [SPARK-4434] Cluster mode jar URLs are broken
> > - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
> > spills
> >
> > The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> > http://s.apache.org/p8
> >
> > The release files, including signatures, digests, etc can be found at:
> > http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/andrewor14.asc
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1043/
> >
> > The documentation corresponding to this release can be found at:
> > http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
> >
> > Please vote on releasing this package as Apache Spark 1.1.1!
> >
> > The vote is open until Saturday, November 22, at 23:00 UTC and passes if
> > a majority of at least 3 +1 PMC votes are cast.
> > [ ] +1 Release this package as Apache Spark 1.1.1
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> > http://spark.apache.org/
> >
> > Cheers,
> > Andrew
> >
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Matei Zaharia <ma...@gmail.com>.
+1
Tested on Mac OS X, checked that bugs with too many small files being spilled are fixed.
Matei
> On Nov 19, 2014, at 7:44 PM, Krishna Sankar <ks...@gmail.com> wrote:
>
> +1
> 1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
> -Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
> 2. Tested pyspark, mlib
> 2.1. statistics OK
> 2.2. Linear/Ridge/Laso Regression OK
> 2.3. Decision Tree, Naive Bayes OK
> 2.4. KMeans OK
> 2.5. rdd operations OK
> 2.6. recommendation OK
> 2.7. Good work ! In 1.1.0, there was an error and my program used to hang
> (over memory allocation) consistently running validation using itertools,
> compute optimum rank, lambda,numofiterations/rmse; data - movielens medium
> dataset (1 million records) . It works well in 1.1.1 !
>
> Cheers
> <k/>
>
> On Wed, Nov 19, 2014 at 6:00 PM, Xiangrui Meng <me...@gmail.com> wrote:
>
>> +1. Checked version numbers and doc. Tested a few ML examples with
>> Java 6 and verified some recently merged bug fixes. -Xiangrui
>>
>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com> wrote:
>>> I will start with a +1
>>>
>>> 2014-11-19 14:51 GMT-08:00 Andrew Or <an...@databricks.com>:
>>>
>>>> Please vote on releasing the following candidate as Apache Spark
>> version 1
>>>> .1.1.
>>>>
>>>> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
>>>> ones are
>>>> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
>>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
>> truncation
>>>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>>>> attached.
>>>>
>>>> Additionally, this candidate fixes two blockers from the previous RC:
>>>> - [SPARK-4434] Cluster mode jar URLs are broken
>>>> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
>>>> spills
>>>>
>>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>>>> http://s.apache.org/p8
>>>>
>>>> The release files, including signatures, digests, etc can be found at:
>>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>>>>
>>>> Release artifacts are signed with the following key:
>>>> https://people.apache.org/keys/committer/andrewor14.asc
>>>>
>>>> The staging repository for this release can be found at:
>>>> https://repository.apache.org/content/repositories/orgapachespark-1043/
>>>>
>>>> The documentation corresponding to this release can be found at:
>>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>>>>
>>>> Please vote on releasing this package as Apache Spark 1.1.1!
>>>>
>>>> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
>>>> a majority of at least 3 +1 PMC votes are cast.
>>>> [ ] +1 Release this package as Apache Spark 1.1.1
>>>> [ ] -1 Do not release this package because ...
>>>>
>>>> To learn more about Apache Spark, please see
>>>> http://spark.apache.org/
>>>>
>>>> Cheers,
>>>> Andrew
>>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Krishna Sankar <ks...@gmail.com>.
+1
1. Compiled OSX 10.10 (Yosemite) mvn -Pyarn -Phadoop-2.4
-Dhadoop.version=2.4.0 -DskipTests clean package 10:49 min
2. Tested pyspark, mlib
2.1. statistics OK
2.2. Linear/Ridge/Laso Regression OK
2.3. Decision Tree, Naive Bayes OK
2.4. KMeans OK
2.5. rdd operations OK
2.6. recommendation OK
2.7. Good work ! In 1.1.0, there was an error and my program used to hang
(over memory allocation) consistently running validation using itertools,
compute optimum rank, lambda,numofiterations/rmse; data - movielens medium
dataset (1 million records) . It works well in 1.1.1 !
Cheers
<k/>
On Wed, Nov 19, 2014 at 6:00 PM, Xiangrui Meng <me...@gmail.com> wrote:
> +1. Checked version numbers and doc. Tested a few ML examples with
> Java 6 and verified some recently merged bug fixes. -Xiangrui
>
> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com> wrote:
> > I will start with a +1
> >
> > 2014-11-19 14:51 GMT-08:00 Andrew Or <an...@databricks.com>:
> >
> >> Please vote on releasing the following candidate as Apache Spark
> version 1
> >> .1.1.
> >>
> >> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
> >> ones are
> >> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> >> - [SPARK-3948] Stream corruption issues in sort-based shuffle
> >> - [SPARK-4107] Incorrect handling of Channel.read() led to data
> truncation
> >> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> >> attached.
> >>
> >> Additionally, this candidate fixes two blockers from the previous RC:
> >> - [SPARK-4434] Cluster mode jar URLs are broken
> >> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
> >> spills
> >>
> >> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> >> http://s.apache.org/p8
> >>
> >> The release files, including signatures, digests, etc can be found at:
> >> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
> >>
> >> Release artifacts are signed with the following key:
> >> https://people.apache.org/keys/committer/andrewor14.asc
> >>
> >> The staging repository for this release can be found at:
> >> https://repository.apache.org/content/repositories/orgapachespark-1043/
> >>
> >> The documentation corresponding to this release can be found at:
> >> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
> >>
> >> Please vote on releasing this package as Apache Spark 1.1.1!
> >>
> >> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
> >> a majority of at least 3 +1 PMC votes are cast.
> >> [ ] +1 Release this package as Apache Spark 1.1.1
> >> [ ] -1 Do not release this package because ...
> >>
> >> To learn more about Apache Spark, please see
> >> http://spark.apache.org/
> >>
> >> Cheers,
> >> Andrew
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Xiangrui Meng <me...@gmail.com>.
+1. Checked version numbers and doc. Tested a few ML examples with
Java 6 and verified some recently merged bug fixes. -Xiangrui
On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com> wrote:
> I will start with a +1
>
> 2014-11-19 14:51 GMT-08:00 Andrew Or <an...@databricks.com>:
>
>> Please vote on releasing the following candidate as Apache Spark version 1
>> .1.1.
>>
>> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
>> ones are
>> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation
>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>> attached.
>>
>> Additionally, this candidate fixes two blockers from the previous RC:
>> - [SPARK-4434] Cluster mode jar URLs are broken
>> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
>> spills
>>
>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>> http://s.apache.org/p8
>>
>> The release files, including signatures, digests, etc can be found at:
>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>>
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/andrewor14.asc
>>
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1043/
>>
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>>
>> Please vote on releasing this package as Apache Spark 1.1.1!
>>
>> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
>> a majority of at least 3 +1 PMC votes are cast.
>> [ ] +1 Release this package as Apache Spark 1.1.1
>> [ ] -1 Do not release this package because ...
>>
>> To learn more about Apache Spark, please see
>> http://spark.apache.org/
>>
>> Cheers,
>> Andrew
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Andrew Or <an...@databricks.com>.
I will start with a +1
2014-11-19 14:51 GMT-08:00 Andrew Or <an...@databricks.com>:
> Please vote on releasing the following candidate as Apache Spark version 1
> .1.1.
>
> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
> ones are
> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> - [SPARK-3948] Stream corruption issues in sort-based shuffle
> - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation
> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> attached.
>
> Additionally, this candidate fixes two blockers from the previous RC:
> - [SPARK-4434] Cluster mode jar URLs are broken
> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
> spills
>
> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> http://s.apache.org/p8
>
> The release files, including signatures, digests, etc can be found at:
> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/andrewor14.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1043/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>
> Please vote on releasing this package as Apache Spark 1.1.1!
>
> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
> a majority of at least 3 +1 PMC votes are cast.
> [ ] +1 Release this package as Apache Spark 1.1.1
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see
> http://spark.apache.org/
>
> Cheers,
> Andrew
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Debasish Das <de...@gmail.com>.
Actually +1 from me...
This is a recommendAll feature we are testing which is really compute
intensive...
For ranking metric calculation I was trying to run through the Netflix
matrix and generate a ranked list of recommendation for all 17K products
and perhaps it needs more compute than what is needed. I was running 6
nodes, 120 cores, 240 GB...It needed to shuffle around 100 GB over 6
nodes...
A version with topK runs fine where K = (some multipler on number of movies
each user saw and we cross validate on that)
Running the following JIRA on Netflix dataset (the dataset is distributed
with Jellyfish code http://i.stanford.edu/hazy/victor/Hogwild/), will
reproduce the failure...
https://issues.apache.org/jira/browse/SPARK-4231
The failed job I will debug more and figure out the real cause. If needed I
will open up new JIRAs.
On Sun, Nov 23, 2014 at 9:50 AM, Debasish Das <de...@gmail.com>
wrote:
> -1 from me...same FetchFailed issue as what Hector saw...
>
> I am running Netflix dataset and dumping out recommendation for all users.
> It shuffles around 100 GB data on disk to run a reduceByKey per user on
> utils.BoundedPriorityQueue...The code runs fine with MovieLens1m dataset...
>
> I gave Spark 10 nodes, 8 cores, 160 GB of memory.
>
> Fails with the following FetchFailed errors.
>
> 14/11/23 11:51:22 WARN TaskSetManager: Lost task 28.0 in stage 188.0 (TID
> 2818, tblpmidn08adv-hdp.tdc.vzwcorp.com): FetchFailed(BlockManagerId(1,
> tblpmidn03adv-hdp.tdc.vzwcorp.com, 52528, 0), shuffleId=35, mapId=28,
> reduceId=28)
>
> It's a consistent behavior on master as well.
>
> I tested it both on YARN and Standalone. I compiled spark-1.1 branch
> (assuming it has all the fixes from RC2 tag.
>
> I am now compiling spark-1.0 branch and see if this issue shows up there
> as well. If it is related to hash/sort based shuffle most likely it won't
> show up on 1.0.
>
> Thanks.
>
> Deb
>
> On Thu, Nov 20, 2014 at 12:16 PM, Hector Yee <he...@gmail.com> wrote:
>
>> Whoops I must have used the 1.2 preview and mixed them up.
>>
>> spark-shell -version shows version 1.2.0
>>
>> Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to
>> 1.2
>>
>> On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>
>> > Ah, I see. But the spark.shuffle.blockTransferService property doesn't
>> > exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem?
>> >
>> > Matei
>> >
>> > On Nov 20, 2014, at 11:50 AM, Hector Yee <he...@gmail.com> wrote:
>> >
>> > This is whatever was in http://people.apache.org/~andrewor14/spark-1
>> > .1.1-rc2/
>> >
>> > On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia <
>> matei.zaharia@gmail.com>
>> > wrote:
>> >
>> >> Hector, is this a comment on 1.1.1 or on the 1.2 preview?
>> >>
>> >> Matei
>> >>
>> >> > On Nov 20, 2014, at 11:39 AM, Hector Yee <he...@gmail.com>
>> wrote:
>> >> >
>> >> > I think it is a race condition caused by netty deactivating a channel
>> >> while
>> >> > it is active.
>> >> > Switched to nio and it works fine
>> >> > --conf spark.shuffle.blockTransferService=nio
>> >> >
>> >> > On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <he...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> I'm still seeing the fetch failed error and updated
>> >> >> https://issues.apache.org/jira/browse/SPARK-3633
>> >> >>
>> >> >> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <
>> vanzin@cloudera.com>
>> >> >> wrote:
>> >> >>
>> >> >>> +1 (non-binding)
>> >> >>>
>> >> >>> . ran simple things on spark-shell
>> >> >>> . ran jobs in yarn client & cluster modes, and standalone cluster
>> mode
>> >> >>>
>> >> >>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com>
>> >> wrote:
>> >> >>>> Please vote on releasing the following candidate as Apache Spark
>> >> version
>> >> >>>> 1.1.1.
>> >> >>>>
>> >> >>>> This release fixes a number of bugs in Spark 1.1.0. Some of the
>> >> notable
>> >> >>> ones
>> >> >>>> are
>> >> >>>> - [SPARK-3426] Sort-based shuffle compression settings are
>> >> incompatible
>> >> >>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> >> >>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
>> >> >>> truncation
>> >> >>>> The full list is at http://s.apache.org/z9h and in the
>> CHANGES.txt
>> >> >>> attached.
>> >> >>>>
>> >> >>>> Additionally, this candidate fixes two blockers from the previous
>> RC:
>> >> >>>> - [SPARK-4434] Cluster mode jar URLs are broken
>> >> >>>> - [SPARK-4480][SPARK-4467] Too many open files exception from
>> shuffle
>> >> >>> spills
>> >> >>>>
>> >> >>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>> >> >>>> http://s.apache.org/p8
>> >> >>>>
>> >> >>>> The release files, including signatures, digests, etc can be found
>> >> at:
>> >> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>> >> >>>>
>> >> >>>> Release artifacts are signed with the following key:
>> >> >>>> https://people.apache.org/keys/committer/andrewor14.asc
>> >> >>>>
>> >> >>>> The staging repository for this release can be found at:
>> >> >>>>
>> >>
>> https://repository.apache.org/content/repositories/orgapachespark-1043/
>> >> >>>>
>> >> >>>> The documentation corresponding to this release can be found at:
>> >> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>> >> >>>>
>> >> >>>> Please vote on releasing this package as Apache Spark 1.1.1!
>> >> >>>>
>> >> >>>> The vote is open until Saturday, November 22, at 23:00 UTC and
>> >> passes if
>> >> >>>> a majority of at least 3 +1 PMC votes are cast.
>> >> >>>> [ ] +1 Release this package as Apache Spark 1.1.1
>> >> >>>> [ ] -1 Do not release this package because ...
>> >> >>>>
>> >> >>>> To learn more about Apache Spark, please see
>> >> >>>> http://spark.apache.org/
>> >> >>>>
>> >> >>>> Cheers,
>> >> >>>> Andrew
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> ---------------------------------------------------------------------
>> >> >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> >>>> For additional commands, e-mail: dev-help@spark.apache.org
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Marcelo
>> >> >>>
>> >> >>>
>> ---------------------------------------------------------------------
>> >> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> >>> For additional commands, e-mail: dev-help@spark.apache.org
>> >> >>>
>> >> >>>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Yee Yang Li Hector <http://google.com/+HectorYee>
>> >> >> *google.com/+HectorYee <http://google.com/+HectorYee>*
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Yee Yang Li Hector <http://google.com/+HectorYee>
>> >> > *google.com/+HectorYee <http://google.com/+HectorYee>*
>> >>
>> >>
>> >
>> >
>> > --
>> > Yee Yang Li Hector <http://google.com/+HectorYee>
>> > *google.com/+HectorYee <http://google.com/+HectorYee>*
>> >
>> >
>> >
>>
>>
>> --
>> Yee Yang Li Hector <http://google.com/+HectorYee>
>> *google.com/+HectorYee <http://google.com/+HectorYee>*
>>
>
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Patrick Wendell <pw...@gmail.com>.
+1 (binding).
Don't see any evidence of regressions at this point. The issue
reported by Hector was not related to this rlease.
On Sun, Nov 23, 2014 at 9:50 AM, Debasish Das <de...@gmail.com> wrote:
> -1 from me...same FetchFailed issue as what Hector saw...
>
> I am running Netflix dataset and dumping out recommendation for all users.
> It shuffles around 100 GB data on disk to run a reduceByKey per user on
> utils.BoundedPriorityQueue...The code runs fine with MovieLens1m dataset...
>
> I gave Spark 10 nodes, 8 cores, 160 GB of memory.
>
> Fails with the following FetchFailed errors.
>
> 14/11/23 11:51:22 WARN TaskSetManager: Lost task 28.0 in stage 188.0 (TID
> 2818, tblpmidn08adv-hdp.tdc.vzwcorp.com): FetchFailed(BlockManagerId(1,
> tblpmidn03adv-hdp.tdc.vzwcorp.com, 52528, 0), shuffleId=35, mapId=28,
> reduceId=28)
>
> It's a consistent behavior on master as well.
>
> I tested it both on YARN and Standalone. I compiled spark-1.1 branch
> (assuming it has all the fixes from RC2 tag.
>
> I am now compiling spark-1.0 branch and see if this issue shows up there as
> well. If it is related to hash/sort based shuffle most likely it won't show
> up on 1.0.
>
> Thanks.
>
> Deb
>
> On Thu, Nov 20, 2014 at 12:16 PM, Hector Yee <he...@gmail.com> wrote:
>
>> Whoops I must have used the 1.2 preview and mixed them up.
>>
>> spark-shell -version shows version 1.2.0
>>
>> Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to
>> 1.2
>>
>> On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia <ma...@gmail.com>
>> wrote:
>>
>> > Ah, I see. But the spark.shuffle.blockTransferService property doesn't
>> > exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem?
>> >
>> > Matei
>> >
>> > On Nov 20, 2014, at 11:50 AM, Hector Yee <he...@gmail.com> wrote:
>> >
>> > This is whatever was in http://people.apache.org/~andrewor14/spark-1
>> > .1.1-rc2/
>> >
>> > On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia <matei.zaharia@gmail.com
>> >
>> > wrote:
>> >
>> >> Hector, is this a comment on 1.1.1 or on the 1.2 preview?
>> >>
>> >> Matei
>> >>
>> >> > On Nov 20, 2014, at 11:39 AM, Hector Yee <he...@gmail.com>
>> wrote:
>> >> >
>> >> > I think it is a race condition caused by netty deactivating a channel
>> >> while
>> >> > it is active.
>> >> > Switched to nio and it works fine
>> >> > --conf spark.shuffle.blockTransferService=nio
>> >> >
>> >> > On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <he...@gmail.com>
>> >> wrote:
>> >> >
>> >> >> I'm still seeing the fetch failed error and updated
>> >> >> https://issues.apache.org/jira/browse/SPARK-3633
>> >> >>
>> >> >> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <
>> vanzin@cloudera.com>
>> >> >> wrote:
>> >> >>
>> >> >>> +1 (non-binding)
>> >> >>>
>> >> >>> . ran simple things on spark-shell
>> >> >>> . ran jobs in yarn client & cluster modes, and standalone cluster
>> mode
>> >> >>>
>> >> >>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com>
>> >> wrote:
>> >> >>>> Please vote on releasing the following candidate as Apache Spark
>> >> version
>> >> >>>> 1.1.1.
>> >> >>>>
>> >> >>>> This release fixes a number of bugs in Spark 1.1.0. Some of the
>> >> notable
>> >> >>> ones
>> >> >>>> are
>> >> >>>> - [SPARK-3426] Sort-based shuffle compression settings are
>> >> incompatible
>> >> >>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> >> >>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
>> >> >>> truncation
>> >> >>>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>> >> >>> attached.
>> >> >>>>
>> >> >>>> Additionally, this candidate fixes two blockers from the previous
>> RC:
>> >> >>>> - [SPARK-4434] Cluster mode jar URLs are broken
>> >> >>>> - [SPARK-4480][SPARK-4467] Too many open files exception from
>> shuffle
>> >> >>> spills
>> >> >>>>
>> >> >>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>> >> >>>> http://s.apache.org/p8
>> >> >>>>
>> >> >>>> The release files, including signatures, digests, etc can be found
>> >> at:
>> >> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>> >> >>>>
>> >> >>>> Release artifacts are signed with the following key:
>> >> >>>> https://people.apache.org/keys/committer/andrewor14.asc
>> >> >>>>
>> >> >>>> The staging repository for this release can be found at:
>> >> >>>>
>> >> https://repository.apache.org/content/repositories/orgapachespark-1043/
>> >> >>>>
>> >> >>>> The documentation corresponding to this release can be found at:
>> >> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>> >> >>>>
>> >> >>>> Please vote on releasing this package as Apache Spark 1.1.1!
>> >> >>>>
>> >> >>>> The vote is open until Saturday, November 22, at 23:00 UTC and
>> >> passes if
>> >> >>>> a majority of at least 3 +1 PMC votes are cast.
>> >> >>>> [ ] +1 Release this package as Apache Spark 1.1.1
>> >> >>>> [ ] -1 Do not release this package because ...
>> >> >>>>
>> >> >>>> To learn more about Apache Spark, please see
>> >> >>>> http://spark.apache.org/
>> >> >>>>
>> >> >>>> Cheers,
>> >> >>>> Andrew
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> ---------------------------------------------------------------------
>> >> >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> >>>> For additional commands, e-mail: dev-help@spark.apache.org
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Marcelo
>> >> >>>
>> >> >>>
>> ---------------------------------------------------------------------
>> >> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >> >>> For additional commands, e-mail: dev-help@spark.apache.org
>> >> >>>
>> >> >>>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Yee Yang Li Hector <http://google.com/+HectorYee>
>> >> >> *google.com/+HectorYee <http://google.com/+HectorYee>*
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Yee Yang Li Hector <http://google.com/+HectorYee>
>> >> > *google.com/+HectorYee <http://google.com/+HectorYee>*
>> >>
>> >>
>> >
>> >
>> > --
>> > Yee Yang Li Hector <http://google.com/+HectorYee>
>> > *google.com/+HectorYee <http://google.com/+HectorYee>*
>> >
>> >
>> >
>>
>>
>> --
>> Yee Yang Li Hector <http://google.com/+HectorYee>
>> *google.com/+HectorYee <http://google.com/+HectorYee>*
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Debasish Das <de...@gmail.com>.
-1 from me...same FetchFailed issue as what Hector saw...
I am running Netflix dataset and dumping out recommendation for all users.
It shuffles around 100 GB data on disk to run a reduceByKey per user on
utils.BoundedPriorityQueue...The code runs fine with MovieLens1m dataset...
I gave Spark 10 nodes, 8 cores, 160 GB of memory.
Fails with the following FetchFailed errors.
14/11/23 11:51:22 WARN TaskSetManager: Lost task 28.0 in stage 188.0 (TID
2818, tblpmidn08adv-hdp.tdc.vzwcorp.com): FetchFailed(BlockManagerId(1,
tblpmidn03adv-hdp.tdc.vzwcorp.com, 52528, 0), shuffleId=35, mapId=28,
reduceId=28)
It's a consistent behavior on master as well.
I tested it both on YARN and Standalone. I compiled spark-1.1 branch
(assuming it has all the fixes from RC2 tag.
I am now compiling spark-1.0 branch and see if this issue shows up there as
well. If it is related to hash/sort based shuffle most likely it won't show
up on 1.0.
Thanks.
Deb
On Thu, Nov 20, 2014 at 12:16 PM, Hector Yee <he...@gmail.com> wrote:
> Whoops I must have used the 1.2 preview and mixed them up.
>
> spark-shell -version shows version 1.2.0
>
> Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to
> 1.2
>
> On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
> > Ah, I see. But the spark.shuffle.blockTransferService property doesn't
> > exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem?
> >
> > Matei
> >
> > On Nov 20, 2014, at 11:50 AM, Hector Yee <he...@gmail.com> wrote:
> >
> > This is whatever was in http://people.apache.org/~andrewor14/spark-1
> > .1.1-rc2/
> >
> > On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia <matei.zaharia@gmail.com
> >
> > wrote:
> >
> >> Hector, is this a comment on 1.1.1 or on the 1.2 preview?
> >>
> >> Matei
> >>
> >> > On Nov 20, 2014, at 11:39 AM, Hector Yee <he...@gmail.com>
> wrote:
> >> >
> >> > I think it is a race condition caused by netty deactivating a channel
> >> while
> >> > it is active.
> >> > Switched to nio and it works fine
> >> > --conf spark.shuffle.blockTransferService=nio
> >> >
> >> > On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <he...@gmail.com>
> >> wrote:
> >> >
> >> >> I'm still seeing the fetch failed error and updated
> >> >> https://issues.apache.org/jira/browse/SPARK-3633
> >> >>
> >> >> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <
> vanzin@cloudera.com>
> >> >> wrote:
> >> >>
> >> >>> +1 (non-binding)
> >> >>>
> >> >>> . ran simple things on spark-shell
> >> >>> . ran jobs in yarn client & cluster modes, and standalone cluster
> mode
> >> >>>
> >> >>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com>
> >> wrote:
> >> >>>> Please vote on releasing the following candidate as Apache Spark
> >> version
> >> >>>> 1.1.1.
> >> >>>>
> >> >>>> This release fixes a number of bugs in Spark 1.1.0. Some of the
> >> notable
> >> >>> ones
> >> >>>> are
> >> >>>> - [SPARK-3426] Sort-based shuffle compression settings are
> >> incompatible
> >> >>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
> >> >>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
> >> >>> truncation
> >> >>>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> >> >>> attached.
> >> >>>>
> >> >>>> Additionally, this candidate fixes two blockers from the previous
> RC:
> >> >>>> - [SPARK-4434] Cluster mode jar URLs are broken
> >> >>>> - [SPARK-4480][SPARK-4467] Too many open files exception from
> shuffle
> >> >>> spills
> >> >>>>
> >> >>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> >> >>>> http://s.apache.org/p8
> >> >>>>
> >> >>>> The release files, including signatures, digests, etc can be found
> >> at:
> >> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
> >> >>>>
> >> >>>> Release artifacts are signed with the following key:
> >> >>>> https://people.apache.org/keys/committer/andrewor14.asc
> >> >>>>
> >> >>>> The staging repository for this release can be found at:
> >> >>>>
> >> https://repository.apache.org/content/repositories/orgapachespark-1043/
> >> >>>>
> >> >>>> The documentation corresponding to this release can be found at:
> >> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
> >> >>>>
> >> >>>> Please vote on releasing this package as Apache Spark 1.1.1!
> >> >>>>
> >> >>>> The vote is open until Saturday, November 22, at 23:00 UTC and
> >> passes if
> >> >>>> a majority of at least 3 +1 PMC votes are cast.
> >> >>>> [ ] +1 Release this package as Apache Spark 1.1.1
> >> >>>> [ ] -1 Do not release this package because ...
> >> >>>>
> >> >>>> To learn more about Apache Spark, please see
> >> >>>> http://spark.apache.org/
> >> >>>>
> >> >>>> Cheers,
> >> >>>> Andrew
> >> >>>>
> >> >>>>
> >> >>>>
> ---------------------------------------------------------------------
> >> >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> >>>> For additional commands, e-mail: dev-help@spark.apache.org
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Marcelo
> >> >>>
> >> >>>
> ---------------------------------------------------------------------
> >> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >> >>> For additional commands, e-mail: dev-help@spark.apache.org
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >> Yee Yang Li Hector <http://google.com/+HectorYee>
> >> >> *google.com/+HectorYee <http://google.com/+HectorYee>*
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Yee Yang Li Hector <http://google.com/+HectorYee>
> >> > *google.com/+HectorYee <http://google.com/+HectorYee>*
> >>
> >>
> >
> >
> > --
> > Yee Yang Li Hector <http://google.com/+HectorYee>
> > *google.com/+HectorYee <http://google.com/+HectorYee>*
> >
> >
> >
>
>
> --
> Yee Yang Li Hector <http://google.com/+HectorYee>
> *google.com/+HectorYee <http://google.com/+HectorYee>*
>
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Hector Yee <he...@gmail.com>.
Whoops I must have used the 1.2 preview and mixed them up.
spark-shell -version shows version 1.2.0
Will update the bug https://issues.apache.org/jira/browse/SPARK-4516 to 1.2
On Thu, Nov 20, 2014 at 11:59 AM, Matei Zaharia <ma...@gmail.com>
wrote:
> Ah, I see. But the spark.shuffle.blockTransferService property doesn't
> exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem?
>
> Matei
>
> On Nov 20, 2014, at 11:50 AM, Hector Yee <he...@gmail.com> wrote:
>
> This is whatever was in http://people.apache.org/~andrewor14/spark-1
> .1.1-rc2/
>
> On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia <ma...@gmail.com>
> wrote:
>
>> Hector, is this a comment on 1.1.1 or on the 1.2 preview?
>>
>> Matei
>>
>> > On Nov 20, 2014, at 11:39 AM, Hector Yee <he...@gmail.com> wrote:
>> >
>> > I think it is a race condition caused by netty deactivating a channel
>> while
>> > it is active.
>> > Switched to nio and it works fine
>> > --conf spark.shuffle.blockTransferService=nio
>> >
>> > On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <he...@gmail.com>
>> wrote:
>> >
>> >> I'm still seeing the fetch failed error and updated
>> >> https://issues.apache.org/jira/browse/SPARK-3633
>> >>
>> >> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <va...@cloudera.com>
>> >> wrote:
>> >>
>> >>> +1 (non-binding)
>> >>>
>> >>> . ran simple things on spark-shell
>> >>> . ran jobs in yarn client & cluster modes, and standalone cluster mode
>> >>>
>> >>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com>
>> wrote:
>> >>>> Please vote on releasing the following candidate as Apache Spark
>> version
>> >>>> 1.1.1.
>> >>>>
>> >>>> This release fixes a number of bugs in Spark 1.1.0. Some of the
>> notable
>> >>> ones
>> >>>> are
>> >>>> - [SPARK-3426] Sort-based shuffle compression settings are
>> incompatible
>> >>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> >>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
>> >>> truncation
>> >>>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>> >>> attached.
>> >>>>
>> >>>> Additionally, this candidate fixes two blockers from the previous RC:
>> >>>> - [SPARK-4434] Cluster mode jar URLs are broken
>> >>>> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
>> >>> spills
>> >>>>
>> >>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>> >>>> http://s.apache.org/p8
>> >>>>
>> >>>> The release files, including signatures, digests, etc can be found
>> at:
>> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>> >>>>
>> >>>> Release artifacts are signed with the following key:
>> >>>> https://people.apache.org/keys/committer/andrewor14.asc
>> >>>>
>> >>>> The staging repository for this release can be found at:
>> >>>>
>> https://repository.apache.org/content/repositories/orgapachespark-1043/
>> >>>>
>> >>>> The documentation corresponding to this release can be found at:
>> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>> >>>>
>> >>>> Please vote on releasing this package as Apache Spark 1.1.1!
>> >>>>
>> >>>> The vote is open until Saturday, November 22, at 23:00 UTC and
>> passes if
>> >>>> a majority of at least 3 +1 PMC votes are cast.
>> >>>> [ ] +1 Release this package as Apache Spark 1.1.1
>> >>>> [ ] -1 Do not release this package because ...
>> >>>>
>> >>>> To learn more about Apache Spark, please see
>> >>>> http://spark.apache.org/
>> >>>>
>> >>>> Cheers,
>> >>>> Andrew
>> >>>>
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >>>> For additional commands, e-mail: dev-help@spark.apache.org
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Marcelo
>> >>>
>> >>> ---------------------------------------------------------------------
>> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> >>> For additional commands, e-mail: dev-help@spark.apache.org
>> >>>
>> >>>
>> >>
>> >>
>> >> --
>> >> Yee Yang Li Hector <http://google.com/+HectorYee>
>> >> *google.com/+HectorYee <http://google.com/+HectorYee>*
>> >>
>> >
>> >
>> >
>> > --
>> > Yee Yang Li Hector <http://google.com/+HectorYee>
>> > *google.com/+HectorYee <http://google.com/+HectorYee>*
>>
>>
>
>
> --
> Yee Yang Li Hector <http://google.com/+HectorYee>
> *google.com/+HectorYee <http://google.com/+HectorYee>*
>
>
>
--
Yee Yang Li Hector <http://google.com/+HectorYee>
*google.com/+HectorYee <http://google.com/+HectorYee>*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Matei Zaharia <ma...@gmail.com>.
Ah, I see. But the spark.shuffle.blockTransferService property doesn't exist in 1.1 (AFAIK) -- what exactly are you doing to get this problem?
Matei
> On Nov 20, 2014, at 11:50 AM, Hector Yee <he...@gmail.com> wrote:
>
> This is whatever was in http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ <http://people.apache.org/~andrewor14/spark-1.1.1-rc2/>
>
> On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia <matei.zaharia@gmail.com <ma...@gmail.com>> wrote:
> Hector, is this a comment on 1.1.1 or on the 1.2 preview?
>
> Matei
>
> > On Nov 20, 2014, at 11:39 AM, Hector Yee <hector.yee@gmail.com <ma...@gmail.com>> wrote:
> >
> > I think it is a race condition caused by netty deactivating a channel while
> > it is active.
> > Switched to nio and it works fine
> > --conf spark.shuffle.blockTransferService=nio
> >
> > On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <hector.yee@gmail.com <ma...@gmail.com>> wrote:
> >
> >> I'm still seeing the fetch failed error and updated
> >> https://issues.apache.org/jira/browse/SPARK-3633 <https://issues.apache.org/jira/browse/SPARK-3633>
> >>
> >> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <vanzin@cloudera.com <ma...@cloudera.com>>
> >> wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> . ran simple things on spark-shell
> >>> . ran jobs in yarn client & cluster modes, and standalone cluster mode
> >>>
> >>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <andrew@databricks.com <ma...@databricks.com>> wrote:
> >>>> Please vote on releasing the following candidate as Apache Spark version
> >>>> 1.1.1.
> >>>>
> >>>> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
> >>> ones
> >>>> are
> >>>> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> >>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
> >>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
> >>> truncation
> >>>> The full list is at http://s.apache.org/z9h <http://s.apache.org/z9h> and in the CHANGES.txt
> >>> attached.
> >>>>
> >>>> Additionally, this candidate fixes two blockers from the previous RC:
> >>>> - [SPARK-4434] Cluster mode jar URLs are broken
> >>>> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
> >>> spills
> >>>>
> >>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> >>>> http://s.apache.org/p8 <http://s.apache.org/p8>
> >>>>
> >>>> The release files, including signatures, digests, etc can be found at:
> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/ <http://people.apache.org/~andrewor14/spark-1.1.1-rc2/>
> >>>>
> >>>> Release artifacts are signed with the following key:
> >>>> https://people.apache.org/keys/committer/andrewor14.asc <https://people.apache.org/keys/committer/andrewor14.asc>
> >>>>
> >>>> The staging repository for this release can be found at:
> >>>> https://repository.apache.org/content/repositories/orgapachespark-1043/ <https://repository.apache.org/content/repositories/orgapachespark-1043/>
> >>>>
> >>>> The documentation corresponding to this release can be found at:
> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/ <http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/>
> >>>>
> >>>> Please vote on releasing this package as Apache Spark 1.1.1!
> >>>>
> >>>> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
> >>>> a majority of at least 3 +1 PMC votes are cast.
> >>>> [ ] +1 Release this package as Apache Spark 1.1.1
> >>>> [ ] -1 Do not release this package because ...
> >>>>
> >>>> To learn more about Apache Spark, please see
> >>>> http://spark.apache.org/ <http://spark.apache.org/>
> >>>>
> >>>> Cheers,
> >>>> Andrew
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> >>>> For additional commands, e-mail: dev-help@spark.apache.org <ma...@spark.apache.org>
> >>>
> >>>
> >>>
> >>> --
> >>> Marcelo
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org <ma...@spark.apache.org>
> >>> For additional commands, e-mail: dev-help@spark.apache.org <ma...@spark.apache.org>
> >>>
> >>>
> >>
> >>
> >> --
> >> Yee Yang Li Hector <http://google.com/+HectorYee <http://google.com/+HectorYee>>
> >> *google.com/+HectorYee <http://google.com/+HectorYee> <http://google.com/+HectorYee <http://google.com/+HectorYee>>*
> >>
> >
> >
> >
> > --
> > Yee Yang Li Hector <http://google.com/+HectorYee <http://google.com/+HectorYee>>
> > *google.com/+HectorYee <http://google.com/+HectorYee> <http://google.com/+HectorYee <http://google.com/+HectorYee>>*
>
>
>
>
> --
> Yee Yang Li Hector <http://google.com/+HectorYee>
> google.com/+HectorYee <http://google.com/+HectorYee>
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Hector Yee <he...@gmail.com>.
This is whatever was in http://people.apache.org/~andrewor14/spark-1
.1.1-rc2/
On Thu, Nov 20, 2014 at 11:48 AM, Matei Zaharia <ma...@gmail.com>
wrote:
> Hector, is this a comment on 1.1.1 or on the 1.2 preview?
>
> Matei
>
> > On Nov 20, 2014, at 11:39 AM, Hector Yee <he...@gmail.com> wrote:
> >
> > I think it is a race condition caused by netty deactivating a channel
> while
> > it is active.
> > Switched to nio and it works fine
> > --conf spark.shuffle.blockTransferService=nio
> >
> > On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <he...@gmail.com>
> wrote:
> >
> >> I'm still seeing the fetch failed error and updated
> >> https://issues.apache.org/jira/browse/SPARK-3633
> >>
> >> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <va...@cloudera.com>
> >> wrote:
> >>
> >>> +1 (non-binding)
> >>>
> >>> . ran simple things on spark-shell
> >>> . ran jobs in yarn client & cluster modes, and standalone cluster mode
> >>>
> >>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com>
> wrote:
> >>>> Please vote on releasing the following candidate as Apache Spark
> version
> >>>> 1.1.1.
> >>>>
> >>>> This release fixes a number of bugs in Spark 1.1.0. Some of the
> notable
> >>> ones
> >>>> are
> >>>> - [SPARK-3426] Sort-based shuffle compression settings are
> incompatible
> >>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
> >>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
> >>> truncation
> >>>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> >>> attached.
> >>>>
> >>>> Additionally, this candidate fixes two blockers from the previous RC:
> >>>> - [SPARK-4434] Cluster mode jar URLs are broken
> >>>> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
> >>> spills
> >>>>
> >>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> >>>> http://s.apache.org/p8
> >>>>
> >>>> The release files, including signatures, digests, etc can be found at:
> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
> >>>>
> >>>> Release artifacts are signed with the following key:
> >>>> https://people.apache.org/keys/committer/andrewor14.asc
> >>>>
> >>>> The staging repository for this release can be found at:
> >>>>
> https://repository.apache.org/content/repositories/orgapachespark-1043/
> >>>>
> >>>> The documentation corresponding to this release can be found at:
> >>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
> >>>>
> >>>> Please vote on releasing this package as Apache Spark 1.1.1!
> >>>>
> >>>> The vote is open until Saturday, November 22, at 23:00 UTC and passes
> if
> >>>> a majority of at least 3 +1 PMC votes are cast.
> >>>> [ ] +1 Release this package as Apache Spark 1.1.1
> >>>> [ ] -1 Do not release this package because ...
> >>>>
> >>>> To learn more about Apache Spark, please see
> >>>> http://spark.apache.org/
> >>>>
> >>>> Cheers,
> >>>> Andrew
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>
> >>>
> >>>
> >>> --
> >>> Marcelo
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> >>> For additional commands, e-mail: dev-help@spark.apache.org
> >>>
> >>>
> >>
> >>
> >> --
> >> Yee Yang Li Hector <http://google.com/+HectorYee>
> >> *google.com/+HectorYee <http://google.com/+HectorYee>*
> >>
> >
> >
> >
> > --
> > Yee Yang Li Hector <http://google.com/+HectorYee>
> > *google.com/+HectorYee <http://google.com/+HectorYee>*
>
>
--
Yee Yang Li Hector <http://google.com/+HectorYee>
*google.com/+HectorYee <http://google.com/+HectorYee>*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Matei Zaharia <ma...@gmail.com>.
Hector, is this a comment on 1.1.1 or on the 1.2 preview?
Matei
> On Nov 20, 2014, at 11:39 AM, Hector Yee <he...@gmail.com> wrote:
>
> I think it is a race condition caused by netty deactivating a channel while
> it is active.
> Switched to nio and it works fine
> --conf spark.shuffle.blockTransferService=nio
>
> On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <he...@gmail.com> wrote:
>
>> I'm still seeing the fetch failed error and updated
>> https://issues.apache.org/jira/browse/SPARK-3633
>>
>> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <va...@cloudera.com>
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> . ran simple things on spark-shell
>>> . ran jobs in yarn client & cluster modes, and standalone cluster mode
>>>
>>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com> wrote:
>>>> Please vote on releasing the following candidate as Apache Spark version
>>>> 1.1.1.
>>>>
>>>> This release fixes a number of bugs in Spark 1.1.0. Some of the notable
>>> ones
>>>> are
>>>> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
>>>> - [SPARK-3948] Stream corruption issues in sort-based shuffle
>>>> - [SPARK-4107] Incorrect handling of Channel.read() led to data
>>> truncation
>>>> The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>>> attached.
>>>>
>>>> Additionally, this candidate fixes two blockers from the previous RC:
>>>> - [SPARK-4434] Cluster mode jar URLs are broken
>>>> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
>>> spills
>>>>
>>>> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>>>> http://s.apache.org/p8
>>>>
>>>> The release files, including signatures, digests, etc can be found at:
>>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>>>>
>>>> Release artifacts are signed with the following key:
>>>> https://people.apache.org/keys/committer/andrewor14.asc
>>>>
>>>> The staging repository for this release can be found at:
>>>> https://repository.apache.org/content/repositories/orgapachespark-1043/
>>>>
>>>> The documentation corresponding to this release can be found at:
>>>> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>>>>
>>>> Please vote on releasing this package as Apache Spark 1.1.1!
>>>>
>>>> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
>>>> a majority of at least 3 +1 PMC votes are cast.
>>>> [ ] +1 Release this package as Apache Spark 1.1.1
>>>> [ ] -1 Do not release this package because ...
>>>>
>>>> To learn more about Apache Spark, please see
>>>> http://spark.apache.org/
>>>>
>>>> Cheers,
>>>> Andrew
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: dev-help@spark.apache.org
>>>
>>>
>>
>>
>> --
>> Yee Yang Li Hector <http://google.com/+HectorYee>
>> *google.com/+HectorYee <http://google.com/+HectorYee>*
>>
>
>
>
> --
> Yee Yang Li Hector <http://google.com/+HectorYee>
> *google.com/+HectorYee <http://google.com/+HectorYee>*
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Hector Yee <he...@gmail.com>.
I think it is a race condition caused by netty deactivating a channel while
it is active.
Switched to nio and it works fine
--conf spark.shuffle.blockTransferService=nio
On Thu, Nov 20, 2014 at 10:44 AM, Hector Yee <he...@gmail.com> wrote:
> I'm still seeing the fetch failed error and updated
> https://issues.apache.org/jira/browse/SPARK-3633
>
> On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> +1 (non-binding)
>>
>> . ran simple things on spark-shell
>> . ran jobs in yarn client & cluster modes, and standalone cluster mode
>>
>> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com> wrote:
>> > Please vote on releasing the following candidate as Apache Spark version
>> > 1.1.1.
>> >
>> > This release fixes a number of bugs in Spark 1.1.0. Some of the notable
>> ones
>> > are
>> > - [SPARK-3426] Sort-based shuffle compression settings are incompatible
>> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
>> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
>> truncation
>> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
>> attached.
>> >
>> > Additionally, this candidate fixes two blockers from the previous RC:
>> > - [SPARK-4434] Cluster mode jar URLs are broken
>> > - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
>> spills
>> >
>> > The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
>> > http://s.apache.org/p8
>> >
>> > The release files, including signatures, digests, etc can be found at:
>> > http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>> >
>> > Release artifacts are signed with the following key:
>> > https://people.apache.org/keys/committer/andrewor14.asc
>> >
>> > The staging repository for this release can be found at:
>> > https://repository.apache.org/content/repositories/orgapachespark-1043/
>> >
>> > The documentation corresponding to this release can be found at:
>> > http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>> >
>> > Please vote on releasing this package as Apache Spark 1.1.1!
>> >
>> > The vote is open until Saturday, November 22, at 23:00 UTC and passes if
>> > a majority of at least 3 +1 PMC votes are cast.
>> > [ ] +1 Release this package as Apache Spark 1.1.1
>> > [ ] -1 Do not release this package because ...
>> >
>> > To learn more about Apache Spark, please see
>> > http://spark.apache.org/
>> >
>> > Cheers,
>> > Andrew
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
>
> --
> Yee Yang Li Hector <http://google.com/+HectorYee>
> *google.com/+HectorYee <http://google.com/+HectorYee>*
>
--
Yee Yang Li Hector <http://google.com/+HectorYee>
*google.com/+HectorYee <http://google.com/+HectorYee>*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Hector Yee <he...@gmail.com>.
I'm still seeing the fetch failed error and updated
https://issues.apache.org/jira/browse/SPARK-3633
On Thu, Nov 20, 2014 at 10:21 AM, Marcelo Vanzin <va...@cloudera.com>
wrote:
> +1 (non-binding)
>
> . ran simple things on spark-shell
> . ran jobs in yarn client & cluster modes, and standalone cluster mode
>
> On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com> wrote:
> > Please vote on releasing the following candidate as Apache Spark version
> > 1.1.1.
> >
> > This release fixes a number of bugs in Spark 1.1.0. Some of the notable
> ones
> > are
> > - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> > - [SPARK-3948] Stream corruption issues in sort-based shuffle
> > - [SPARK-4107] Incorrect handling of Channel.read() led to data
> truncation
> > The full list is at http://s.apache.org/z9h and in the CHANGES.txt
> attached.
> >
> > Additionally, this candidate fixes two blockers from the previous RC:
> > - [SPARK-4434] Cluster mode jar URLs are broken
> > - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle
> spills
> >
> > The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> > http://s.apache.org/p8
> >
> > The release files, including signatures, digests, etc can be found at:
> > http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/andrewor14.asc
> >
> > The staging repository for this release can be found at:
> > https://repository.apache.org/content/repositories/orgapachespark-1043/
> >
> > The documentation corresponding to this release can be found at:
> > http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
> >
> > Please vote on releasing this package as Apache Spark 1.1.1!
> >
> > The vote is open until Saturday, November 22, at 23:00 UTC and passes if
> > a majority of at least 3 +1 PMC votes are cast.
> > [ ] +1 Release this package as Apache Spark 1.1.1
> > [ ] -1 Do not release this package because ...
> >
> > To learn more about Apache Spark, please see
> > http://spark.apache.org/
> >
> > Cheers,
> > Andrew
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
--
Yee Yang Li Hector <http://google.com/+HectorYee>
*google.com/+HectorYee <http://google.com/+HectorYee>*
Re: [VOTE] Release Apache Spark 1.1.1 (RC2)
Posted by Marcelo Vanzin <va...@cloudera.com>.
+1 (non-binding)
. ran simple things on spark-shell
. ran jobs in yarn client & cluster modes, and standalone cluster mode
On Wed, Nov 19, 2014 at 2:51 PM, Andrew Or <an...@databricks.com> wrote:
> Please vote on releasing the following candidate as Apache Spark version
> 1.1.1.
>
> This release fixes a number of bugs in Spark 1.1.0. Some of the notable ones
> are
> - [SPARK-3426] Sort-based shuffle compression settings are incompatible
> - [SPARK-3948] Stream corruption issues in sort-based shuffle
> - [SPARK-4107] Incorrect handling of Channel.read() led to data truncation
> The full list is at http://s.apache.org/z9h and in the CHANGES.txt attached.
>
> Additionally, this candidate fixes two blockers from the previous RC:
> - [SPARK-4434] Cluster mode jar URLs are broken
> - [SPARK-4480][SPARK-4467] Too many open files exception from shuffle spills
>
> The tag to be voted on is v1.1.1-rc2 (commit 3693ae5d):
> http://s.apache.org/p8
>
> The release files, including signatures, digests, etc can be found at:
> http://people.apache.org/~andrewor14/spark-1.1.1-rc2/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/andrewor14.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1043/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~andrewor14/spark-1.1.1-rc2-docs/
>
> Please vote on releasing this package as Apache Spark 1.1.1!
>
> The vote is open until Saturday, November 22, at 23:00 UTC and passes if
> a majority of at least 3 +1 PMC votes are cast.
> [ ] +1 Release this package as Apache Spark 1.1.1
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see
> http://spark.apache.org/
>
> Cheers,
> Andrew
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org