You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Patrick Wendell <pw...@gmail.com> on 2014/01/19 23:15:03 UTC

[VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Please vote on releasing the following candidate as Apache Spark
(incubating) version 0.9.0.

A draft of the release notes along with the changes file is attached
to this e-mail.

The tag to be voted on is v0.9.0-incubating (commit a7760eff):
https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d

The release files, including signatures, digests, etc can be found at:
http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1004/

The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/

Please vote on releasing this package as Apache Spark 0.9.0-incubating!

The vote is open until Wednesday, January 22, at 22:15 UTC and passes
if a majority of at least 3 +1 PPMC votes are cast.

[ ] +1 Release this package as Apache Spark 0.9.0-incubating
[ ] -1 Do not release this package because ...

To learn more about Apache Spark, please see
http://spark.incubator.apache.org/

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Patrick Wendell <pw...@gmail.com>.
Ah oops Andy I see - thought you were discussing something different.

On Tue, Jan 21, 2014 at 1:52 PM, Patrick Wendell <pw...@gmail.com> wrote:
> This isn't a new RC or a result - I just tried to make an new vote
> thread and the mail clients get confused and merge it in with the old
> thread :(
>
> On Tue, Jan 21, 2014 at 1:04 PM, Andy Konwinski <an...@gmail.com> wrote:
>> I think we might need to change the subject to [RESULT][VOTE] or something
>> so that the apache scripts can tell when a vote thread ends, even if it's
>> simply being replaced by another vote thread for rc{n+1}.
>>
>>
>> On Tue, Jan 21, 2014 at 3:15 AM, Patrick Wendell <pw...@gmail.com> wrote:
>>
>>> This vote is cancelled in favor of rc4.
>>>
>>>
>>> On Mon, Jan 20, 2014 at 4:05 PM, Ewen Cheslack-Postava <me...@ewencp.org>wrote:
>>>
>>>> I'm using sbt and also tried doubling all the values in the sbt/sbt
>>>> script. Still fails during the tests, and leaves a sbt.ForkMain process
>>>> running (and eating up 100% CPU...).
>>>>
>>>> I can break this out into a separate issue/thread, I just figured it
>>>> might be relevant since I'd imagine this is a fairly common setup (Macbook
>>>> Air).
>>>>
>>>> -Ewen
>>>>
>>>>   Reynold Xin <rx...@databricks.com>
>>>>  January 20, 2014 10:15 AM
>>>> That's a perm gen issue - you need to adjust the perm gem size. In sbt it
>>>> should've been set automatically, but I think for Maven, you need to set
>>>> the maven opts, which is documented in the build instructions.
>>>>
>>>>
>>>>
>>>>   Ewen Cheslack-Postava <ew...@fastmail.fm>
>>>>  January 19, 2014 11:35 PM
>>>>  I can't get the tests to run on a Mac, 10.7.5, java -version output:
>>>>
>>>> java version "1.6.0_65"
>>>> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>>>>
>>>> For reference, Spark 0.8.* build and test find on the same configuration.
>>>> 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not sure what it's
>>>> running at that time since all the tests in PrimitiveVectorSuite seem to
>>>> have finished:
>>>>
>>>> [info] PrimitiveVectorSuite:
>>>> [info] - primitive value (4 milliseconds)
>>>> [info] - non-primitive value (5 milliseconds)
>>>> [info] - ideal growth (4 milliseconds)
>>>> [info] - ideal size (5 milliseconds)
>>>> [info] - resizing (6 milliseconds)
>>>> [ERROR] [01/19/2014 23:16:27.508] [spark-akka.actor.default-dispatcher-4]
>>>> [ActorSystem(spark)] exception while executing timer task
>>>> org.apache.spark.SparkException: Error sending message to
>>>> BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, localhost,
>>>> 51634, 0))]
>>>>     at
>>>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>>>>     at
>>>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>>>     at org.apache.spark.storage.BlockManager.org
>>>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>>>     at
>>>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>>>>     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>>>     at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>>>>     at
>>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
>>>>     at
>>>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
>>>>     at
>>>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
>>>>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>>>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>>>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>>>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>>>>     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
>>>>     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
>>>>     at
>>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
>>>>     at
>>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>>>     at
>>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>>>     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
>>>>     at
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
>>>>     at
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
>>>>     at
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>>>     at
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>>>     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
>>>>     at
>>>> akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
>>>>     at
>>>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>>>     at
>>>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>>>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>>>     at
>>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>>>>     at
>>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>>>>     at
>>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>>>     at
>>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>>>     at
>>>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>>>     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>>>>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
>>>>     at
>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>>>>     at
>>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>>     at
>>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>>     at
>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>>     at
>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>> Caused by: akka.pattern.AskTimeoutException:
>>>> Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had
>>>> already been terminated.
>>>>     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>>>>     at
>>>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
>>>>     ... 39 more
>>>>
>>>> [ERROR] [01/19/2014 23:20:03.131]
>>>> [sparkWorker2-akka.actor.default-dispatcher-19] [ActorSystem(sparkWorker2)]
>>>> Uncaught fatal error from thread
>>>> [sparkWorker2-akka.actor.default-dispatcher-19] shutting down ActorSystem
>>>> [sparkWorker2]
>>>> java.lang.OutOfMemoryError: PermGen space
>>>>
>>>> [ERROR] [01/19/2014 23:20:05.226]
>>>> [sparkWorker2-akka.actor.default-dispatcher-2] [ActorSystem(sparkWorker2)]
>>>> Uncaught fatal error from thread
>>>> [sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem
>>>> [sparkWorker2]
>>>> java.lang.OutOfMemoryError: PermGen space
>>>>
>>>> [... snip ... followed by a bunch more like these ]
>>>>
>>>> It looks like it's just a heap size issue. But since it's leaving a
>>>> process running that has -Xmx3g explicitly specified on the command line
>>>> I'm surprised it works for everyone else but consistently fails for me.
>>>> Almost certainly not a blocker, but some memory setting in the tests might
>>>> need adjustment.
>>>>
>>>> -Ewen
>>>>   Henry Saputra <he...@gmail.com>
>>>>  January 19, 2014 10:07 PM
>>>> Ah yes, makes sense, thanks!
>>>>
>>>> - Henry
>>>>   Patrick Wendell <pw...@gmail.com>
>>>>  January 19, 2014 10:01 PM
>>>> Eventually the notes get posted on the apache website. I attached them
>>>> to this e-mail so that people can get a sense of what is in the
>>>> release before they vote on it.
>>>>   Henry Saputra <he...@gmail.com>
>>>>  January 19, 2014 9:57 PM
>>>> Hi Patrick, quick question, where are you planning to add the release
>>>> notes?
>>>> I dont think it is part of the source, is it?
>>>>
>>>> - Henry
>>>>
>>>>
>>>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Patrick Wendell <pw...@gmail.com>.
This isn't a new RC or a result - I just tried to make an new vote
thread and the mail clients get confused and merge it in with the old
thread :(

On Tue, Jan 21, 2014 at 1:04 PM, Andy Konwinski <an...@gmail.com> wrote:
> I think we might need to change the subject to [RESULT][VOTE] or something
> so that the apache scripts can tell when a vote thread ends, even if it's
> simply being replaced by another vote thread for rc{n+1}.
>
>
> On Tue, Jan 21, 2014 at 3:15 AM, Patrick Wendell <pw...@gmail.com> wrote:
>
>> This vote is cancelled in favor of rc4.
>>
>>
>> On Mon, Jan 20, 2014 at 4:05 PM, Ewen Cheslack-Postava <me...@ewencp.org>wrote:
>>
>>> I'm using sbt and also tried doubling all the values in the sbt/sbt
>>> script. Still fails during the tests, and leaves a sbt.ForkMain process
>>> running (and eating up 100% CPU...).
>>>
>>> I can break this out into a separate issue/thread, I just figured it
>>> might be relevant since I'd imagine this is a fairly common setup (Macbook
>>> Air).
>>>
>>> -Ewen
>>>
>>>   Reynold Xin <rx...@databricks.com>
>>>  January 20, 2014 10:15 AM
>>> That's a perm gen issue - you need to adjust the perm gem size. In sbt it
>>> should've been set automatically, but I think for Maven, you need to set
>>> the maven opts, which is documented in the build instructions.
>>>
>>>
>>>
>>>   Ewen Cheslack-Postava <ew...@fastmail.fm>
>>>  January 19, 2014 11:35 PM
>>>  I can't get the tests to run on a Mac, 10.7.5, java -version output:
>>>
>>> java version "1.6.0_65"
>>> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>>>
>>> For reference, Spark 0.8.* build and test find on the same configuration.
>>> 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not sure what it's
>>> running at that time since all the tests in PrimitiveVectorSuite seem to
>>> have finished:
>>>
>>> [info] PrimitiveVectorSuite:
>>> [info] - primitive value (4 milliseconds)
>>> [info] - non-primitive value (5 milliseconds)
>>> [info] - ideal growth (4 milliseconds)
>>> [info] - ideal size (5 milliseconds)
>>> [info] - resizing (6 milliseconds)
>>> [ERROR] [01/19/2014 23:16:27.508] [spark-akka.actor.default-dispatcher-4]
>>> [ActorSystem(spark)] exception while executing timer task
>>> org.apache.spark.SparkException: Error sending message to
>>> BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, localhost,
>>> 51634, 0))]
>>>     at
>>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>>>     at
>>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>>     at org.apache.spark.storage.BlockManager.org
>>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>>     at
>>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>>>     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
>>>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>>>     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
>>>     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>>     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>>     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>>     at
>>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>>     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>>>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
>>>     at
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> Caused by: akka.pattern.AskTimeoutException:
>>> Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had
>>> already been terminated.
>>>     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>>>     at
>>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
>>>     ... 39 more
>>>
>>> [ERROR] [01/19/2014 23:20:03.131]
>>> [sparkWorker2-akka.actor.default-dispatcher-19] [ActorSystem(sparkWorker2)]
>>> Uncaught fatal error from thread
>>> [sparkWorker2-akka.actor.default-dispatcher-19] shutting down ActorSystem
>>> [sparkWorker2]
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> [ERROR] [01/19/2014 23:20:05.226]
>>> [sparkWorker2-akka.actor.default-dispatcher-2] [ActorSystem(sparkWorker2)]
>>> Uncaught fatal error from thread
>>> [sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem
>>> [sparkWorker2]
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> [... snip ... followed by a bunch more like these ]
>>>
>>> It looks like it's just a heap size issue. But since it's leaving a
>>> process running that has -Xmx3g explicitly specified on the command line
>>> I'm surprised it works for everyone else but consistently fails for me.
>>> Almost certainly not a blocker, but some memory setting in the tests might
>>> need adjustment.
>>>
>>> -Ewen
>>>   Henry Saputra <he...@gmail.com>
>>>  January 19, 2014 10:07 PM
>>> Ah yes, makes sense, thanks!
>>>
>>> - Henry
>>>   Patrick Wendell <pw...@gmail.com>
>>>  January 19, 2014 10:01 PM
>>> Eventually the notes get posted on the apache website. I attached them
>>> to this e-mail so that people can get a sense of what is in the
>>> release before they vote on it.
>>>   Henry Saputra <he...@gmail.com>
>>>  January 19, 2014 9:57 PM
>>> Hi Patrick, quick question, where are you planning to add the release
>>> notes?
>>> I dont think it is part of the source, is it?
>>>
>>> - Henry
>>>
>>>
>>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Andy Konwinski <an...@gmail.com>.
I think we might need to change the subject to [RESULT][VOTE] or something
so that the apache scripts can tell when a vote thread ends, even if it's
simply being replaced by another vote thread for rc{n+1}.


On Tue, Jan 21, 2014 at 3:15 AM, Patrick Wendell <pw...@gmail.com> wrote:

> This vote is cancelled in favor of rc4.
>
>
> On Mon, Jan 20, 2014 at 4:05 PM, Ewen Cheslack-Postava <me...@ewencp.org>wrote:
>
>> I'm using sbt and also tried doubling all the values in the sbt/sbt
>> script. Still fails during the tests, and leaves a sbt.ForkMain process
>> running (and eating up 100% CPU...).
>>
>> I can break this out into a separate issue/thread, I just figured it
>> might be relevant since I'd imagine this is a fairly common setup (Macbook
>> Air).
>>
>> -Ewen
>>
>>   Reynold Xin <rx...@databricks.com>
>>  January 20, 2014 10:15 AM
>> That's a perm gen issue - you need to adjust the perm gem size. In sbt it
>> should've been set automatically, but I think for Maven, you need to set
>> the maven opts, which is documented in the build instructions.
>>
>>
>>
>>   Ewen Cheslack-Postava <ew...@fastmail.fm>
>>  January 19, 2014 11:35 PM
>>  I can't get the tests to run on a Mac, 10.7.5, java -version output:
>>
>> java version "1.6.0_65"
>> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>>
>> For reference, Spark 0.8.* build and test find on the same configuration.
>> 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not sure what it's
>> running at that time since all the tests in PrimitiveVectorSuite seem to
>> have finished:
>>
>> [info] PrimitiveVectorSuite:
>> [info] - primitive value (4 milliseconds)
>> [info] - non-primitive value (5 milliseconds)
>> [info] - ideal growth (4 milliseconds)
>> [info] - ideal size (5 milliseconds)
>> [info] - resizing (6 milliseconds)
>> [ERROR] [01/19/2014 23:16:27.508] [spark-akka.actor.default-dispatcher-4]
>> [ActorSystem(spark)] exception while executing timer task
>> org.apache.spark.SparkException: Error sending message to
>> BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, localhost,
>> 51634, 0))]
>>     at
>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>>     at
>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>     at org.apache.spark.storage.BlockManager.org
>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>     at
>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>>     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>     at
>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>>     at
>> akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
>>     at
>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
>>     at
>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
>>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>>     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
>>     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
>>     at
>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
>>     at
>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>     at
>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
>>     at
>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
>>     at
>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
>>     at
>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>     at
>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
>>     at
>> akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
>>     at
>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>     at
>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>     at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>>     at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>>     at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>     at
>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>     at
>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
>>     at
>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>>     at
>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>     at
>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>     at
>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>     at
>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>> Caused by: akka.pattern.AskTimeoutException:
>> Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had
>> already been terminated.
>>     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>>     at
>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
>>     ... 39 more
>>
>> [ERROR] [01/19/2014 23:20:03.131]
>> [sparkWorker2-akka.actor.default-dispatcher-19] [ActorSystem(sparkWorker2)]
>> Uncaught fatal error from thread
>> [sparkWorker2-akka.actor.default-dispatcher-19] shutting down ActorSystem
>> [sparkWorker2]
>> java.lang.OutOfMemoryError: PermGen space
>>
>> [ERROR] [01/19/2014 23:20:05.226]
>> [sparkWorker2-akka.actor.default-dispatcher-2] [ActorSystem(sparkWorker2)]
>> Uncaught fatal error from thread
>> [sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem
>> [sparkWorker2]
>> java.lang.OutOfMemoryError: PermGen space
>>
>> [... snip ... followed by a bunch more like these ]
>>
>> It looks like it's just a heap size issue. But since it's leaving a
>> process running that has -Xmx3g explicitly specified on the command line
>> I'm surprised it works for everyone else but consistently fails for me.
>> Almost certainly not a blocker, but some memory setting in the tests might
>> need adjustment.
>>
>> -Ewen
>>   Henry Saputra <he...@gmail.com>
>>  January 19, 2014 10:07 PM
>> Ah yes, makes sense, thanks!
>>
>> - Henry
>>   Patrick Wendell <pw...@gmail.com>
>>  January 19, 2014 10:01 PM
>> Eventually the notes get posted on the apache website. I attached them
>> to this e-mail so that people can get a sense of what is in the
>> release before they vote on it.
>>   Henry Saputra <he...@gmail.com>
>>  January 19, 2014 9:57 PM
>> Hi Patrick, quick question, where are you planning to add the release
>> notes?
>> I dont think it is part of the source, is it?
>>
>> - Henry
>>
>>
>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Patrick Wendell <pw...@gmail.com>.
This vote is cancelled in favor of rc4.


On Mon, Jan 20, 2014 at 4:05 PM, Ewen Cheslack-Postava <me...@ewencp.org>wrote:

> I'm using sbt and also tried doubling all the values in the sbt/sbt
> script. Still fails during the tests, and leaves a sbt.ForkMain process
> running (and eating up 100% CPU...).
>
> I can break this out into a separate issue/thread, I just figured it might
> be relevant since I'd imagine this is a fairly common setup (Macbook Air).
>
> -Ewen
>
>   Reynold Xin <rx...@databricks.com>
>  January 20, 2014 10:15 AM
> That's a perm gen issue - you need to adjust the perm gem size. In sbt it
> should've been set automatically, but I think for Maven, you need to set
> the maven opts, which is documented in the build instructions.
>
>
>
>   Ewen Cheslack-Postava <ew...@fastmail.fm>
>  January 19, 2014 11:35 PM
>  I can't get the tests to run on a Mac, 10.7.5, java -version output:
>
> java version "1.6.0_65"
> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>
> For reference, Spark 0.8.* build and test find on the same configuration.
> 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not sure what it's
> running at that time since all the tests in PrimitiveVectorSuite seem to
> have finished:
>
> [info] PrimitiveVectorSuite:
> [info] - primitive value (4 milliseconds)
> [info] - non-primitive value (5 milliseconds)
> [info] - ideal growth (4 milliseconds)
> [info] - ideal size (5 milliseconds)
> [info] - resizing (6 milliseconds)
> [ERROR] [01/19/2014 23:16:27.508] [spark-akka.actor.default-dispatcher-4]
> [ActorSystem(spark)] exception while executing timer task
> org.apache.spark.SparkException: Error sending message to
> BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, localhost,
> 51634, 0))]
>     at
> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>     at
> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>     at org.apache.spark.storage.BlockManager.org
> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>     at
> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>     at
> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>     at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
>     at
> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
>     at
> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
>     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
>     at
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
>     at
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>     at
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
>     at
> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>     at
> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>     at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
>     at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>     at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>     at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>     at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had
> already been terminated.
>     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>     at
> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
>     ... 39 more
>
> [ERROR] [01/19/2014 23:20:03.131]
> [sparkWorker2-akka.actor.default-dispatcher-19] [ActorSystem(sparkWorker2)]
> Uncaught fatal error from thread
> [sparkWorker2-akka.actor.default-dispatcher-19] shutting down ActorSystem
> [sparkWorker2]
> java.lang.OutOfMemoryError: PermGen space
>
> [ERROR] [01/19/2014 23:20:05.226]
> [sparkWorker2-akka.actor.default-dispatcher-2] [ActorSystem(sparkWorker2)]
> Uncaught fatal error from thread
> [sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem
> [sparkWorker2]
> java.lang.OutOfMemoryError: PermGen space
>
> [... snip ... followed by a bunch more like these ]
>
> It looks like it's just a heap size issue. But since it's leaving a
> process running that has -Xmx3g explicitly specified on the command line
> I'm surprised it works for everyone else but consistently fails for me.
> Almost certainly not a blocker, but some memory setting in the tests might
> need adjustment.
>
> -Ewen
>   Henry Saputra <he...@gmail.com>
>  January 19, 2014 10:07 PM
> Ah yes, makes sense, thanks!
>
> - Henry
>   Patrick Wendell <pw...@gmail.com>
>  January 19, 2014 10:01 PM
> Eventually the notes get posted on the apache website. I attached them
> to this e-mail so that people can get a sense of what is in the
> release before they vote on it.
>   Henry Saputra <he...@gmail.com>
>  January 19, 2014 9:57 PM
> Hi Patrick, quick question, where are you planning to add the release
> notes?
> I dont think it is part of the source, is it?
>
> - Henry
>
>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Ewen Cheslack-Postava <me...@ewencp.org>.
I'm using sbt and also tried doubling all the values in the sbt/sbt 
script. Still fails during the tests, and leaves a sbt.ForkMain process 
running (and eating up 100% CPU...).

I can break this out into a separate issue/thread, I just figured it 
might be relevant since I'd imagine this is a fairly common setup 
(Macbook Air).

-Ewen
> Reynold Xin <ma...@databricks.com>
> January 20, 2014 10:15 AM
> That's a perm gen issue - you need to adjust the perm gem size. In sbt 
> it should've been set automatically, but I think for Maven, you need 
> to set the maven opts, which is documented in the build instructions.
>
>
>
> Ewen Cheslack-Postava <ma...@fastmail.fm>
> January 19, 2014 11:35 PM
> I can't get the tests to run on a Mac, 10.7.5, java -version output:
>
> java version "1.6.0_65"
> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>
> For reference, Spark 0.8.* build and test find on the same 
> configuration. 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not 
> sure what it's running at that time since all the tests in 
> PrimitiveVectorSuite seem to have finished:
>
> [info] PrimitiveVectorSuite:
> [info] - primitive value (4 milliseconds)
> [info] - non-primitive value (5 milliseconds)
> [info] - ideal growth (4 milliseconds)
> [info] - ideal size (5 milliseconds)
> [info] - resizing (6 milliseconds)
> [ERROR] [01/19/2014 23:16:27.508] 
> [spark-akka.actor.default-dispatcher-4] [ActorSystem(spark)] exception 
> while executing timer task
> org.apache.spark.SparkException: Error sending message to 
> BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, 
> localhost, 51634, 0))]
>     at 
> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>     at 
> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>     at 
> org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>     at 
> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>     at 
> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>     at 
> akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
>     at 
> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
>     at 
> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
>     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
>     at 
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
>     at 
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>     at 
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
>     at 
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
>     at 
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
>     at 
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>     at 
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
>     at 
> akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
>     at 
> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>     at 
> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>     at 
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>     at 
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>     at 
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>     at 
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>     at 
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
>     at 
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>     at 
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>     at 
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>     at 
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>     at 
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: akka.pattern.AskTimeoutException: 
> Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had 
> already been terminated.
>     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>     at 
> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
>     ... 39 more
>
> [ERROR] [01/19/2014 23:20:03.131] 
> [sparkWorker2-akka.actor.default-dispatcher-19] 
> [ActorSystem(sparkWorker2)] Uncaught fatal error from thread 
> [sparkWorker2-akka.actor.default-dispatcher-19] shutting down 
> ActorSystem [sparkWorker2]
> java.lang.OutOfMemoryError: PermGen space
>
> [ERROR] [01/19/2014 23:20:05.226] 
> [sparkWorker2-akka.actor.default-dispatcher-2] 
> [ActorSystem(sparkWorker2)] Uncaught fatal error from thread 
> [sparkWorker2-akka.actor.default-dispatcher-2] shutting down 
> ActorSystem [sparkWorker2]
> java.lang.OutOfMemoryError: PermGen space
>
> [... snip ... followed by a bunch more like these ]
>
> It looks like it's just a heap size issue. But since it's leaving a 
> process running that has -Xmx3g explicitly specified on the command 
> line I'm surprised it works for everyone else but consistently fails 
> for me. Almost certainly not a blocker, but some memory setting in the 
> tests might need adjustment.
>
> -Ewen
> Henry Saputra <ma...@gmail.com>
> January 19, 2014 10:07 PM
> Ah yes, makes sense, thanks!
>
> - Henry
> Patrick Wendell <ma...@gmail.com>
> January 19, 2014 10:01 PM
> Eventually the notes get posted on the apache website. I attached them
> to this e-mail so that people can get a sense of what is in the
> release before they vote on it.
> Henry Saputra <ma...@gmail.com>
> January 19, 2014 9:57 PM
> Hi Patrick, quick question, where are you planning to add the release 
> notes?
> I dont think it is part of the source, is it?
>
> - Henry

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Nan Zhu <zh...@gmail.com>.
running the test cases, always block after passing “Interact with Files” in ReplSuite.scala

I checked the code, it is test("local-cluster mode") {

What happened here?  

--  
Nan Zhu


On Monday, January 20, 2014 at 1:15 PM, Reynold Xin wrote:

> That's a perm gen issue - you need to adjust the perm gem size. In sbt it should've been set automatically, but I think for Maven, you need to set the maven opts, which is documented in the build instructions.  
>  
>  
> On Sun, Jan 19, 2014 at 11:35 PM, Ewen Cheslack-Postava <ewencp@fastmail.fm (mailto:ewencp@fastmail.fm)> wrote:
> > I can't get the tests to run on a Mac, 10.7.5, java -version output:
> >  
> > java version "1.6.0_65"
> > Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
> > Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
> >  
> > For reference, Spark 0.8.* build and test find on the same configuration. 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not sure what it's running at that time since all the tests in PrimitiveVectorSuite seem to have finished:
> >  
> > [info] PrimitiveVectorSuite:
> > [info] - primitive value (4 milliseconds)
> > [info] - non-primitive value (5 milliseconds)
> > [info] - ideal growth (4 milliseconds)
> > [info] - ideal size (5 milliseconds)
> > [info] - resizing (6 milliseconds)
> > [ERROR] [01/19/2014 23:16:27.508] [spark-akka.actor.default-dispatcher-4] [ActorSystem(spark)] exception while executing timer task
> > org.apache.spark.SparkException: Error sending message to BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, localhost, 51634, 0))]
> >     at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
> >     at org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
> >     at org.apache.spark.storage.BlockManager.org (http://org.apache.spark.storage.BlockManager.org)$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
> >     at org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
> >     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
> >     at akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
> >     at akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
> >     at akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
> >     at akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
> >     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> >     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> >     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> >     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> >     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
> >     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
> >     at akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
> >     at akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
> >     at akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
> >     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
> >     at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
> >     at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
> >     at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
> >     at akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
> >     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
> >     at akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
> >     at akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
> >     at akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
> >     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> >     at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> >     at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> >     at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> >     at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> >     at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> >     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> >     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
> >     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> >     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> >     at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> >     at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> >     at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > Caused by: akka.pattern.AskTimeoutException: Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had already been terminated.
> >     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
> >     at org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
> >     ... 39 more
> >  
> > [ERROR] [01/19/2014 23:20:03.131] [sparkWorker2-akka.actor.default-dispatcher-19] [ActorSystem(sparkWorker2)] Uncaught fatal error from thread [sparkWorker2-akka.actor.default-dispatcher-19] shutting down ActorSystem [sparkWorker2]
> > java.lang.OutOfMemoryError: PermGen space
> >  
> > [ERROR] [01/19/2014 23:20:05.226] [sparkWorker2-akka.actor.default-dispatcher-2] [ActorSystem(sparkWorker2)] Uncaught fatal error from thread [sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem [sparkWorker2]
> > java.lang.OutOfMemoryError: PermGen space
> >  
> > [... snip ... followed by a bunch more like these ]
> >  
> > It looks like it's just a heap size issue. But since it's leaving a process running that has -Xmx3g explicitly specified on the command line I'm surprised it works for everyone else but consistently fails for me. Almost certainly not a blocker, but some memory setting in the tests might need adjustment.
> >  
> > -Ewen
> > > Henry Saputra (mailto:henry.saputra@gmail.com)  
> > > January 19, 2014 10:07 PM
> > >  
> > >  
> > > Ah yes, makes sense, thanks!
> > >  
> > > - Henry
> > > Patrick Wendell (mailto:pwendell@gmail.com)  
> > > January 19, 2014 10:01 PM
> > >  
> > >  
> > > Eventually the notes get posted on the apache website. I attached them
> > > to this e-mail so that people can get a sense of what is in the
> > > release before they vote on it.
> > > Henry Saputra (mailto:henry.saputra@gmail.com)  
> > > January 19, 2014 9:57 PM
> > >  
> > >  
> > > Hi Patrick, quick question, where are you planning to add the release notes?
> > > I dont think it is part of the source, is it?
> > >  
> > > - Henry
> > > Patrick Wendell (mailto:pwendell@gmail.com)  
> > > January 19, 2014 8:41 PM
> > >  
> > >  
> > > Attempting to attach the release notes again (I think it may have been
> > > blocked previously due to not having an extension).
> > > Patrick Wendell (mailto:pwendell@gmail.com)  
> > > January 19, 2014 8:05 PM
> > >  
> > >  
> > > I'll add my +1 as well
>  


Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Reynold Xin <rx...@databricks.com>.
That's a perm gen issue - you need to adjust the perm gem size. In sbt it
should've been set automatically, but I think for Maven, you need to set
the maven opts, which is documented in the build instructions.


On Sun, Jan 19, 2014 at 11:35 PM, Ewen Cheslack-Postava
<ew...@fastmail.fm>wrote:

> I can't get the tests to run on a Mac, 10.7.5, java -version output:
>
> java version "1.6.0_65"
> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>
> For reference, Spark 0.8.* build and test find on the same configuration.
> 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not sure what it's
> running at that time since all the tests in PrimitiveVectorSuite seem to
> have finished:
>
> [info] PrimitiveVectorSuite:
> [info] - primitive value (4 milliseconds)
> [info] - non-primitive value (5 milliseconds)
> [info] - ideal growth (4 milliseconds)
> [info] - ideal size (5 milliseconds)
> [info] - resizing (6 milliseconds)
> [ERROR] [01/19/2014 23:16:27.508] [spark-akka.actor.default-dispatcher-4]
> [ActorSystem(spark)] exception while executing timer task
> org.apache.spark.SparkException: Error sending message to
> BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, localhost,
> 51634, 0))]
>     at
> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>     at
> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>     at org.apache.spark.storage.BlockManager.org
> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>     at
> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>     at
> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>     at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
>     at
> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
>     at
> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
>     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
>     at
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
>     at
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>     at
> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
>     at
> akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
>     at
> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>     at
> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>     at
> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>     at
> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
>     at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>     at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>     at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>     at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: akka.pattern.AskTimeoutException:
> Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had
> already been terminated.
>     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>     at
> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
>     ... 39 more
>
> [ERROR] [01/19/2014 23:20:03.131]
> [sparkWorker2-akka.actor.default-dispatcher-19] [ActorSystem(sparkWorker2)]
> Uncaught fatal error from thread
> [sparkWorker2-akka.actor.default-dispatcher-19] shutting down ActorSystem
> [sparkWorker2]
> java.lang.OutOfMemoryError: PermGen space
>
> [ERROR] [01/19/2014 23:20:05.226]
> [sparkWorker2-akka.actor.default-dispatcher-2] [ActorSystem(sparkWorker2)]
> Uncaught fatal error from thread
> [sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem
> [sparkWorker2]
> java.lang.OutOfMemoryError: PermGen space
>
> [... snip ... followed by a bunch more like these ]
>
> It looks like it's just a heap size issue. But since it's leaving a
> process running that has -Xmx3g explicitly specified on the command line
> I'm surprised it works for everyone else but consistently fails for me.
> Almost certainly not a blocker, but some memory setting in the tests might
> need adjustment.
>
> -Ewen
>
>   Henry Saputra <he...@gmail.com>
>  January 19, 2014 10:07 PM
> Ah yes, makes sense, thanks!
>
> - Henry
>   Patrick Wendell <pw...@gmail.com>
>  January 19, 2014 10:01 PM
> Eventually the notes get posted on the apache website. I attached them
> to this e-mail so that people can get a sense of what is in the
> release before they vote on it.
>   Henry Saputra <he...@gmail.com>
>  January 19, 2014 9:57 PM
> Hi Patrick, quick question, where are you planning to add the release
> notes?
> I dont think it is part of the source, is it?
>
> - Henry
>   Patrick Wendell <pw...@gmail.com>
>  January 19, 2014 8:41 PM
> Attempting to attach the release notes again (I think it may have been
> blocked previously due to not having an extension).
>   Patrick Wendell <pw...@gmail.com>
>  January 19, 2014 8:05 PM
> I'll add my +1 as well
>
>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Ewen Cheslack-Postava <ew...@fastmail.fm>.
I can't get the tests to run on a Mac, 10.7.5, java -version output:

java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)

For reference, Spark 0.8.* build and test find on the same 
configuration. 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not 
sure what it's running at that time since all the tests in 
PrimitiveVectorSuite seem to have finished:

[info] PrimitiveVectorSuite:
[info] - primitive value (4 milliseconds)
[info] - non-primitive value (5 milliseconds)
[info] - ideal growth (4 milliseconds)
[info] - ideal size (5 milliseconds)
[info] - resizing (6 milliseconds)
[ERROR] [01/19/2014 23:16:27.508] 
[spark-akka.actor.default-dispatcher-4] [ActorSystem(spark)] exception 
while executing timer task
org.apache.spark.SparkException: Error sending message to 
BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, 
localhost, 51634, 0))]
     at 
org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
     at 
org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
     at 
org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
     at 
org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
     at 
akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
     at 
akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
     at 
akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
     at 
akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
     at 
akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
     at 
akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
     at 
akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
     at 
akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
     at 
akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
     at 
akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
     at 
akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
     at 
akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
     at 
akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
     at 
akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
     at 
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
     at 
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
     at 
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
     at 
akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
     at 
scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
     at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
     at 
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
     at 
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
     at 
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: akka.pattern.AskTimeoutException: 
Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had 
already been terminated.
     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
     at 
org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
     ... 39 more

[ERROR] [01/19/2014 23:20:03.131] 
[sparkWorker2-akka.actor.default-dispatcher-19] 
[ActorSystem(sparkWorker2)] Uncaught fatal error from thread 
[sparkWorker2-akka.actor.default-dispatcher-19] shutting down 
ActorSystem [sparkWorker2]
java.lang.OutOfMemoryError: PermGen space

[ERROR] [01/19/2014 23:20:05.226] 
[sparkWorker2-akka.actor.default-dispatcher-2] 
[ActorSystem(sparkWorker2)] Uncaught fatal error from thread 
[sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem 
[sparkWorker2]
java.lang.OutOfMemoryError: PermGen space

[... snip ... followed by a bunch more like these ]

It looks like it's just a heap size issue. But since it's leaving a 
process running that has -Xmx3g explicitly specified on the command line 
I'm surprised it works for everyone else but consistently fails for me. 
Almost certainly not a blocker, but some memory setting in the tests 
might need adjustment.

-Ewen
> Henry Saputra <ma...@gmail.com>
> January 19, 2014 10:07 PM
> Ah yes, makes sense, thanks!
>
> - Henry
> Patrick Wendell <ma...@gmail.com>
> January 19, 2014 10:01 PM
> Eventually the notes get posted on the apache website. I attached them
> to this e-mail so that people can get a sense of what is in the
> release before they vote on it.
> Henry Saputra <ma...@gmail.com>
> January 19, 2014 9:57 PM
> Hi Patrick, quick question, where are you planning to add the release 
> notes?
> I dont think it is part of the source, is it?
>
> - Henry
> Patrick Wendell <ma...@gmail.com>
> January 19, 2014 8:41 PM
> Attempting to attach the release notes again (I think it may have been
> blocked previously due to not having an extension).
> Patrick Wendell <ma...@gmail.com>
> January 19, 2014 8:05 PM
> I'll add my +1 as well

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Henry Saputra <he...@gmail.com>.
Ah yes, makes sense, thanks!

- Henry

On Sun, Jan 19, 2014 at 10:01 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Eventually the notes get posted on the apache website. I attached them
> to this e-mail so that people can get a sense of what is in the
> release before they vote on it.
>
> On Sun, Jan 19, 2014 at 9:57 PM, Henry Saputra <he...@gmail.com> wrote:
>> Hi Patrick, quick question, where are you planning to add the release notes?
>> I dont think it is part of the source, is it?
>>
>> - Henry
>>
>> On Sun, Jan 19, 2014 at 8:41 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>> Attempting to attach the release notes again (I think it may have been
>>> blocked previously due to not having an extension).
>>>
>>> On Sun, Jan 19, 2014 at 8:05 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>>> I'll add my +1 as well
>>>>
>>>> On Sun, Jan 19, 2014 at 7:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
>>>>> +1
>>>>>
>>>>> Re-tested on Mac.
>>>>>
>>>>> Matei
>>>>>
>>>>> On Jan 19, 2014, at 7:09 PM, Tathagata Das <ta...@gmail.com> wrote:
>>>>>
>>>>>> Starting off.
>>>>>> +1
>>>>>>
>>>>>>
>>>>>> On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>>>>>
>>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>>> (incubating) version 0.9.0.
>>>>>>>
>>>>>>> A draft of the release notes along with the changes file is attached
>>>>>>> to this e-mail.
>>>>>>>
>>>>>>> The tag to be voted on is v0.9.0-incubating (commit a7760eff):
>>>>>>>
>>>>>>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d
>>>>>>>
>>>>>>> The release files, including signatures, digests, etc can be found at:
>>>>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/
>>>>>>>
>>>>>>> Release artifacts are signed with the following key:
>>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>>
>>>>>>> The staging repository for this release can be found at:
>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1004/
>>>>>>>
>>>>>>> The documentation corresponding to this release can be found at:
>>>>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/
>>>>>>>
>>>>>>> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>>>>>>>
>>>>>>> The vote is open until Wednesday, January 22, at 22:15 UTC and passes
>>>>>>> if a majority of at least 3 +1 PPMC votes are cast.
>>>>>>>
>>>>>>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>>>>>>> [ ] -1 Do not release this package because ...
>>>>>>>
>>>>>>> To learn more about Apache Spark, please see
>>>>>>> http://spark.incubator.apache.org/
>>>>>>>
>>>>>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Patrick Wendell <pw...@gmail.com>.
Eventually the notes get posted on the apache website. I attached them
to this e-mail so that people can get a sense of what is in the
release before they vote on it.

On Sun, Jan 19, 2014 at 9:57 PM, Henry Saputra <he...@gmail.com> wrote:
> Hi Patrick, quick question, where are you planning to add the release notes?
> I dont think it is part of the source, is it?
>
> - Henry
>
> On Sun, Jan 19, 2014 at 8:41 PM, Patrick Wendell <pw...@gmail.com> wrote:
>> Attempting to attach the release notes again (I think it may have been
>> blocked previously due to not having an extension).
>>
>> On Sun, Jan 19, 2014 at 8:05 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>> I'll add my +1 as well
>>>
>>> On Sun, Jan 19, 2014 at 7:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
>>>> +1
>>>>
>>>> Re-tested on Mac.
>>>>
>>>> Matei
>>>>
>>>> On Jan 19, 2014, at 7:09 PM, Tathagata Das <ta...@gmail.com> wrote:
>>>>
>>>>> Starting off.
>>>>> +1
>>>>>
>>>>>
>>>>> On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>>>>
>>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>>> (incubating) version 0.9.0.
>>>>>>
>>>>>> A draft of the release notes along with the changes file is attached
>>>>>> to this e-mail.
>>>>>>
>>>>>> The tag to be voted on is v0.9.0-incubating (commit a7760eff):
>>>>>>
>>>>>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d
>>>>>>
>>>>>> The release files, including signatures, digests, etc can be found at:
>>>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/
>>>>>>
>>>>>> Release artifacts are signed with the following key:
>>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>>
>>>>>> The staging repository for this release can be found at:
>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1004/
>>>>>>
>>>>>> The documentation corresponding to this release can be found at:
>>>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/
>>>>>>
>>>>>> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>>>>>>
>>>>>> The vote is open until Wednesday, January 22, at 22:15 UTC and passes
>>>>>> if a majority of at least 3 +1 PPMC votes are cast.
>>>>>>
>>>>>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>>>>>> [ ] -1 Do not release this package because ...
>>>>>>
>>>>>> To learn more about Apache Spark, please see
>>>>>> http://spark.incubator.apache.org/
>>>>>>
>>>>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Henry Saputra <he...@gmail.com>.
Hi Patrick, quick question, where are you planning to add the release notes?
I dont think it is part of the source, is it?

- Henry

On Sun, Jan 19, 2014 at 8:41 PM, Patrick Wendell <pw...@gmail.com> wrote:
> Attempting to attach the release notes again (I think it may have been
> blocked previously due to not having an extension).
>
> On Sun, Jan 19, 2014 at 8:05 PM, Patrick Wendell <pw...@gmail.com> wrote:
>> I'll add my +1 as well
>>
>> On Sun, Jan 19, 2014 at 7:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
>>> +1
>>>
>>> Re-tested on Mac.
>>>
>>> Matei
>>>
>>> On Jan 19, 2014, at 7:09 PM, Tathagata Das <ta...@gmail.com> wrote:
>>>
>>>> Starting off.
>>>> +1
>>>>
>>>>
>>>> On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>>>
>>>>> Please vote on releasing the following candidate as Apache Spark
>>>>> (incubating) version 0.9.0.
>>>>>
>>>>> A draft of the release notes along with the changes file is attached
>>>>> to this e-mail.
>>>>>
>>>>> The tag to be voted on is v0.9.0-incubating (commit a7760eff):
>>>>>
>>>>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d
>>>>>
>>>>> The release files, including signatures, digests, etc can be found at:
>>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/
>>>>>
>>>>> Release artifacts are signed with the following key:
>>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>>
>>>>> The staging repository for this release can be found at:
>>>>> https://repository.apache.org/content/repositories/orgapachespark-1004/
>>>>>
>>>>> The documentation corresponding to this release can be found at:
>>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/
>>>>>
>>>>> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>>>>>
>>>>> The vote is open until Wednesday, January 22, at 22:15 UTC and passes
>>>>> if a majority of at least 3 +1 PPMC votes are cast.
>>>>>
>>>>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>>>>> [ ] -1 Do not release this package because ...
>>>>>
>>>>> To learn more about Apache Spark, please see
>>>>> http://spark.incubator.apache.org/
>>>>>
>>>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Patrick Wendell <pw...@gmail.com>.
Attempting to attach the release notes again (I think it may have been
blocked previously due to not having an extension).

On Sun, Jan 19, 2014 at 8:05 PM, Patrick Wendell <pw...@gmail.com> wrote:
> I'll add my +1 as well
>
> On Sun, Jan 19, 2014 at 7:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
>> +1
>>
>> Re-tested on Mac.
>>
>> Matei
>>
>> On Jan 19, 2014, at 7:09 PM, Tathagata Das <ta...@gmail.com> wrote:
>>
>>> Starting off.
>>> +1
>>>
>>>
>>> On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>>
>>>> Please vote on releasing the following candidate as Apache Spark
>>>> (incubating) version 0.9.0.
>>>>
>>>> A draft of the release notes along with the changes file is attached
>>>> to this e-mail.
>>>>
>>>> The tag to be voted on is v0.9.0-incubating (commit a7760eff):
>>>>
>>>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d
>>>>
>>>> The release files, including signatures, digests, etc can be found at:
>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/
>>>>
>>>> Release artifacts are signed with the following key:
>>>> https://people.apache.org/keys/committer/pwendell.asc
>>>>
>>>> The staging repository for this release can be found at:
>>>> https://repository.apache.org/content/repositories/orgapachespark-1004/
>>>>
>>>> The documentation corresponding to this release can be found at:
>>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/
>>>>
>>>> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>>>>
>>>> The vote is open until Wednesday, January 22, at 22:15 UTC and passes
>>>> if a majority of at least 3 +1 PPMC votes are cast.
>>>>
>>>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>>>> [ ] -1 Do not release this package because ...
>>>>
>>>> To learn more about Apache Spark, please see
>>>> http://spark.incubator.apache.org/
>>>>
>>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Patrick Wendell <pw...@gmail.com>.
I'll add my +1 as well

On Sun, Jan 19, 2014 at 7:33 PM, Matei Zaharia <ma...@gmail.com> wrote:
> +1
>
> Re-tested on Mac.
>
> Matei
>
> On Jan 19, 2014, at 7:09 PM, Tathagata Das <ta...@gmail.com> wrote:
>
>> Starting off.
>> +1
>>
>>
>> On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <pw...@gmail.com> wrote:
>>
>>> Please vote on releasing the following candidate as Apache Spark
>>> (incubating) version 0.9.0.
>>>
>>> A draft of the release notes along with the changes file is attached
>>> to this e-mail.
>>>
>>> The tag to be voted on is v0.9.0-incubating (commit a7760eff):
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d
>>>
>>> The release files, including signatures, digests, etc can be found at:
>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/
>>>
>>> Release artifacts are signed with the following key:
>>> https://people.apache.org/keys/committer/pwendell.asc
>>>
>>> The staging repository for this release can be found at:
>>> https://repository.apache.org/content/repositories/orgapachespark-1004/
>>>
>>> The documentation corresponding to this release can be found at:
>>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/
>>>
>>> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>>>
>>> The vote is open until Wednesday, January 22, at 22:15 UTC and passes
>>> if a majority of at least 3 +1 PPMC votes are cast.
>>>
>>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>>> [ ] -1 Do not release this package because ...
>>>
>>> To learn more about Apache Spark, please see
>>> http://spark.incubator.apache.org/
>>>
>

Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Matei Zaharia <ma...@gmail.com>.
+1

Re-tested on Mac.

Matei

On Jan 19, 2014, at 7:09 PM, Tathagata Das <ta...@gmail.com> wrote:

> Starting off.
> +1
> 
> 
> On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <pw...@gmail.com> wrote:
> 
>> Please vote on releasing the following candidate as Apache Spark
>> (incubating) version 0.9.0.
>> 
>> A draft of the release notes along with the changes file is attached
>> to this e-mail.
>> 
>> The tag to be voted on is v0.9.0-incubating (commit a7760eff):
>> 
>> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d
>> 
>> The release files, including signatures, digests, etc can be found at:
>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/
>> 
>> Release artifacts are signed with the following key:
>> https://people.apache.org/keys/committer/pwendell.asc
>> 
>> The staging repository for this release can be found at:
>> https://repository.apache.org/content/repositories/orgapachespark-1004/
>> 
>> The documentation corresponding to this release can be found at:
>> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/
>> 
>> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>> 
>> The vote is open until Wednesday, January 22, at 22:15 UTC and passes
>> if a majority of at least 3 +1 PPMC votes are cast.
>> 
>> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
>> [ ] -1 Do not release this package because ...
>> 
>> To learn more about Apache Spark, please see
>> http://spark.incubator.apache.org/
>> 


Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)

Posted by Tathagata Das <ta...@gmail.com>.
Starting off.
 +1


On Sun, Jan 19, 2014 at 2:15 PM, Patrick Wendell <pw...@gmail.com> wrote:

> Please vote on releasing the following candidate as Apache Spark
> (incubating) version 0.9.0.
>
> A draft of the release notes along with the changes file is attached
> to this e-mail.
>
> The tag to be voted on is v0.9.0-incubating (commit a7760eff):
>
> https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=a7760eff4ea6a474cab68896a88550f63bae8b0d
>
> The release files, including signatures, digests, etc can be found at:
> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/pwendell.asc
>
> The staging repository for this release can be found at:
> https://repository.apache.org/content/repositories/orgapachespark-1004/
>
> The documentation corresponding to this release can be found at:
> http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc3-docs/
>
> Please vote on releasing this package as Apache Spark 0.9.0-incubating!
>
> The vote is open until Wednesday, January 22, at 22:15 UTC and passes
> if a majority of at least 3 +1 PPMC votes are cast.
>
> [ ] +1 Release this package as Apache Spark 0.9.0-incubating
> [ ] -1 Do not release this package because ...
>
> To learn more about Apache Spark, please see
> http://spark.incubator.apache.org/
>