You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Michael Chang <mi...@tellapart.com> on 2014/06/02 18:42:54 UTC
Re: Failed to remove RDD error
Hey Mayur,
Thanks for the suggestion, I didn't realize that was configurable. I don't
think I'm running out of memory, though it does seem like these errors go
away when i turn off the spark.streaming.unpersist configuration and use
spark.cleaner.ttl instead. Do you know if there are known issues with the
unpersist option?
On Sat, May 31, 2014 at 12:17 AM, Mayur Rustagi <ma...@gmail.com>
wrote:
> You can increase your akka timeout, should give you some more life.. are
> you running out of memory by any chance?
>
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Sat, May 31, 2014 at 6:52 AM, Michael Chang <mi...@tellapart.com> wrote:
>
>> I'm running a some kafka streaming spark contexts (on 0.9.1), and they
>> seem to be dying after 10 or so minutes with a lot of these errors. I
>> can't really tell what's going on here, except that maybe the driver is
>> unresponsive somehow? Has anyone seen this before?
>>
>> 14/05/31 01:13:30 ERROR BlockManagerMaster: Failed to remove RDD 12635
>>
>> akka.pattern.AskTimeoutException: Timed out
>>
>> at
>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>>
>> at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118)
>>
>> at
>> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:691)
>>
>> at
>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:688)
>>
>> at
>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455)
>>
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407)
>>
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411)
>>
>> at
>> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>>
>> at java.lang.Thread.run(Thread.java:744)
>>
>> Thanks,
>>
>> Mike
>>
>>
>>
>
Re: Failed to remove RDD error
Posted by Michael Chang <mi...@tellapart.com>.
Unfortunately don't have any logs at the moment. I will post them here if
they occur again!
On Tue, Jun 3, 2014 at 11:04 AM, Tathagata Das <ta...@gmail.com>
wrote:
> It was not intended to be experimental as this improves general
> performance. We tested the feature since 0.9, and didnt see any problems.
> We need to investigate the cause of this. Can you give us the logs showing
> this error so that we can analyze this.
>
> TD
>
>
> On Tue, Jun 3, 2014 at 10:08 AM, Michael Chang <mi...@tellapart.com> wrote:
>
>> Thanks Tathagata,
>>
>> Thanks for all your hard work! In the future, is it possible to mark
>> "experimental" features as such on the online documentation?
>>
>> Thanks,
>> Michael
>>
>>
>> On Mon, Jun 2, 2014 at 6:12 PM, Tathagata Das <
>> tathagata.das1565@gmail.com> wrote:
>>
>>> Spark.streaming.unpersist was an experimental feature introduced with
>>> Spark 0.9 (but kept disabled), which actively clears off RDDs that are not
>>> useful any more. in Spark 1.0 that has been enabled by default. It is
>>> possible that this is an unintended side-effect of that. If
>>> spark.cleaner.ttl works then that should be used.
>>>
>>> TD
>>>
>>>
>>> On Mon, Jun 2, 2014 at 9:42 AM, Michael Chang <mi...@tellapart.com>
>>> wrote:
>>>
>>>> Hey Mayur,
>>>>
>>>> Thanks for the suggestion, I didn't realize that was configurable. I
>>>> don't think I'm running out of memory, though it does seem like these
>>>> errors go away when i turn off the spark.streaming.unpersist configuration
>>>> and use spark.cleaner.ttl instead. Do you know if there are known issues
>>>> with the unpersist option?
>>>>
>>>>
>>>> On Sat, May 31, 2014 at 12:17 AM, Mayur Rustagi <
>>>> mayur.rustagi@gmail.com> wrote:
>>>>
>>>>> You can increase your akka timeout, should give you some more life..
>>>>> are you running out of memory by any chance?
>>>>>
>>>>>
>>>>> Mayur Rustagi
>>>>> Ph: +1 (760) 203 3257
>>>>> http://www.sigmoidanalytics.com
>>>>> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>>>>>
>>>>>
>>>>>
>>>>> On Sat, May 31, 2014 at 6:52 AM, Michael Chang <mi...@tellapart.com>
>>>>> wrote:
>>>>>
>>>>>> I'm running a some kafka streaming spark contexts (on 0.9.1), and
>>>>>> they seem to be dying after 10 or so minutes with a lot of these errors. I
>>>>>> can't really tell what's going on here, except that maybe the driver is
>>>>>> unresponsive somehow? Has anyone seen this before?
>>>>>>
>>>>>> 14/05/31 01:13:30 ERROR BlockManagerMaster: Failed to remove RDD 12635
>>>>>>
>>>>>> akka.pattern.AskTimeoutException: Timed out
>>>>>>
>>>>>> at
>>>>>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>>>>>>
>>>>>> at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118)
>>>>>>
>>>>>> at
>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:691)
>>>>>>
>>>>>> at
>>>>>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:688)
>>>>>>
>>>>>> at
>>>>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455)
>>>>>>
>>>>>> at
>>>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407)
>>>>>>
>>>>>> at
>>>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411)
>>>>>>
>>>>>> at
>>>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Failed to remove RDD error
Posted by Tathagata Das <ta...@gmail.com>.
It was not intended to be experimental as this improves general
performance. We tested the feature since 0.9, and didnt see any problems.
We need to investigate the cause of this. Can you give us the logs showing
this error so that we can analyze this.
TD
On Tue, Jun 3, 2014 at 10:08 AM, Michael Chang <mi...@tellapart.com> wrote:
> Thanks Tathagata,
>
> Thanks for all your hard work! In the future, is it possible to mark
> "experimental" features as such on the online documentation?
>
> Thanks,
> Michael
>
>
> On Mon, Jun 2, 2014 at 6:12 PM, Tathagata Das <tathagata.das1565@gmail.com
> > wrote:
>
>> Spark.streaming.unpersist was an experimental feature introduced with
>> Spark 0.9 (but kept disabled), which actively clears off RDDs that are not
>> useful any more. in Spark 1.0 that has been enabled by default. It is
>> possible that this is an unintended side-effect of that. If
>> spark.cleaner.ttl works then that should be used.
>>
>> TD
>>
>>
>> On Mon, Jun 2, 2014 at 9:42 AM, Michael Chang <mi...@tellapart.com> wrote:
>>
>>> Hey Mayur,
>>>
>>> Thanks for the suggestion, I didn't realize that was configurable. I
>>> don't think I'm running out of memory, though it does seem like these
>>> errors go away when i turn off the spark.streaming.unpersist configuration
>>> and use spark.cleaner.ttl instead. Do you know if there are known issues
>>> with the unpersist option?
>>>
>>>
>>> On Sat, May 31, 2014 at 12:17 AM, Mayur Rustagi <mayur.rustagi@gmail.com
>>> > wrote:
>>>
>>>> You can increase your akka timeout, should give you some more life..
>>>> are you running out of memory by any chance?
>>>>
>>>>
>>>> Mayur Rustagi
>>>> Ph: +1 (760) 203 3257
>>>> http://www.sigmoidanalytics.com
>>>> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>>>>
>>>>
>>>>
>>>> On Sat, May 31, 2014 at 6:52 AM, Michael Chang <mi...@tellapart.com>
>>>> wrote:
>>>>
>>>>> I'm running a some kafka streaming spark contexts (on 0.9.1), and they
>>>>> seem to be dying after 10 or so minutes with a lot of these errors. I
>>>>> can't really tell what's going on here, except that maybe the driver is
>>>>> unresponsive somehow? Has anyone seen this before?
>>>>>
>>>>> 14/05/31 01:13:30 ERROR BlockManagerMaster: Failed to remove RDD 12635
>>>>>
>>>>> akka.pattern.AskTimeoutException: Timed out
>>>>>
>>>>> at
>>>>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>>>>>
>>>>> at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118)
>>>>>
>>>>> at
>>>>> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:691)
>>>>>
>>>>> at
>>>>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:688)
>>>>>
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455)
>>>>>
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407)
>>>>>
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411)
>>>>>
>>>>> at
>>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>>>>>
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Failed to remove RDD error
Posted by Michael Chang <mi...@tellapart.com>.
Thanks Tathagata,
Thanks for all your hard work! In the future, is it possible to mark
"experimental" features as such on the online documentation?
Thanks,
Michael
On Mon, Jun 2, 2014 at 6:12 PM, Tathagata Das <ta...@gmail.com>
wrote:
> Spark.streaming.unpersist was an experimental feature introduced with
> Spark 0.9 (but kept disabled), which actively clears off RDDs that are not
> useful any more. in Spark 1.0 that has been enabled by default. It is
> possible that this is an unintended side-effect of that. If
> spark.cleaner.ttl works then that should be used.
>
> TD
>
>
> On Mon, Jun 2, 2014 at 9:42 AM, Michael Chang <mi...@tellapart.com> wrote:
>
>> Hey Mayur,
>>
>> Thanks for the suggestion, I didn't realize that was configurable. I
>> don't think I'm running out of memory, though it does seem like these
>> errors go away when i turn off the spark.streaming.unpersist configuration
>> and use spark.cleaner.ttl instead. Do you know if there are known issues
>> with the unpersist option?
>>
>>
>> On Sat, May 31, 2014 at 12:17 AM, Mayur Rustagi <ma...@gmail.com>
>> wrote:
>>
>>> You can increase your akka timeout, should give you some more life.. are
>>> you running out of memory by any chance?
>>>
>>>
>>> Mayur Rustagi
>>> Ph: +1 (760) 203 3257
>>> http://www.sigmoidanalytics.com
>>> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>>>
>>>
>>>
>>> On Sat, May 31, 2014 at 6:52 AM, Michael Chang <mi...@tellapart.com>
>>> wrote:
>>>
>>>> I'm running a some kafka streaming spark contexts (on 0.9.1), and they
>>>> seem to be dying after 10 or so minutes with a lot of these errors. I
>>>> can't really tell what's going on here, except that maybe the driver is
>>>> unresponsive somehow? Has anyone seen this before?
>>>>
>>>> 14/05/31 01:13:30 ERROR BlockManagerMaster: Failed to remove RDD 12635
>>>>
>>>> akka.pattern.AskTimeoutException: Timed out
>>>>
>>>> at
>>>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>>>>
>>>> at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118)
>>>>
>>>> at
>>>> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:691)
>>>>
>>>> at
>>>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:688)
>>>>
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455)
>>>>
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407)
>>>>
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411)
>>>>
>>>> at
>>>> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>>>>
>>>> at java.lang.Thread.run(Thread.java:744)
>>>>
>>>> Thanks,
>>>>
>>>> Mike
>>>>
>>>>
>>>>
>>>
>>
>
Re: Failed to remove RDD error
Posted by Tathagata Das <ta...@gmail.com>.
Spark.streaming.unpersist was an experimental feature introduced with Spark
0.9 (but kept disabled), which actively clears off RDDs that are not useful
any more. in Spark 1.0 that has been enabled by default. It is possible
that this is an unintended side-effect of that. If spark.cleaner.ttl works
then that should be used.
TD
On Mon, Jun 2, 2014 at 9:42 AM, Michael Chang <mi...@tellapart.com> wrote:
> Hey Mayur,
>
> Thanks for the suggestion, I didn't realize that was configurable. I
> don't think I'm running out of memory, though it does seem like these
> errors go away when i turn off the spark.streaming.unpersist configuration
> and use spark.cleaner.ttl instead. Do you know if there are known issues
> with the unpersist option?
>
>
> On Sat, May 31, 2014 at 12:17 AM, Mayur Rustagi <ma...@gmail.com>
> wrote:
>
>> You can increase your akka timeout, should give you some more life.. are
>> you running out of memory by any chance?
>>
>>
>> Mayur Rustagi
>> Ph: +1 (760) 203 3257
>> http://www.sigmoidanalytics.com
>> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>>
>>
>>
>> On Sat, May 31, 2014 at 6:52 AM, Michael Chang <mi...@tellapart.com>
>> wrote:
>>
>>> I'm running a some kafka streaming spark contexts (on 0.9.1), and they
>>> seem to be dying after 10 or so minutes with a lot of these errors. I
>>> can't really tell what's going on here, except that maybe the driver is
>>> unresponsive somehow? Has anyone seen this before?
>>>
>>> 14/05/31 01:13:30 ERROR BlockManagerMaster: Failed to remove RDD 12635
>>>
>>> akka.pattern.AskTimeoutException: Timed out
>>>
>>> at
>>> akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
>>>
>>> at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118)
>>>
>>> at
>>> scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:691)
>>>
>>> at
>>> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:688)
>>>
>>> at
>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455)
>>>
>>> at
>>> akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407)
>>>
>>> at
>>> akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411)
>>>
>>> at
>>> akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
>>>
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>>
>>>
>>
>