You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Guillermo Ortiz <ko...@gmail.com> on 2015/07/30 13:04:18 UTC

Error SparkStreaming after a while executing.

I'm executing a job with Spark Streaming and got this error all times when
the job has been executing for a while (usually hours of days).

I have no idea why it's happening.

15/07/30 13:02:14 ERROR LiveListenerBus: Listener EventLoggingListener
threw an exception
java.lang.reflect.InvocationTargetException
        at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
        at
org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
        at scala.Option.foreach(Option.scala:236)
        at
org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
        at
org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:169)
        at
org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:36)
        at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
        at
org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
        at
org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
        at
org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
        at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
        at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
        at
org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
        at
org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
        at
org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
Caused by: java.io.IOException: Lease timeout of 0 seconds expired.
        at
org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2192)
        at
org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:935)
        at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:889)
        at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
        at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
        at
org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
        at java.lang.Thread.run(Thread.java:745)
15/07/30 13:02:14 INFO SparkContext: Starting job: foreachRDD at
MetricsSpark.scala:67
15/07/30 13:02:14 INFO DAGScheduler: Got job 5050 (foreachRDD at
MetricsSpark.scala:67) with 3 output partitions (allowLocal=false)
15/07/30 13:02:14 INFO DAGScheduler: Final stage: Stage 5050(foreachRDD at
MetricsSpark.scala:67)

Sometimes this error happens, but it doesn't mean that Spark stops working
forever. Because it looks like after this error Spark works correctly some
iterations, but most of time just fails after this error producing this
error all the time.

The code just filters some records and index the record in
ElasticSearch.after adding some new fields.

Re: Error SparkStreaming after a while executing.

Posted by Guillermo Ortiz <ko...@gmail.com>.

Yes, I use YARN.
I could run the job an delete the Kerberos credentials and check if it's
because there aren't credetianls. I don't know if it's the same destroy the
credential that the credentials expire. But I can try that.

2015-07-31 7:09 GMT+02:00 Tathagata Das <td...@databricks.com>:

> Are you running it on YARN? Might be a known YARN credential expiring
> issue.
>
> Hari, any insights?
>
> On Thu, Jul 30, 2015 at 4:04 AM, Guillermo Ortiz <ko...@gmail.com>
> wrote:
>
>> I'm executing a job with Spark Streaming and got this error all times
>> when the job has been executing for a while (usually hours of days).
>>
>> I have no idea why it's happening.
>>
>> 15/07/30 13:02:14 ERROR LiveListenerBus: Listener EventLoggingListener
>> threw an exception
>> java.lang.reflect.InvocationTargetException
>>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at
>> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
>>         at
>> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
>>         at scala.Option.foreach(Option.scala:236)
>>         at
>> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
>>         at
>> org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:169)
>>         at
>> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:36)
>>         at
>> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>>         at
>> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>>         at
>> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
>>         at
>> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
>>         at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
>>         at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>         at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>>         at
>> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>>         at
>> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
>> Caused by: java.io.IOException: Lease timeout of 0 seconds expired.
>>         at
>> org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2192)
>>         at
>> org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:935)
>>         at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:889)
>>         at
>> org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
>>         at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
>>         at
>> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>>         at
>> org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>>         at java.lang.Thread.run(Thread.java:745)
>> 15/07/30 13:02:14 INFO SparkContext: Starting job: foreachRDD at
>> MetricsSpark.scala:67
>> 15/07/30 13:02:14 INFO DAGScheduler: Got job 5050 (foreachRDD at
>> MetricsSpark.scala:67) with 3 output partitions (allowLocal=false)
>> 15/07/30 13:02:14 INFO DAGScheduler: Final stage: Stage 5050(foreachRDD
>> at MetricsSpark.scala:67)
>>
>> Sometimes this error happens, but it doesn't mean that Spark stops
>> working forever. Because it looks like after this error Spark works
>> correctly some iterations, but most of time just fails after this error
>> producing this error all the time.
>>
>> The code just filters some records and index the record in
>> ElasticSearch.after adding some new fields.
>>
>
>

Re: Error SparkStreaming after a while executing.

Posted by Tathagata Das <td...@databricks.com>.

Are you running it on YARN? Might be a known YARN credential expiring
issue.

Hari, any insights?

On Thu, Jul 30, 2015 at 4:04 AM, Guillermo Ortiz <ko...@gmail.com>
wrote:

> I'm executing a job with Spark Streaming and got this error all times when
> the job has been executing for a while (usually hours of days).
>
> I have no idea why it's happening.
>
> 15/07/30 13:02:14 ERROR LiveListenerBus: Listener EventLoggingListener
> threw an exception
> java.lang.reflect.InvocationTargetException
>         at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
>         at
> org.apache.spark.scheduler.EventLoggingListener$$anonfun$logEvent$3.apply(EventLoggingListener.scala:144)
>         at scala.Option.foreach(Option.scala:236)
>         at
> org.apache.spark.scheduler.EventLoggingListener.logEvent(EventLoggingListener.scala:144)
>         at
> org.apache.spark.scheduler.EventLoggingListener.onJobEnd(EventLoggingListener.scala:169)
>         at
> org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:36)
>         at
> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>         at
> org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31)
>         at
> org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:53)
>         at
> org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:36)
>         at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:76)
>         at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>         at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply(AsynchronousListenerBus.scala:61)
>         at
> org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1617)
>         at
> org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:60)
> Caused by: java.io.IOException: Lease timeout of 0 seconds expired.
>         at
> org.apache.hadoop.hdfs.DFSOutputStream.abort(DFSOutputStream.java:2192)
>         at
> org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:935)
>         at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:889)
>         at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
>         at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
>         at
> org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
>         at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
>         at java.lang.Thread.run(Thread.java:745)
> 15/07/30 13:02:14 INFO SparkContext: Starting job: foreachRDD at
> MetricsSpark.scala:67
> 15/07/30 13:02:14 INFO DAGScheduler: Got job 5050 (foreachRDD at
> MetricsSpark.scala:67) with 3 output partitions (allowLocal=false)
> 15/07/30 13:02:14 INFO DAGScheduler: Final stage: Stage 5050(foreachRDD at
> MetricsSpark.scala:67)
>
> Sometimes this error happens, but it doesn't mean that Spark stops working
> forever. Because it looks like after this error Spark works correctly some
> iterations, but most of time just fails after this error producing this
> error all the time.
>
> The code just filters some records and index the record in
> ElasticSearch.after adding some new fields.
>