You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by nizang <ni...@windward.eu> on 2015/06/14 08:39:06 UTC
Job marked as killed in spark 1.4
hi,
I have a running and working cluster with spark 1.3.1, and I tried to
install a new cluster that is working with spark 1.4.0
I ran a job on the new 1.4.0 cluster, and the same job on the old 1.3.1
cluster
After the job finished (in both clusters), I entered the job in the UI, and
in the new 1.4.0 cluster, the workers are marked as KILLED (I didn't killed
them, and every place I checked, the logs and output seems fine):
2 worker-20150613111158-172.31.0.104-37240 4 10240 KILLED stdout stderr
1 worker-20150613111158-172.31.15.149-58710 4 10240 KILLED stdout stderr
3 worker-20150613111158-172.31.0.196-52939 4 10240 KILLED stdout stderr
0 worker-20150613111158-172.31.1.233-53467 4 10240 KILLED stdout stderr
In the old 1.3.1 cluster, the workers are marked as EXITED:
1 worker-20150608115639-ip-172-31-6-134.us-west-2.compute.internal-47572 2
10240 EXITED stdout stderr
0 worker-20150608115639-ip-172-31-4-169.us-west-2.compute.internal-41828 2
10240 EXITED stdout stderr
2 worker-20150608115640-ip-172-31-0-37.us-west-2.compute.internal-32847 1
10240 EXITED stdout stderr
Another thing (which I think is related) is that the history server is not
working (even though I can see the logs on s3)
I didn't killed the jobs on the 1.4.0 cluster. The output seems ok, the logs
on s3 seems fine
does anybody have any idea what is wrong here? with the jobs marked as
KILLED and with the history server
thanks, nizan
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Job-marked-as-killed-in-spark-1-4-tp23305.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Job marked as killed in spark 1.4
Posted by Nizan Grauer <ni...@windward.eu>.
and last update for that -
The job itself seems to be working and generates output on s3, it just
reports itself as KILLED, and history server can't find the logs
On Sun, Jun 14, 2015 at 3:55 PM, Nizan Grauer <ni...@windward.eu> wrote:
> hi
>
> update regarding that, hope it will get me some answers...
>
> When I enter one the workers log (for of its task), I can see the
> following exception:
>
> Exception in thread "main" akka.actor.ActorNotFound: Actor not found for: ActorSelection[Anchor(akka.tcp://sparkDriver@172.31.0.186:38560/), Path(/user/CoarseGrainedScheduler)]
> at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
> at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
> at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
> at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
> at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
> at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
> at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
> at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
> at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
> at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
> at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
> at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
> at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
> at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
> at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
> at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
> at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
> at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
> at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
> at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
> at akka.actor.ActorCell.terminate(ActorCell.scala:369)
> at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
> at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
> at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
> Is it related?
>
> thanks, nizan
>
> On Sun, Jun 14, 2015 at 9:39 AM, nizang <ni...@windward.eu> wrote:
>
>> hi,
>>
>> I have a running and working cluster with spark 1.3.1, and I tried to
>> install a new cluster that is working with spark 1.4.0
>>
>> I ran a job on the new 1.4.0 cluster, and the same job on the old 1.3.1
>> cluster
>>
>> After the job finished (in both clusters), I entered the job in the UI,
>> and
>> in the new 1.4.0 cluster, the workers are marked as KILLED (I didn't
>> killed
>> them, and every place I checked, the logs and output seems fine):
>>
>> 2 worker-20150613111158-172.31.0.104-37240 4 10240
>> KILLED stdout stderr
>> 1 worker-20150613111158-172.31.15.149-58710 4 10240
>> KILLED stdout stderr
>> 3 worker-20150613111158-172.31.0.196-52939 4 10240
>> KILLED stdout stderr
>> 0 worker-20150613111158-172.31.1.233-53467 4 10240
>> KILLED stdout stderr
>>
>> In the old 1.3.1 cluster, the workers are marked as EXITED:
>>
>> 1
>> worker-20150608115639-ip-172-31-6-134.us-west-2.compute.internal-47572 2
>> 10240 EXITED stdout stderr
>> 0
>> worker-20150608115639-ip-172-31-4-169.us-west-2.compute.internal-41828 2
>> 10240 EXITED stdout stderr
>> 2
>> worker-20150608115640-ip-172-31-0-37.us-west-2.compute.internal-32847 1
>> 10240 EXITED stdout stderr
>>
>> Another thing (which I think is related) is that the history server is not
>> working (even though I can see the logs on s3)
>> I didn't killed the jobs on the 1.4.0 cluster. The output seems ok, the
>> logs
>> on s3 seems fine
>>
>> does anybody have any idea what is wrong here? with the jobs marked as
>> KILLED and with the history server
>>
>> thanks, nizan
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Job-marked-as-killed-in-spark-1-4-tp23305.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Job marked as killed in spark 1.4
Posted by Nizan Grauer <ni...@windward.eu>.
hi
update regarding that, hope it will get me some answers...
When I enter one the workers log (for of its task), I can see the following
exception:
Exception in thread "main" akka.actor.ActorNotFound: Actor not found
for: ActorSelection[Anchor(akka.tcp://sparkDriver@172.31.0.186:38560/),
Path(/user/CoarseGrainedScheduler)]
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:65)
at akka.actor.ActorSelection$$anonfun$resolveOne$1.apply(ActorSelection.scala:63)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.unbatchedExecute(Future.scala:74)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:110)
at akka.dispatch.ExecutionContexts$sameThreadExecutionContext$.execute(Future.scala:73)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:40)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:248)
at akka.pattern.PromiseActorRef.$bang(AskSupport.scala:267)
at akka.actor.EmptyLocalActorRef.specialHandle(ActorRef.scala:508)
at akka.actor.DeadLetterActorRef.specialHandle(ActorRef.scala:541)
at akka.actor.DeadLetterActorRef.$bang(ActorRef.scala:531)
at akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef.$bang(RemoteActorRefProvider.scala:87)
at akka.remote.EndpointWriter.postStop(Endpoint.scala:561)
at akka.actor.Actor$class.aroundPostStop(Actor.scala:475)
at akka.remote.EndpointActor.aroundPostStop(Endpoint.scala:415)
at akka.actor.dungeon.FaultHandling$class.akka$actor$dungeon$FaultHandling$$finishTerminate(FaultHandling.scala:210)
at akka.actor.dungeon.FaultHandling$class.terminate(FaultHandling.scala:172)
at akka.actor.ActorCell.terminate(ActorCell.scala:369)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:462)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Is it related?
thanks, nizan
On Sun, Jun 14, 2015 at 9:39 AM, nizang <ni...@windward.eu> wrote:
> hi,
>
> I have a running and working cluster with spark 1.3.1, and I tried to
> install a new cluster that is working with spark 1.4.0
>
> I ran a job on the new 1.4.0 cluster, and the same job on the old 1.3.1
> cluster
>
> After the job finished (in both clusters), I entered the job in the UI, and
> in the new 1.4.0 cluster, the workers are marked as KILLED (I didn't killed
> them, and every place I checked, the logs and output seems fine):
>
> 2 worker-20150613111158-172.31.0.104-37240 4 10240
> KILLED stdout stderr
> 1 worker-20150613111158-172.31.15.149-58710 4 10240
> KILLED stdout stderr
> 3 worker-20150613111158-172.31.0.196-52939 4 10240
> KILLED stdout stderr
> 0 worker-20150613111158-172.31.1.233-53467 4 10240
> KILLED stdout stderr
>
> In the old 1.3.1 cluster, the workers are marked as EXITED:
>
> 1
> worker-20150608115639-ip-172-31-6-134.us-west-2.compute.internal-47572 2
> 10240 EXITED stdout stderr
> 0
> worker-20150608115639-ip-172-31-4-169.us-west-2.compute.internal-41828 2
> 10240 EXITED stdout stderr
> 2
> worker-20150608115640-ip-172-31-0-37.us-west-2.compute.internal-32847 1
> 10240 EXITED stdout stderr
>
> Another thing (which I think is related) is that the history server is not
> working (even though I can see the logs on s3)
> I didn't killed the jobs on the 1.4.0 cluster. The output seems ok, the
> logs
> on s3 seems fine
>
> does anybody have any idea what is wrong here? with the jobs marked as
> KILLED and with the history server
>
> thanks, nizan
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Job-marked-as-killed-in-spark-1-4-tp23305.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Job marked as killed in spark 1.4
Posted by nizang <ni...@windward.eu>.
hi,
A simple way to recreate the problem -
I have two servers installations, one with spark 1.3.1, and one with spark
1.4.0
I ran the following on both servers:
root@ip-172-31-6-108 ~]$ spark/bin/spark-shell --total-executor-cores 1
scala> val text = sc.textFile("hdfs:///some-file.txt”);
scala> text.count()
—here I get the correct output in both servers
At this stage, by checking spark-ui, both are marked as RUNNING
Now, I exit the spark shell (using ctrl+d), and if I check the spark UI now,
the job on 1.3.1 is marked as EXITED, and the job on 1.4.0 is marked as
KILLED
Why is the job marked as killed?
This is a simple case that simulates problem on a real server I’m running
thanks, nizan
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Job-marked-as-killed-in-spark-1-4-tp23305p23311.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org