KafkaReceiver executor in spark streaming job on YARN suddenly killed by ResourceManager

[Junki Kim <wi...@gmail.com>]

Hi, guys

I tried to run job of spark streaming with kafka on YARN.
My business logic is very simple.
Just listen on kafka topic and  write dstream to hdfs on each batch
iteration.
After launching streaming job few hours, it works well. However suddenly
died by ResourceManager.
ResourceManager send SIGTERM to kafka receiver executor and other executors
also died without any log error messages.
No Outofmemory exception, no other exceptions…. just died.

Do you know about this issue??
Please help me to resolve this problem.

My environments are below described.

* Scala : 2.10
* Spark : 1.1
* YARN : 2.5.0-cdh5.2.1
* CDH : 5.2.1 version

Thanks in advance.

Have a nice day~



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/KafkaReceiver-executor-in-spark-streaming-job-on-YARN-suddenly-killed-by-ResourceManager-tp20945.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org