You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by James King <ja...@gmail.com> on 2015/04/13 10:55:15 UTC
Spark Cluster: RECEIVED SIGNAL 15: SIGTERM
Any idea what this means, many thanks
==>
logs/spark-XXXX.XXXX-org.apache.spark.deploy.worker.Worker-1-XXXX09.out.1
<==
15/04/13 07:07:22 INFO Worker: Starting Spark worker XXXX09:39910 with 4
cores, 6.6 GB RAM
15/04/13 07:07:22 INFO Worker: Running Spark version 1.3.0
15/04/13 07:07:22 INFO Worker: Spark home:
/remote/users/XXXX/work/tools/spark-1.3.0-bin-hadoop2.4
15/04/13 07:07:22 INFO Server: jetty-8.y.z-SNAPSHOT
15/04/13 07:07:22 INFO AbstractConnector: Started
SelectChannelConnector@0.0.0.0:8081
15/04/13 07:07:22 INFO Utils: Successfully started service 'WorkerUI' on
port 8081.
15/04/13 07:07:22 INFO WorkerWebUI: Started WorkerWebUI at
http://XXXX09:8081
15/04/13 07:07:22 INFO Worker: Connecting to master
akka.tcp://sparkMaster@nceuhamnr08:7077/user/Master...
15/04/13 07:07:22 INFO Worker: Successfully registered with master
spark://XXXX08:7077
*15/04/13 08:35:07 ERROR Worker: RECEIVED SIGNAL 15: SIGTERM*
Re: Spark Cluster: RECEIVED SIGNAL 15: SIGTERM
Posted by Guillaume Pitel <gu...@exensa.com>.
That's why I think it's the OOM killer. There are several cases of
memory overuse / errors :
1 - The application tries to allocate more than the Heap limit and GC
cannot free more memory => OutOfMemory : Java Heap Space exception from JVM
2 - The jvm is configured with a max heap size larger than the available
memory. At some point the application needs to allocate memory in JVM,
the JVM tries to extend its heap and allocate real memory (or maybe the
OS is configured with overcommit virtual memory), but fails => Kill
process of sacrifice child (or others, depending on various factors :
https://plumbr.eu/outofmemoryerror)
3 - The jvm has allocated its memory from the beginning and it has been
served, but other processes start starving from memory shortage, the
pressure on memory grows beyond the threshold configured in the OOM
Killer, and boom, the java process is selected for a sacrifice because
it is the main culprit of memory consumption.
Guillaume
> Linux OOM throws SIGTERM, but if I remember correctly JVM handles heap
> memory limits differently and throws OutOfMemoryError and eventually
> sends SIGINT.
>
> Not sure what happened but the worker simply received a SIGTERM
> signal, so perhaps the daemon was terminated by someone or a parent
> process. Just my guess.
>
> Tim
>
> On Mon, Apr 13, 2015 at 2:28 AM, Guillaume Pitel
> <guillaume.pitel@exensa.com <ma...@exensa.com>> wrote:
>
> Very likely to be this :
> http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html?page=2
>
> Your worker ran out of memory => maybe you're asking for too much
> memory for the JVM, or something else is running on the worker
>
> Guillaume
>> Any idea what this means, many thanks
>>
>> ==>
>> logs/spark-XXXX.XXXX-org.apache.spark.deploy.worker.Worker-1-XXXX09.out.1
>> <==
>> 15/04/13 07:07:22 INFO Worker: Starting Spark worker XXXX09:39910
>> with 4 cores, 6.6 GB RAM
>> 15/04/13 07:07:22 INFO Worker: Running Spark version 1.3.0
>> 15/04/13 07:07:22 INFO Worker: Spark home:
>> /remote/users/XXXX/work/tools/spark-1.3.0-bin-hadoop2.4
>> 15/04/13 07:07:22 INFO Server: jetty-8.y.z-SNAPSHOT
>> 15/04/13 07:07:22 INFO AbstractConnector: Started
>> SelectChannelConnector@0.0.0.0:8081
>> <http://SelectChannelConnector@0.0.0.0:8081>
>> 15/04/13 07:07:22 INFO Utils: Successfully started service
>> 'WorkerUI' on port 8081.
>> 15/04/13 07:07:22 INFO WorkerWebUI: Started WorkerWebUI at
>> http://XXXX09:8081
>> 15/04/13 07:07:22 INFO Worker: Connecting to master
>> akka.tcp://sparkMaster@nceuhamnr08:7077/user/Master...
>> 15/04/13 07:07:22 INFO Worker: Successfully registered with
>> master spark://XXXX08:7077
>> *15/04/13 08:35:07 ERROR Worker: RECEIVED SIGNAL 15: SIGTERM*
>>
>
>
> --
> eXenSa
>
>
> *Guillaume PITEL, Président*
> +33(0)626 222 431
>
> eXenSa S.A.S. <http://www.exensa.com/>
> 41, rue Périer - 92120 Montrouge - FRANCE
> Tel +33(0)184 163 677 / Fax +33(0)972 283 705
>
>
--
eXenSa
*Guillaume PITEL, Président*
+33(0)626 222 431
eXenSa S.A.S. <http://www.exensa.com/>
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)184 163 677 / Fax +33(0)972 283 705
Re: Spark Cluster: RECEIVED SIGNAL 15: SIGTERM
Posted by Tim Chen <ti...@mesosphere.io>.
Linux OOM throws SIGTERM, but if I remember correctly JVM handles heap
memory limits differently and throws OutOfMemoryError and eventually sends
SIGINT.
Not sure what happened but the worker simply received a SIGTERM signal, so
perhaps the daemon was terminated by someone or a parent process. Just my
guess.
Tim
On Mon, Apr 13, 2015 at 2:28 AM, Guillaume Pitel <guillaume.pitel@exensa.com
> wrote:
> Very likely to be this :
>
> http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html?page=2
>
> Your worker ran out of memory => maybe you're asking for too much memory
> for the JVM, or something else is running on the worker
>
> Guillaume
>
> Any idea what this means, many thanks
>
> ==>
> logs/spark-XXXX.XXXX-org.apache.spark.deploy.worker.Worker-1-XXXX09.out.1
> <==
> 15/04/13 07:07:22 INFO Worker: Starting Spark worker XXXX09:39910 with 4
> cores, 6.6 GB RAM
> 15/04/13 07:07:22 INFO Worker: Running Spark version 1.3.0
> 15/04/13 07:07:22 INFO Worker: Spark home:
> /remote/users/XXXX/work/tools/spark-1.3.0-bin-hadoop2.4
> 15/04/13 07:07:22 INFO Server: jetty-8.y.z-SNAPSHOT
> 15/04/13 07:07:22 INFO AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:8081
> 15/04/13 07:07:22 INFO Utils: Successfully started service 'WorkerUI' on
> port 8081.
> 15/04/13 07:07:22 INFO WorkerWebUI: Started WorkerWebUI at
> http://XXXX09:8081
> 15/04/13 07:07:22 INFO Worker: Connecting to master
> akka.tcp://sparkMaster@nceuhamnr08:7077/user/Master...
> 15/04/13 07:07:22 INFO Worker: Successfully registered with master
> spark://XXXX08:7077
> *15/04/13 08:35:07 ERROR Worker: RECEIVED SIGNAL 15: SIGTERM*
>
>
>
> --
> [image: eXenSa]
> *Guillaume PITEL, Président*
> +33(0)626 222 431
>
> eXenSa S.A.S. <http://www.exensa.com/>
> 41, rue Périer - 92120 Montrouge - FRANCE
> Tel +33(0)184 163 677 / Fax +33(0)972 283 705
>
Re: Spark Cluster: RECEIVED SIGNAL 15: SIGTERM
Posted by Guillaume Pitel <gu...@exensa.com>.
Very likely to be this :
http://www.linuxdevcenter.com/pub/a/linux/2006/11/30/linux-out-of-memory.html?page=2
Your worker ran out of memory => maybe you're asking for too much memory
for the JVM, or something else is running on the worker
Guillaume
> Any idea what this means, many thanks
>
> ==>
> logs/spark-XXXX.XXXX-org.apache.spark.deploy.worker.Worker-1-XXXX09.out.1
> <==
> 15/04/13 07:07:22 INFO Worker: Starting Spark worker XXXX09:39910 with
> 4 cores, 6.6 GB RAM
> 15/04/13 07:07:22 INFO Worker: Running Spark version 1.3.0
> 15/04/13 07:07:22 INFO Worker: Spark home:
> /remote/users/XXXX/work/tools/spark-1.3.0-bin-hadoop2.4
> 15/04/13 07:07:22 INFO Server: jetty-8.y.z-SNAPSHOT
> 15/04/13 07:07:22 INFO AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:8081
> <http://SelectChannelConnector@0.0.0.0:8081>
> 15/04/13 07:07:22 INFO Utils: Successfully started service 'WorkerUI'
> on port 8081.
> 15/04/13 07:07:22 INFO WorkerWebUI: Started WorkerWebUI at
> http://XXXX09:8081
> 15/04/13 07:07:22 INFO Worker: Connecting to master
> akka.tcp://sparkMaster@nceuhamnr08:7077/user/Master...
> 15/04/13 07:07:22 INFO Worker: Successfully registered with master
> spark://XXXX08:7077
> *15/04/13 08:35:07 ERROR Worker: RECEIVED SIGNAL 15: SIGTERM*
>
--
eXenSa
*Guillaume PITEL, Président*
+33(0)626 222 431
eXenSa S.A.S. <http://www.exensa.com/>
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)184 163 677 / Fax +33(0)972 283 705