You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2014/09/02 22:02:08 UTC
YARN userapp cache lifetime: can't find core dump
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
Re: YARN userapp cache lifetime: can't find core dump
Posted by Hitesh Shah <hi...@apache.org>.
Hi John
You need to set "yarn.nodemanager.delete.debug-delay-sec” to a non-zero value. Will need an NM restart to bring this change into effect though.
— Hitesh
On Sep 2, 2014, at 1:02 PM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
> <winmail.dat>
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
Shahab,
Thanks, but I think that is just for log aggregation.
I want to retain the entire localized directory structure for a YARN task, including any files written to that place, after the task has exited.
John
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, September 02, 2014 2:13 PM
To: user@hadoop.apache.org
Subject: Re: YARN userapp cache lifetime: can't find core dump
Perhaps the following? I get the application logs from here after job completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
Shahab,
Thanks, but I think that is just for log aggregation.
I want to retain the entire localized directory structure for a YARN task, including any files written to that place, after the task has exited.
John
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, September 02, 2014 2:13 PM
To: user@hadoop.apache.org
Subject: Re: YARN userapp cache lifetime: can't find core dump
Perhaps the following? I get the application logs from here after job completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
Shahab,
Thanks, but I think that is just for log aggregation.
I want to retain the entire localized directory structure for a YARN task, including any files written to that place, after the task has exited.
John
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, September 02, 2014 2:13 PM
To: user@hadoop.apache.org
Subject: Re: YARN userapp cache lifetime: can't find core dump
Perhaps the following? I get the application logs from here after job completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
Shahab,
Thanks, but I think that is just for log aggregation.
I want to retain the entire localized directory structure for a YARN task, including any files written to that place, after the task has exited.
John
From: Shahab Yunus [mailto:shahab.yunus@gmail.com]
Sent: Tuesday, September 02, 2014 2:13 PM
To: user@hadoop.apache.org
Subject: Re: YARN userapp cache lifetime: can't find core dump
Perhaps the following? I get the application logs from here after job completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>> wrote:
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
Re: YARN userapp cache lifetime: can't find core dump
Posted by Shahab Yunus <sh...@gmail.com>.
Perhaps the following? I get the application logs from here after job
completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>
wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location:
> /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core
> or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files
> are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
>
Re: YARN userapp cache lifetime: can't find core dump
Posted by Hitesh Shah <hi...@apache.org>.
Hi John
You need to set "yarn.nodemanager.delete.debug-delay-sec” to a non-zero value. Will need an NM restart to bring this change into effect though.
— Hitesh
On Sep 2, 2014, at 1:02 PM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
> <winmail.dat>
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
I think I found it:
yarn.nodemanager.delete.debug-delay-sec
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Tuesday, September 02, 2014 2:02 PM
To: 'user@hadoop.apache.org'
Subject: YARN userapp cache lifetime: can't find core dump
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
Re: YARN userapp cache lifetime: can't find core dump
Posted by Shahab Yunus <sh...@gmail.com>.
Perhaps the following? I get the application logs from here after job
completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>
wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location:
> /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core
> or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files
> are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
>
Re: YARN userapp cache lifetime: can't find core dump
Posted by Shahab Yunus <sh...@gmail.com>.
Perhaps the following? I get the application logs from here after job
completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>
wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location:
> /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core
> or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files
> are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
>
Re: YARN userapp cache lifetime: can't find core dump
Posted by Hitesh Shah <hi...@apache.org>.
Hi John
You need to set "yarn.nodemanager.delete.debug-delay-sec” to a non-zero value. Will need an NM restart to bring this change into effect though.
— Hitesh
On Sep 2, 2014, at 1:02 PM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
> <winmail.dat>
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
I think I found it:
yarn.nodemanager.delete.debug-delay-sec
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Tuesday, September 02, 2014 2:02 PM
To: 'user@hadoop.apache.org'
Subject: YARN userapp cache lifetime: can't find core dump
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
I think I found it:
yarn.nodemanager.delete.debug-delay-sec
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Tuesday, September 02, 2014 2:02 PM
To: 'user@hadoop.apache.org'
Subject: YARN userapp cache lifetime: can't find core dump
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
RE: YARN userapp cache lifetime: can't find core dump
Posted by John Lilley <jo...@redpoint.net>.
I think I found it:
yarn.nodemanager.delete.debug-delay-sec
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Tuesday, September 02, 2014 2:02 PM
To: 'user@hadoop.apache.org'
Subject: YARN userapp cache lifetime: can't find core dump
We have a YARN task that is core-dumping, and the JVM error log says:
# Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
However when I look at the node, everything below here is empty
/data2/hadoop/yarn/local/usercache/jlilley/appcache
I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
Thanks,
john
Re: YARN userapp cache lifetime: can't find core dump
Posted by Hitesh Shah <hi...@apache.org>.
Hi John
You need to set "yarn.nodemanager.delete.debug-delay-sec” to a non-zero value. Will need an NM restart to bring this change into effect though.
— Hitesh
On Sep 2, 2014, at 1:02 PM, John Lilley <jo...@redpoint.net> wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location: /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
> <winmail.dat>
Re: YARN userapp cache lifetime: can't find core dump
Posted by Shahab Yunus <sh...@gmail.com>.
Perhaps the following? I get the application logs from here after job
completion. This is path on hdfs.
yarn.nodemanager.remote-app-log-dir
Regards,
Shahab
On Tue, Sep 2, 2014 at 4:02 PM, John Lilley <jo...@redpoint.net>
wrote:
> We have a YARN task that is core-dumping, and the JVM error log says:
> # Core dump written. Default location:
> /data2/hadoop/yarn/local/usercache/jlilley/appcache/application_1405724043176_2453/container_1405724043176_2453_01_000002/core
> or core.14801
>
> However when I look at the node, everything below here is empty
> /data2/hadoop/yarn/local/usercache/jlilley/appcache
>
> I seem to recall there is a YARN setting to control the time these files
> are kept around after application exit, but I can't figure out what it is.
>
> Thanks,
> john
>
>