You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Kevin <ke...@gmail.com> on 2015/04/28 21:41:22 UTC
Lifetime of jhist files
Hi,
I am running CDH5.1.3 with YARN. The one issue I am having is that the
jhist files are being deleted too quickly. I
set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of
course), but I am noticing that the job historyserver is expiring any jhist
file that is seven days old. I thought
yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on
the NodeManager; I didn't think it also affected the job history files too.
Am I right in my assumptions? Is there anyway to extend the lifetime of the
job history files?
BTW, my history files are stored in /user/history/done/<year>/<month>/<day>
Any feedback would be great.
Thanks,
Kevin
Re: Lifetime of jhist files
Posted by Kevin <ke...@gmail.com>.
Thanks, Naga, that worked. I didn't catch that property in the
mapred-default.xml. Sorry for such a late response.
On Tue, Apr 28, 2015 at 11:01 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Kevin,
>
> Could check the below configuration for the job history server
> mapreduce.jobhistory.max-age-ms 604800000 Job history files older than
> this many milliseconds will be deleted when the history cleaner runs.
> Defaults to 604800000 (1 week).
> you could refer other job history configs in* http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
> <http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml>*
>
> +Naga
>
> ------------------------------
> *From:* Kevin [kevin.macksamie@gmail.com]
> *Sent:* Wednesday, April 29, 2015 01:11
> *To:* user@hadoop.apache.org
> *Subject:* Lifetime of jhist files
>
> Hi,
>
> I am running CDH5.1.3 with YARN. The one issue I am having is that the
> jhist files are being deleted too quickly. I
> set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of
> course), but I am noticing that the job historyserver is expiring any jhist
> file that is seven days old. I thought
> yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on
> the NodeManager; I didn't think it also affected the job history files too.
>
> Am I right in my assumptions? Is there anyway to extend the lifetime of
> the job history files?
>
> BTW, my history files are stored in
> /user/history/done/<year>/<month>/<day>
>
> Any feedback would be great.
>
> Thanks,
> Kevin
>
Re: Lifetime of jhist files
Posted by Kevin <ke...@gmail.com>.
Thanks, Naga, that worked. I didn't catch that property in the
mapred-default.xml. Sorry for such a late response.
On Tue, Apr 28, 2015 at 11:01 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Kevin,
>
> Could check the below configuration for the job history server
> mapreduce.jobhistory.max-age-ms 604800000 Job history files older than
> this many milliseconds will be deleted when the history cleaner runs.
> Defaults to 604800000 (1 week).
> you could refer other job history configs in* http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
> <http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml>*
>
> +Naga
>
> ------------------------------
> *From:* Kevin [kevin.macksamie@gmail.com]
> *Sent:* Wednesday, April 29, 2015 01:11
> *To:* user@hadoop.apache.org
> *Subject:* Lifetime of jhist files
>
> Hi,
>
> I am running CDH5.1.3 with YARN. The one issue I am having is that the
> jhist files are being deleted too quickly. I
> set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of
> course), but I am noticing that the job historyserver is expiring any jhist
> file that is seven days old. I thought
> yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on
> the NodeManager; I didn't think it also affected the job history files too.
>
> Am I right in my assumptions? Is there anyway to extend the lifetime of
> the job history files?
>
> BTW, my history files are stored in
> /user/history/done/<year>/<month>/<day>
>
> Any feedback would be great.
>
> Thanks,
> Kevin
>
Re: Lifetime of jhist files
Posted by Kevin <ke...@gmail.com>.
Thanks, Naga, that worked. I didn't catch that property in the
mapred-default.xml. Sorry for such a late response.
On Tue, Apr 28, 2015 at 11:01 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Kevin,
>
> Could check the below configuration for the job history server
> mapreduce.jobhistory.max-age-ms 604800000 Job history files older than
> this many milliseconds will be deleted when the history cleaner runs.
> Defaults to 604800000 (1 week).
> you could refer other job history configs in* http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
> <http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml>*
>
> +Naga
>
> ------------------------------
> *From:* Kevin [kevin.macksamie@gmail.com]
> *Sent:* Wednesday, April 29, 2015 01:11
> *To:* user@hadoop.apache.org
> *Subject:* Lifetime of jhist files
>
> Hi,
>
> I am running CDH5.1.3 with YARN. The one issue I am having is that the
> jhist files are being deleted too quickly. I
> set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of
> course), but I am noticing that the job historyserver is expiring any jhist
> file that is seven days old. I thought
> yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on
> the NodeManager; I didn't think it also affected the job history files too.
>
> Am I right in my assumptions? Is there anyway to extend the lifetime of
> the job history files?
>
> BTW, my history files are stored in
> /user/history/done/<year>/<month>/<day>
>
> Any feedback would be great.
>
> Thanks,
> Kevin
>
Re: Lifetime of jhist files
Posted by Kevin <ke...@gmail.com>.
Thanks, Naga, that worked. I didn't catch that property in the
mapred-default.xml. Sorry for such a late response.
On Tue, Apr 28, 2015 at 11:01 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Kevin,
>
> Could check the below configuration for the job history server
> mapreduce.jobhistory.max-age-ms 604800000 Job history files older than
> this many milliseconds will be deleted when the history cleaner runs.
> Defaults to 604800000 (1 week).
> you could refer other job history configs in* http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
> <http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml>*
>
> +Naga
>
> ------------------------------
> *From:* Kevin [kevin.macksamie@gmail.com]
> *Sent:* Wednesday, April 29, 2015 01:11
> *To:* user@hadoop.apache.org
> *Subject:* Lifetime of jhist files
>
> Hi,
>
> I am running CDH5.1.3 with YARN. The one issue I am having is that the
> jhist files are being deleted too quickly. I
> set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of
> course), but I am noticing that the job historyserver is expiring any jhist
> file that is seven days old. I thought
> yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on
> the NodeManager; I didn't think it also affected the job history files too.
>
> Am I right in my assumptions? Is there anyway to extend the lifetime of
> the job history files?
>
> BTW, my history files are stored in
> /user/history/done/<year>/<month>/<day>
>
> Any feedback would be great.
>
> Thanks,
> Kevin
>
RE: Lifetime of jhist files
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Kevin,
Could check the below configuration for the job history server
mapreduce.jobhistory.max-age-ms 604800000 Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
you could refer other job history configs in http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
+Naga
________________________________
From: Kevin [kevin.macksamie@gmail.com]
Sent: Wednesday, April 29, 2015 01:11
To: user@hadoop.apache.org
Subject: Lifetime of jhist files
Hi,
I am running CDH5.1.3 with YARN. The one issue I am having is that the jhist files are being deleted too quickly. I set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of course), but I am noticing that the job historyserver is expiring any jhist file that is seven days old. I thought yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on the NodeManager; I didn't think it also affected the job history files too.
Am I right in my assumptions? Is there anyway to extend the lifetime of the job history files?
BTW, my history files are stored in /user/history/done/<year>/<month>/<day>
Any feedback would be great.
Thanks,
Kevin
RE: Lifetime of jhist files
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Kevin,
Could check the below configuration for the job history server
mapreduce.jobhistory.max-age-ms 604800000 Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
you could refer other job history configs in http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
+Naga
________________________________
From: Kevin [kevin.macksamie@gmail.com]
Sent: Wednesday, April 29, 2015 01:11
To: user@hadoop.apache.org
Subject: Lifetime of jhist files
Hi,
I am running CDH5.1.3 with YARN. The one issue I am having is that the jhist files are being deleted too quickly. I set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of course), but I am noticing that the job historyserver is expiring any jhist file that is seven days old. I thought yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on the NodeManager; I didn't think it also affected the job history files too.
Am I right in my assumptions? Is there anyway to extend the lifetime of the job history files?
BTW, my history files are stored in /user/history/done/<year>/<month>/<day>
Any feedback would be great.
Thanks,
Kevin
RE: Lifetime of jhist files
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Kevin,
Could check the below configuration for the job history server
mapreduce.jobhistory.max-age-ms 604800000 Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
you could refer other job history configs in http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
+Naga
________________________________
From: Kevin [kevin.macksamie@gmail.com]
Sent: Wednesday, April 29, 2015 01:11
To: user@hadoop.apache.org
Subject: Lifetime of jhist files
Hi,
I am running CDH5.1.3 with YARN. The one issue I am having is that the jhist files are being deleted too quickly. I set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of course), but I am noticing that the job historyserver is expiring any jhist file that is seven days old. I thought yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on the NodeManager; I didn't think it also affected the job history files too.
Am I right in my assumptions? Is there anyway to extend the lifetime of the job history files?
BTW, my history files are stored in /user/history/done/<year>/<month>/<day>
Any feedback would be great.
Thanks,
Kevin
RE: Lifetime of jhist files
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Kevin,
Could check the below configuration for the job history server
mapreduce.jobhistory.max-age-ms 604800000 Job history files older than this many milliseconds will be deleted when the history cleaner runs. Defaults to 604800000 (1 week).
you could refer other job history configs in http://hadoop.apache.org/docs/r2.7.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml
+Naga
________________________________
From: Kevin [kevin.macksamie@gmail.com]
Sent: Wednesday, April 29, 2015 01:11
To: user@hadoop.apache.org
Subject: Lifetime of jhist files
Hi,
I am running CDH5.1.3 with YARN. The one issue I am having is that the jhist files are being deleted too quickly. I set yarn.nodemanager.delete.debug-delay-sec to seven days (in seconds, of course), but I am noticing that the job historyserver is expiring any jhist file that is seven days old. I thought yarn.nodemanager.delete.debug-delay-sec was just to clean up local logs on the NodeManager; I didn't think it also affected the job history files too.
Am I right in my assumptions? Is there anyway to extend the lifetime of the job history files?
BTW, my history files are stored in /user/history/done/<year>/<month>/<day>
Any feedback would be great.
Thanks,
Kevin