You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by zheyi rong <zh...@gmail.com> on 2013/04/02 12:00:44 UTC
Job log location and retention
Dear all,
I would like to ask why the logs (configurations and status) of my job
disappeared in the Jobtracker web UI?
Specifically, I finished a job four days ago, but I cannot find it in the
Jobtracker web UI, neither in the
homepage nor "Job Tracker History" on the left-bottom corner.
The cluster is deployed with CDH3, and users actively submit jobs.
Some configurations:
*mapred.jobtracker.retirejob.interval*3600000
*mapred.jobtracker.completeuserjobs.maximum*5
Thank you very much.
Regards,
Zheyi Rong
Re: Job log location and retention
Posted by zheyi rong <zh...@gmail.com>.
Thank you for the answer.
However, I do not have the access to the jobtracker's local system...
Fortunately I have found a workaround -- using
$hadoop job -history [output]
Regards,
Zheyi.
Regards,
Zheyi Rong
On Wed, Apr 3, 2013 at 2:06 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Zheyi,
>
> The jobtracker doesn't keep a reference to the job to save memory, but you
> may still find it in the filesystem. For a default CDH3 installation, it
> will be in the jobtracker's local filesystem, at
> /var/log/hadoop-0.20/history/done/
>
> Logs from individual tasks are a little trickier to find. They will be in
> the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if
> they haven't been deleted yet -- deletion is controled by
> mapreduce.job.userlog.retain.hours, and the default is 24 hours.
>
> Regards,
> Marcos
>
>
> I would like to ask why the logs (configurations and status) of my job
> disappeared in the Jobtracker web UI?
>
> Specifically, I finished a job four days ago, but I cannot find it in
> the Jobtracker web UI, neither in the
> homepage nor "Job Tracker History" on the left-bottom corner.
>
> The cluster is deployed with CDH3, and users actively submit jobs.
> Some configurations:
> *mapred.jobtracker.retirejob.interval* 3600000
> *mapred.jobtracker.completeuserjobs.maximum* 5
>
>
>
Re: Job log location and retention
Posted by zheyi rong <zh...@gmail.com>.
Thank you for the answer.
However, I do not have the access to the jobtracker's local system...
Fortunately I have found a workaround -- using
$hadoop job -history [output]
Regards,
Zheyi.
Regards,
Zheyi Rong
On Wed, Apr 3, 2013 at 2:06 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Zheyi,
>
> The jobtracker doesn't keep a reference to the job to save memory, but you
> may still find it in the filesystem. For a default CDH3 installation, it
> will be in the jobtracker's local filesystem, at
> /var/log/hadoop-0.20/history/done/
>
> Logs from individual tasks are a little trickier to find. They will be in
> the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if
> they haven't been deleted yet -- deletion is controled by
> mapreduce.job.userlog.retain.hours, and the default is 24 hours.
>
> Regards,
> Marcos
>
>
> I would like to ask why the logs (configurations and status) of my job
> disappeared in the Jobtracker web UI?
>
> Specifically, I finished a job four days ago, but I cannot find it in
> the Jobtracker web UI, neither in the
> homepage nor "Job Tracker History" on the left-bottom corner.
>
> The cluster is deployed with CDH3, and users actively submit jobs.
> Some configurations:
> *mapred.jobtracker.retirejob.interval* 3600000
> *mapred.jobtracker.completeuserjobs.maximum* 5
>
>
>
Re: Job log location and retention
Posted by zheyi rong <zh...@gmail.com>.
Thank you for the answer.
However, I do not have the access to the jobtracker's local system...
Fortunately I have found a workaround -- using
$hadoop job -history [output]
Regards,
Zheyi.
Regards,
Zheyi Rong
On Wed, Apr 3, 2013 at 2:06 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Zheyi,
>
> The jobtracker doesn't keep a reference to the job to save memory, but you
> may still find it in the filesystem. For a default CDH3 installation, it
> will be in the jobtracker's local filesystem, at
> /var/log/hadoop-0.20/history/done/
>
> Logs from individual tasks are a little trickier to find. They will be in
> the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if
> they haven't been deleted yet -- deletion is controled by
> mapreduce.job.userlog.retain.hours, and the default is 24 hours.
>
> Regards,
> Marcos
>
>
> I would like to ask why the logs (configurations and status) of my job
> disappeared in the Jobtracker web UI?
>
> Specifically, I finished a job four days ago, but I cannot find it in
> the Jobtracker web UI, neither in the
> homepage nor "Job Tracker History" on the left-bottom corner.
>
> The cluster is deployed with CDH3, and users actively submit jobs.
> Some configurations:
> *mapred.jobtracker.retirejob.interval* 3600000
> *mapred.jobtracker.completeuserjobs.maximum* 5
>
>
>
Re: Job log location and retention
Posted by zheyi rong <zh...@gmail.com>.
Thank you for the answer.
However, I do not have the access to the jobtracker's local system...
Fortunately I have found a workaround -- using
$hadoop job -history [output]
Regards,
Zheyi.
Regards,
Zheyi Rong
On Wed, Apr 3, 2013 at 2:06 PM, MARCOS MEDRADO RUBINELLI <
marcosm@buscapecompany.com> wrote:
> Zheyi,
>
> The jobtracker doesn't keep a reference to the job to save memory, but you
> may still find it in the filesystem. For a default CDH3 installation, it
> will be in the jobtracker's local filesystem, at
> /var/log/hadoop-0.20/history/done/
>
> Logs from individual tasks are a little trickier to find. They will be in
> the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if
> they haven't been deleted yet -- deletion is controled by
> mapreduce.job.userlog.retain.hours, and the default is 24 hours.
>
> Regards,
> Marcos
>
>
> I would like to ask why the logs (configurations and status) of my job
> disappeared in the Jobtracker web UI?
>
> Specifically, I finished a job four days ago, but I cannot find it in
> the Jobtracker web UI, neither in the
> homepage nor "Job Tracker History" on the left-bottom corner.
>
> The cluster is deployed with CDH3, and users actively submit jobs.
> Some configurations:
> *mapred.jobtracker.retirejob.interval* 3600000
> *mapred.jobtracker.completeuserjobs.maximum* 5
>
>
>
Re: Job log location and retention
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Zheyi,
The jobtracker doesn't keep a reference to the job to save memory, but you may still find it in the filesystem. For a default CDH3 installation, it will be in the jobtracker's local filesystem, at /var/log/hadoop-0.20/history/done/
Logs from individual tasks are a little trickier to find. They will be in the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if they haven't been deleted yet -- deletion is controled by mapreduce.job.userlog.retain.hours, and the default is 24 hours.
Regards,
Marcos
I would like to ask why the logs (configurations and status) of my job disappeared in the Jobtracker web UI?
Specifically, I finished a job four days ago, but I cannot find it in the Jobtracker web UI, neither in the
homepage nor "Job Tracker History" on the left-bottom corner.
The cluster is deployed with CDH3, and users actively submit jobs.
Some configurations:
mapred.jobtracker.retirejob.interval 3600000
mapred.jobtracker.completeuserjobs.maximum 5
Re: Job log location and retention
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Zheyi,
The jobtracker doesn't keep a reference to the job to save memory, but you may still find it in the filesystem. For a default CDH3 installation, it will be in the jobtracker's local filesystem, at /var/log/hadoop-0.20/history/done/
Logs from individual tasks are a little trickier to find. They will be in the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if they haven't been deleted yet -- deletion is controled by mapreduce.job.userlog.retain.hours, and the default is 24 hours.
Regards,
Marcos
I would like to ask why the logs (configurations and status) of my job disappeared in the Jobtracker web UI?
Specifically, I finished a job four days ago, but I cannot find it in the Jobtracker web UI, neither in the
homepage nor "Job Tracker History" on the left-bottom corner.
The cluster is deployed with CDH3, and users actively submit jobs.
Some configurations:
mapred.jobtracker.retirejob.interval 3600000
mapred.jobtracker.completeuserjobs.maximum 5
Re: Job log location and retention
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Zheyi,
The jobtracker doesn't keep a reference to the job to save memory, but you may still find it in the filesystem. For a default CDH3 installation, it will be in the jobtracker's local filesystem, at /var/log/hadoop-0.20/history/done/
Logs from individual tasks are a little trickier to find. They will be in the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if they haven't been deleted yet -- deletion is controled by mapreduce.job.userlog.retain.hours, and the default is 24 hours.
Regards,
Marcos
I would like to ask why the logs (configurations and status) of my job disappeared in the Jobtracker web UI?
Specifically, I finished a job four days ago, but I cannot find it in the Jobtracker web UI, neither in the
homepage nor "Job Tracker History" on the left-bottom corner.
The cluster is deployed with CDH3, and users actively submit jobs.
Some configurations:
mapred.jobtracker.retirejob.interval 3600000
mapred.jobtracker.completeuserjobs.maximum 5
Re: Job log location and retention
Posted by MARCOS MEDRADO RUBINELLI <ma...@buscapecompany.com>.
Zheyi,
The jobtracker doesn't keep a reference to the job to save memory, but you may still find it in the filesystem. For a default CDH3 installation, it will be in the jobtracker's local filesystem, at /var/log/hadoop-0.20/history/done/
Logs from individual tasks are a little trickier to find. They will be in the node that ran that task, at /var/log/hadoop-0.20/history/userlogs if they haven't been deleted yet -- deletion is controled by mapreduce.job.userlog.retain.hours, and the default is 24 hours.
Regards,
Marcos
I would like to ask why the logs (configurations and status) of my job disappeared in the Jobtracker web UI?
Specifically, I finished a job four days ago, but I cannot find it in the Jobtracker web UI, neither in the
homepage nor "Job Tracker History" on the left-bottom corner.
The cluster is deployed with CDH3, and users actively submit jobs.
Some configurations:
mapred.jobtracker.retirejob.interval 3600000
mapred.jobtracker.completeuserjobs.maximum 5