You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Prashant Kommireddi <pr...@gmail.com> on 2012/09/14 23:37:27 UTC

Job history logging

Hi All,

I have a question about job history logging. Seems like history logging is
disabled if file creation fails, is there a reason this is done?
The following snippet is from JobHistory.JobInfo.logSubmitted(....)  -
Hadoop 0.20.2


          // Log the history meta info
          JobHistory.MetaInfoManager.logMetaInfo(writers);

          //add to writer as well
          JobHistory.log(writers, RecordTypes.Job,
                         new Keys[]{Keys.JOBID, Keys.JOBNAME, Keys.USER,
Keys.SUBMIT_TIME, Keys.JOBCONF },
                         new String[]{jobId.toString(), jobName, user,
                                      String.valueOf(submitTime) ,
jobConfPath}
                        );

        }catch(IOException e){
          LOG.error("Failed creating job history log file, disabling
history", e);
          *disableHistory = true; *
        }
      }


Thanks,

Re: Job history logging

Posted by Prashant Kommireddi <pr...@gmail.com>.

Thanks Harsh.

Doesn't seem like the assumption is a correct one? In case when disk
space is exhausted and JT stops writing history logs, does it mean we
require a JT restart for logs to be enabled again?

In my case, I am seeing JT trying to write logs with a different user
than the superuser. I am not sure why this is happening either, but
the attempt to write fails as the other user does not have
permissions.

On Sep 14, 2012, at 7:11 PM, Harsh J <ha...@cloudera.com> wrote:

> I guess the reason is that it assumes it can't write history files
> after that point, and skips the rest of the work?
>
> On Sat, Sep 15, 2012 at 3:07 AM, Prashant Kommireddi
> <pr...@gmail.com> wrote:
>> Hi All,
>>
>> I have a question about job history logging. Seems like history logging is
>> disabled if file creation fails, is there a reason this is done?
>> The following snippet is from JobHistory.JobInfo.logSubmitted(....)  -
>> Hadoop 0.20.2
>>
>>
>>          // Log the history meta info
>>          JobHistory.MetaInfoManager.logMetaInfo(writers);
>>
>>          //add to writer as well
>>          JobHistory.log(writers, RecordTypes.Job,
>>                         new Keys[]{Keys.JOBID, Keys.JOBNAME, Keys.USER,
>> Keys.SUBMIT_TIME, Keys.JOBCONF },
>>                         new String[]{jobId.toString(), jobName, user,
>>                                      String.valueOf(submitTime) ,
>> jobConfPath}
>>                        );
>>
>>        }catch(IOException e){
>>          LOG.error("Failed creating job history log file, disabling
>> history", e);
>>          *disableHistory = true; *
>>        }
>>      }
>>
>>
>> Thanks,
>
>
>
> --
> Harsh J

Re: Job history logging

Posted by Harsh J <ha...@cloudera.com>.

I guess the reason is that it assumes it can't write history files
after that point, and skips the rest of the work?

On Sat, Sep 15, 2012 at 3:07 AM, Prashant Kommireddi
<pr...@gmail.com> wrote:
> Hi All,
>
> I have a question about job history logging. Seems like history logging is
> disabled if file creation fails, is there a reason this is done?
> The following snippet is from JobHistory.JobInfo.logSubmitted(....)  -
> Hadoop 0.20.2
>
>
>           // Log the history meta info
>           JobHistory.MetaInfoManager.logMetaInfo(writers);
>
>           //add to writer as well
>           JobHistory.log(writers, RecordTypes.Job,
>                          new Keys[]{Keys.JOBID, Keys.JOBNAME, Keys.USER,
> Keys.SUBMIT_TIME, Keys.JOBCONF },
>                          new String[]{jobId.toString(), jobName, user,
>                                       String.valueOf(submitTime) ,
> jobConfPath}
>                         );
>
>         }catch(IOException e){
>           LOG.error("Failed creating job history log file, disabling
> history", e);
>           *disableHistory = true; *
>         }
>       }
>
>
> Thanks,



-- 
Harsh J