You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Ivan Tretyakov <it...@griddynamics.com> on 2013/01/09 15:41:58 UTC

JobCache directory cleanup

Hello!

I've found that jobcache directory became very large on our cluster, e.g.:

# du -sh /data?/mapred/local/taskTracker/user/jobcache
465G    /data1/mapred/local/taskTracker/user/jobcache
464G    /data2/mapred/local/taskTracker/user/jobcache
454G    /data3/mapred/local/taskTracker/user/jobcache

And it stores information for about 100 jobs:

# ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
wc -l

I've found that there is following parameter:

<property>
  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
  <value>1000</value>
  <description>The number of retired job status to keep in the cache.
  </description>
</property>

So, if I got it right it intended to control job cache size by limiting
number of jobs to store cache for.

Also, I've seen that some hadoop users uses cron approach to cleanup
jobcache:
http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
 (
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
)

Are there other approaches to control jobcache size?
What is more correct way to do it?

Thanks in advance!

P.S. We are using CDH 4.1.1.

-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot!

That was it. There was following line in our code:
  jobConf.setKeepTaskFilesPattern(".*");


On Fri, Jan 11, 2013 at 2:20 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hmm. Unfortunately, there is another config variable that may be affecting
> this: keep.task.files.pattern
>
> This is set to .* in the job.xml file you sent. I suspect this may be
> causing a problem. Can you please remove this, assuming you have not set it
> intentionally ?
>
> Thanks
> Hemanth
>
>
>
> On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Thanks for replies!
>>
>> keep.failed.task.files set to false.
>> Config of one of the jobs attached.
>>
>>
>> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
>> yhemanth@thoughtworks.com> wrote:
>>
>>> Good point. Forgot that one :-)
>>>
>>>
>>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>>
>>>>
>>>> Can you check the job configuration for these ~100 jobs? Do they have
>>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>>> it doesn't, it could be a bug.
>>>>
>>>> Sharing your configs for these jobs will definitely help.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>>
>>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>>> itretyakov@griddynamics.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I've found that jobcache directory became very large on our cluster,
>>>>> e.g.:
>>>>>
>>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>>
>>>>> And it stores information for about 100 jobs:
>>>>>
>>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>>> uniq | wc -l
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot!

That was it. There was following line in our code:
  jobConf.setKeepTaskFilesPattern(".*");


On Fri, Jan 11, 2013 at 2:20 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hmm. Unfortunately, there is another config variable that may be affecting
> this: keep.task.files.pattern
>
> This is set to .* in the job.xml file you sent. I suspect this may be
> causing a problem. Can you please remove this, assuming you have not set it
> intentionally ?
>
> Thanks
> Hemanth
>
>
>
> On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Thanks for replies!
>>
>> keep.failed.task.files set to false.
>> Config of one of the jobs attached.
>>
>>
>> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
>> yhemanth@thoughtworks.com> wrote:
>>
>>> Good point. Forgot that one :-)
>>>
>>>
>>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>>
>>>>
>>>> Can you check the job configuration for these ~100 jobs? Do they have
>>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>>> it doesn't, it could be a bug.
>>>>
>>>> Sharing your configs for these jobs will definitely help.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>>
>>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>>> itretyakov@griddynamics.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I've found that jobcache directory became very large on our cluster,
>>>>> e.g.:
>>>>>
>>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>>
>>>>> And it stores information for about 100 jobs:
>>>>>
>>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>>> uniq | wc -l
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot!

That was it. There was following line in our code:
  jobConf.setKeepTaskFilesPattern(".*");


On Fri, Jan 11, 2013 at 2:20 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hmm. Unfortunately, there is another config variable that may be affecting
> this: keep.task.files.pattern
>
> This is set to .* in the job.xml file you sent. I suspect this may be
> causing a problem. Can you please remove this, assuming you have not set it
> intentionally ?
>
> Thanks
> Hemanth
>
>
>
> On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Thanks for replies!
>>
>> keep.failed.task.files set to false.
>> Config of one of the jobs attached.
>>
>>
>> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
>> yhemanth@thoughtworks.com> wrote:
>>
>>> Good point. Forgot that one :-)
>>>
>>>
>>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>>
>>>>
>>>> Can you check the job configuration for these ~100 jobs? Do they have
>>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>>> it doesn't, it could be a bug.
>>>>
>>>> Sharing your configs for these jobs will definitely help.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>>
>>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>>> itretyakov@griddynamics.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I've found that jobcache directory became very large on our cluster,
>>>>> e.g.:
>>>>>
>>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>>
>>>>> And it stores information for about 100 jobs:
>>>>>
>>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>>> uniq | wc -l
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot!

That was it. There was following line in our code:
  jobConf.setKeepTaskFilesPattern(".*");


On Fri, Jan 11, 2013 at 2:20 PM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hmm. Unfortunately, there is another config variable that may be affecting
> this: keep.task.files.pattern
>
> This is set to .* in the job.xml file you sent. I suspect this may be
> causing a problem. Can you please remove this, assuming you have not set it
> intentionally ?
>
> Thanks
> Hemanth
>
>
>
> On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Thanks for replies!
>>
>> keep.failed.task.files set to false.
>> Config of one of the jobs attached.
>>
>>
>> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
>> yhemanth@thoughtworks.com> wrote:
>>
>>> Good point. Forgot that one :-)
>>>
>>>
>>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>>> vinodkv@hortonworks.com> wrote:
>>>
>>>>
>>>>
>>>> Can you check the job configuration for these ~100 jobs? Do they have
>>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>>> it doesn't, it could be a bug.
>>>>
>>>> Sharing your configs for these jobs will definitely help.
>>>>
>>>> Thanks,
>>>> +Vinod
>>>>
>>>>
>>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>>> itretyakov@griddynamics.com> wrote:
>>>>
>>>>> Hello!
>>>>>
>>>>> I've found that jobcache directory became very large on our cluster,
>>>>> e.g.:
>>>>>
>>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>>
>>>>> And it stores information for about 100 jobs:
>>>>>
>>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>>> uniq | wc -l
>>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hmm. Unfortunately, there is another config variable that may be affecting
this: keep.task.files.pattern

This is set to .* in the job.xml file you sent. I suspect this may be
causing a problem. Can you please remove this, assuming you have not set it
intentionally ?

Thanks
Hemanth



On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> keep.failed.task.files set to false.
> Config of one of the jobs attached.
>
>
> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Good point. Forgot that one :-)
>>
>>
>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>>
>>>
>>> Can you check the job configuration for these ~100 jobs? Do they have
>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>> it doesn't, it could be a bug.
>>>
>>> Sharing your configs for these jobs will definitely help.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>>
>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> I've found that jobcache directory became very large on our cluster,
>>>> e.g.:
>>>>
>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>
>>>> And it stores information for about 100 jobs:
>>>>
>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>> uniq | wc -l
>>>>
>>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hmm. Unfortunately, there is another config variable that may be affecting
this: keep.task.files.pattern

This is set to .* in the job.xml file you sent. I suspect this may be
causing a problem. Can you please remove this, assuming you have not set it
intentionally ?

Thanks
Hemanth



On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> keep.failed.task.files set to false.
> Config of one of the jobs attached.
>
>
> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Good point. Forgot that one :-)
>>
>>
>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>>
>>>
>>> Can you check the job configuration for these ~100 jobs? Do they have
>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>> it doesn't, it could be a bug.
>>>
>>> Sharing your configs for these jobs will definitely help.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>>
>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> I've found that jobcache directory became very large on our cluster,
>>>> e.g.:
>>>>
>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>
>>>> And it stores information for about 100 jobs:
>>>>
>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>> uniq | wc -l
>>>>
>>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hmm. Unfortunately, there is another config variable that may be affecting
this: keep.task.files.pattern

This is set to .* in the job.xml file you sent. I suspect this may be
causing a problem. Can you please remove this, assuming you have not set it
intentionally ?

Thanks
Hemanth



On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> keep.failed.task.files set to false.
> Config of one of the jobs attached.
>
>
> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Good point. Forgot that one :-)
>>
>>
>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>>
>>>
>>> Can you check the job configuration for these ~100 jobs? Do they have
>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>> it doesn't, it could be a bug.
>>>
>>> Sharing your configs for these jobs will definitely help.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>>
>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> I've found that jobcache directory became very large on our cluster,
>>>> e.g.:
>>>>
>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>
>>>> And it stores information for about 100 jobs:
>>>>
>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>> uniq | wc -l
>>>>
>>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hmm. Unfortunately, there is another config variable that may be affecting
this: keep.task.files.pattern

This is set to .* in the job.xml file you sent. I suspect this may be
causing a problem. Can you please remove this, assuming you have not set it
intentionally ?

Thanks
Hemanth



On Fri, Jan 11, 2013 at 3:28 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> keep.failed.task.files set to false.
> Config of one of the jobs attached.
>
>
> On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Good point. Forgot that one :-)
>>
>>
>> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
>> vinodkv@hortonworks.com> wrote:
>>
>>>
>>>
>>> Can you check the job configuration for these ~100 jobs? Do they have
>>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>>> it doesn't, it could be a bug.
>>>
>>> Sharing your configs for these jobs will definitely help.
>>>
>>> Thanks,
>>> +Vinod
>>>
>>>
>>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>>> itretyakov@griddynamics.com> wrote:
>>>
>>>> Hello!
>>>>
>>>> I've found that jobcache directory became very large on our cluster,
>>>> e.g.:
>>>>
>>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>>
>>>> And it stores information for about 100 jobs:
>>>>
>>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>>>> uniq | wc -l
>>>>
>>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

keep.failed.task.files set to false.
Config of one of the jobs attached.


On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Good point. Forgot that one :-)
>
>
> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>>
>>
>> Can you check the job configuration for these ~100 jobs? Do they have
>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>> it doesn't, it could be a bug.
>>
>> Sharing your configs for these jobs will definitely help.
>>
>> Thanks,
>> +Vinod
>>
>>
>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

keep.failed.task.files set to false.
Config of one of the jobs attached.


On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Good point. Forgot that one :-)
>
>
> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>>
>>
>> Can you check the job configuration for these ~100 jobs? Do they have
>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>> it doesn't, it could be a bug.
>>
>> Sharing your configs for these jobs will definitely help.
>>
>> Thanks,
>> +Vinod
>>
>>
>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

keep.failed.task.files set to false.
Config of one of the jobs attached.


On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Good point. Forgot that one :-)
>
>
> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>>
>>
>> Can you check the job configuration for these ~100 jobs? Do they have
>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>> it doesn't, it could be a bug.
>>
>> Sharing your configs for these jobs will definitely help.
>>
>> Thanks,
>> +Vinod
>>
>>
>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

keep.failed.task.files set to false.
Config of one of the jobs attached.


On Fri, Jan 11, 2013 at 5:44 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Good point. Forgot that one :-)
>
>
> On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
> vinodkv@hortonworks.com> wrote:
>
>>
>>
>> Can you check the job configuration for these ~100 jobs? Do they have
>> keep.failed.task.files set to true? If so, these files won't be deleted. If
>> it doesn't, it could be a bug.
>>
>> Sharing your configs for these jobs will definitely help.
>>
>> Thanks,
>> +Vinod
>>
>>
>> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Good point. Forgot that one :-)


On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

>
>
> Can you check the job configuration for these ~100 jobs? Do they have
> keep.failed.task.files set to true? If so, these files won't be deleted. If
> it doesn't, it could be a bug.
>
> Sharing your configs for these jobs will definitely help.
>
> Thanks,
> +Vinod
>
>
> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Good point. Forgot that one :-)


On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

>
>
> Can you check the job configuration for these ~100 jobs? Do they have
> keep.failed.task.files set to true? If so, these files won't be deleted. If
> it doesn't, it could be a bug.
>
> Sharing your configs for these jobs will definitely help.
>
> Thanks,
> +Vinod
>
>
> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Good point. Forgot that one :-)


On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

>
>
> Can you check the job configuration for these ~100 jobs? Do they have
> keep.failed.task.files set to true? If so, these files won't be deleted. If
> it doesn't, it could be a bug.
>
> Sharing your configs for these jobs will definitely help.
>
> Thanks,
> +Vinod
>
>
> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Good point. Forgot that one :-)


On Thu, Jan 10, 2013 at 10:53 PM, Vinod Kumar Vavilapalli <
vinodkv@hortonworks.com> wrote:

>
>
> Can you check the job configuration for these ~100 jobs? Do they have
> keep.failed.task.files set to true? If so, these files won't be deleted. If
> it doesn't, it could be a bug.
>
> Sharing your configs for these jobs will definitely help.
>
> Thanks,
> +Vinod
>
>
> On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>

Re: JobCache directory cleanup

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

Can you check the job configuration for these ~100 jobs? Do they have
keep.failed.task.files set to true? If so, these files won't be deleted. If
it doesn't, it could be a bug.

Sharing your configs for these jobs will definitely help.

Thanks,
+Vinod

On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

On Thu, Jan 10, 2013 at 5:17 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> Hemanth,
> I could see following exception in TaskTracker log:
> https://issues.apache.org/jira/browse/MAPREDUCE-5
> But I'm not sure if it is related to this issue.
>
> > Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> So, If I've no running jobs, jobcache directory should be empty. Is it
> correct?
>
>
That is correct. I just verified it with my Hadoop 1.0.2 version

Thanks
Hemanth


>
>
> On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
>> This directory is used by the TaskTracker (slave) daemons to localize job
>> files when the tasks are run on the slaves.
>>
>> Hence, I don't think this is related to the parameter "
>> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
>> related to the jobtracker process.
>>
>> Now, when a job completes, the directories under the jobCache must get
>> automatically cleaned up. However it doesn't look like this is happening in
>> your case.
>>
>> Could you please look at the logs of the tasktracker machine where this
>> has gotten filled up to see if there are any errors that could give clues ?
>> Also, since this is a CDH release, it could be a problem specific to that
>> - and maybe reaching out on the CDH mailing lists will also help
>>
>> Thanks
>> hemanth
>>
>> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>> I've found that there is following parameter:
>>>
>>> <property>
>>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>>   <value>1000</value>
>>>   <description>The number of retired job status to keep in the cache.
>>>   </description>
>>> </property>
>>>
>>> So, if I got it right it intended to control job cache size by limiting
>>> number of jobs to store cache for.
>>>
>>> Also, I've seen that some hadoop users uses cron approach to cleanup
>>> jobcache:
>>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>>  (
>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>>> )
>>>
>>> Are there other approaches to control jobcache size?
>>> What is more correct way to do it?
>>>
>>> Thanks in advance!
>>>
>>> P.S. We are using CDH 4.1.1.
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>> Deployment Engineer
>>> Grid Dynamics
>>> +7 812 640 38 76
>>> Skype: ivan.tretyakov
>>> www.griddynamics.com
>>> itretyakov@griddynamics.com
>>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

On Thu, Jan 10, 2013 at 5:17 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> Hemanth,
> I could see following exception in TaskTracker log:
> https://issues.apache.org/jira/browse/MAPREDUCE-5
> But I'm not sure if it is related to this issue.
>
> > Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> So, If I've no running jobs, jobcache directory should be empty. Is it
> correct?
>
>
That is correct. I just verified it with my Hadoop 1.0.2 version

Thanks
Hemanth


>
>
> On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
>> This directory is used by the TaskTracker (slave) daemons to localize job
>> files when the tasks are run on the slaves.
>>
>> Hence, I don't think this is related to the parameter "
>> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
>> related to the jobtracker process.
>>
>> Now, when a job completes, the directories under the jobCache must get
>> automatically cleaned up. However it doesn't look like this is happening in
>> your case.
>>
>> Could you please look at the logs of the tasktracker machine where this
>> has gotten filled up to see if there are any errors that could give clues ?
>> Also, since this is a CDH release, it could be a problem specific to that
>> - and maybe reaching out on the CDH mailing lists will also help
>>
>> Thanks
>> hemanth
>>
>> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>> I've found that there is following parameter:
>>>
>>> <property>
>>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>>   <value>1000</value>
>>>   <description>The number of retired job status to keep in the cache.
>>>   </description>
>>> </property>
>>>
>>> So, if I got it right it intended to control job cache size by limiting
>>> number of jobs to store cache for.
>>>
>>> Also, I've seen that some hadoop users uses cron approach to cleanup
>>> jobcache:
>>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>>  (
>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>>> )
>>>
>>> Are there other approaches to control jobcache size?
>>> What is more correct way to do it?
>>>
>>> Thanks in advance!
>>>
>>> P.S. We are using CDH 4.1.1.
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>> Deployment Engineer
>>> Grid Dynamics
>>> +7 812 640 38 76
>>> Skype: ivan.tretyakov
>>> www.griddynamics.com
>>> itretyakov@griddynamics.com
>>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

On Thu, Jan 10, 2013 at 5:17 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> Hemanth,
> I could see following exception in TaskTracker log:
> https://issues.apache.org/jira/browse/MAPREDUCE-5
> But I'm not sure if it is related to this issue.
>
> > Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> So, If I've no running jobs, jobcache directory should be empty. Is it
> correct?
>
>
That is correct. I just verified it with my Hadoop 1.0.2 version

Thanks
Hemanth


>
>
> On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
>> This directory is used by the TaskTracker (slave) daemons to localize job
>> files when the tasks are run on the slaves.
>>
>> Hence, I don't think this is related to the parameter "
>> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
>> related to the jobtracker process.
>>
>> Now, when a job completes, the directories under the jobCache must get
>> automatically cleaned up. However it doesn't look like this is happening in
>> your case.
>>
>> Could you please look at the logs of the tasktracker machine where this
>> has gotten filled up to see if there are any errors that could give clues ?
>> Also, since this is a CDH release, it could be a problem specific to that
>> - and maybe reaching out on the CDH mailing lists will also help
>>
>> Thanks
>> hemanth
>>
>> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>> I've found that there is following parameter:
>>>
>>> <property>
>>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>>   <value>1000</value>
>>>   <description>The number of retired job status to keep in the cache.
>>>   </description>
>>> </property>
>>>
>>> So, if I got it right it intended to control job cache size by limiting
>>> number of jobs to store cache for.
>>>
>>> Also, I've seen that some hadoop users uses cron approach to cleanup
>>> jobcache:
>>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>>  (
>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>>> )
>>>
>>> Are there other approaches to control jobcache size?
>>> What is more correct way to do it?
>>>
>>> Thanks in advance!
>>>
>>> P.S. We are using CDH 4.1.1.
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>> Deployment Engineer
>>> Grid Dynamics
>>> +7 812 640 38 76
>>> Skype: ivan.tretyakov
>>> www.griddynamics.com
>>> itretyakov@griddynamics.com
>>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

On Thu, Jan 10, 2013 at 5:17 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:

> Thanks for replies!
>
> Hemanth,
> I could see following exception in TaskTracker log:
> https://issues.apache.org/jira/browse/MAPREDUCE-5
> But I'm not sure if it is related to this issue.
>
> > Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> So, If I've no running jobs, jobcache directory should be empty. Is it
> correct?
>
>
That is correct. I just verified it with my Hadoop 1.0.2 version

Thanks
Hemanth


>
>
> On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <
> yhemanth@thoughtworks.com> wrote:
>
>> Hi,
>>
>> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
>> This directory is used by the TaskTracker (slave) daemons to localize job
>> files when the tasks are run on the slaves.
>>
>> Hence, I don't think this is related to the parameter "
>> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
>> related to the jobtracker process.
>>
>> Now, when a job completes, the directories under the jobCache must get
>> automatically cleaned up. However it doesn't look like this is happening in
>> your case.
>>
>> Could you please look at the logs of the tasktracker machine where this
>> has gotten filled up to see if there are any errors that could give clues ?
>> Also, since this is a CDH release, it could be a problem specific to that
>> - and maybe reaching out on the CDH mailing lists will also help
>>
>> Thanks
>> hemanth
>>
>> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
>> itretyakov@griddynamics.com> wrote:
>>
>>> Hello!
>>>
>>> I've found that jobcache directory became very large on our cluster,
>>> e.g.:
>>>
>>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>>
>>> And it stores information for about 100 jobs:
>>>
>>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>>> | wc -l
>>>
>>> I've found that there is following parameter:
>>>
>>> <property>
>>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>>   <value>1000</value>
>>>   <description>The number of retired job status to keep in the cache.
>>>   </description>
>>> </property>
>>>
>>> So, if I got it right it intended to control job cache size by limiting
>>> number of jobs to store cache for.
>>>
>>> Also, I've seen that some hadoop users uses cron approach to cleanup
>>> jobcache:
>>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>>  (
>>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>>> )
>>>
>>> Are there other approaches to control jobcache size?
>>> What is more correct way to do it?
>>>
>>> Thanks in advance!
>>>
>>> P.S. We are using CDH 4.1.1.
>>>
>>> --
>>> Best Regards
>>> Ivan Tretyakov
>>>
>>> Deployment Engineer
>>> Grid Dynamics
>>> +7 812 640 38 76
>>> Skype: ivan.tretyakov
>>> www.griddynamics.com
>>> itretyakov@griddynamics.com
>>>
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

Hemanth,
I could see following exception in TaskTracker log:
https://issues.apache.org/jira/browse/MAPREDUCE-5
But I'm not sure if it is related to this issue.

> Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

So, If I've no running jobs, jobcache directory should be empty. Is it
correct?



On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hi,
>
> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
> This directory is used by the TaskTracker (slave) daemons to localize job
> files when the tasks are run on the slaves.
>
> Hence, I don't think this is related to the parameter "
> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
> related to the jobtracker process.
>
> Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> Could you please look at the logs of the tasktracker machine where this
> has gotten filled up to see if there are any errors that could give clues ?
> Also, since this is a CDH release, it could be a problem specific to that
> - and maybe reaching out on the CDH mailing lists will also help
>
> Thanks
> hemanth
>
> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>> I've found that there is following parameter:
>>
>> <property>
>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>   <value>1000</value>
>>   <description>The number of retired job status to keep in the cache.
>>   </description>
>> </property>
>>
>> So, if I got it right it intended to control job cache size by limiting
>> number of jobs to store cache for.
>>
>> Also, I've seen that some hadoop users uses cron approach to cleanup
>> jobcache:
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>  (
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> )
>>
>> Are there other approaches to control jobcache size?
>> What is more correct way to do it?
>>
>> Thanks in advance!
>>
>> P.S. We are using CDH 4.1.1.
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

Hemanth,
I could see following exception in TaskTracker log:
https://issues.apache.org/jira/browse/MAPREDUCE-5
But I'm not sure if it is related to this issue.

> Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

So, If I've no running jobs, jobcache directory should be empty. Is it
correct?



On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hi,
>
> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
> This directory is used by the TaskTracker (slave) daemons to localize job
> files when the tasks are run on the slaves.
>
> Hence, I don't think this is related to the parameter "
> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
> related to the jobtracker process.
>
> Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> Could you please look at the logs of the tasktracker machine where this
> has gotten filled up to see if there are any errors that could give clues ?
> Also, since this is a CDH release, it could be a problem specific to that
> - and maybe reaching out on the CDH mailing lists will also help
>
> Thanks
> hemanth
>
> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>> I've found that there is following parameter:
>>
>> <property>
>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>   <value>1000</value>
>>   <description>The number of retired job status to keep in the cache.
>>   </description>
>> </property>
>>
>> So, if I got it right it intended to control job cache size by limiting
>> number of jobs to store cache for.
>>
>> Also, I've seen that some hadoop users uses cron approach to cleanup
>> jobcache:
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>  (
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> )
>>
>> Are there other approaches to control jobcache size?
>> What is more correct way to do it?
>>
>> Thanks in advance!
>>
>> P.S. We are using CDH 4.1.1.
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

Hemanth,
I could see following exception in TaskTracker log:
https://issues.apache.org/jira/browse/MAPREDUCE-5
But I'm not sure if it is related to this issue.

> Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

So, If I've no running jobs, jobcache directory should be empty. Is it
correct?



On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hi,
>
> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
> This directory is used by the TaskTracker (slave) daemons to localize job
> files when the tasks are run on the slaves.
>
> Hence, I don't think this is related to the parameter "
> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
> related to the jobtracker process.
>
> Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> Could you please look at the logs of the tasktracker machine where this
> has gotten filled up to see if there are any errors that could give clues ?
> Also, since this is a CDH release, it could be a problem specific to that
> - and maybe reaching out on the CDH mailing lists will also help
>
> Thanks
> hemanth
>
> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>> I've found that there is following parameter:
>>
>> <property>
>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>   <value>1000</value>
>>   <description>The number of retired job status to keep in the cache.
>>   </description>
>> </property>
>>
>> So, if I got it right it intended to control job cache size by limiting
>> number of jobs to store cache for.
>>
>> Also, I've seen that some hadoop users uses cron approach to cleanup
>> jobcache:
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>  (
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> )
>>
>> Are there other approaches to control jobcache size?
>> What is more correct way to do it?
>>
>> Thanks in advance!
>>
>> P.S. We are using CDH 4.1.1.
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks for replies!

Hemanth,
I could see following exception in TaskTracker log:
https://issues.apache.org/jira/browse/MAPREDUCE-5
But I'm not sure if it is related to this issue.

> Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

So, If I've no running jobs, jobcache directory should be empty. Is it
correct?



On Thu, Jan 10, 2013 at 8:18 AM, Hemanth Yamijala <yhemanth@thoughtworks.com
> wrote:

> Hi,
>
> The directory name you have provided is /data?/mapred/local/taskTracker/persona/jobcache/.
> This directory is used by the TaskTracker (slave) daemons to localize job
> files when the tasks are run on the slaves.
>
> Hence, I don't think this is related to the parameter "
> mapreduce.jobtracker.retiredjobs.cache.size", which is a parameter
> related to the jobtracker process.
>
> Now, when a job completes, the directories under the jobCache must get
> automatically cleaned up. However it doesn't look like this is happening in
> your case.
>
> Could you please look at the logs of the tasktracker machine where this
> has gotten filled up to see if there are any errors that could give clues ?
> Also, since this is a CDH release, it could be a problem specific to that
> - and maybe reaching out on the CDH mailing lists will also help
>
> Thanks
> hemanth
>
> On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov <
> itretyakov@griddynamics.com> wrote:
>
>> Hello!
>>
>> I've found that jobcache directory became very large on our cluster, e.g.:
>>
>> # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> 465G    /data1/mapred/local/taskTracker/user/jobcache
>> 464G    /data2/mapred/local/taskTracker/user/jobcache
>> 454G    /data3/mapred/local/taskTracker/user/jobcache
>>
>> And it stores information for about 100 jobs:
>>
>> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
>> | wc -l
>>
>> I've found that there is following parameter:
>>
>> <property>
>>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>>   <value>1000</value>
>>   <description>The number of retired job status to keep in the cache.
>>   </description>
>> </property>
>>
>> So, if I got it right it intended to control job cache size by limiting
>> number of jobs to store cache for.
>>
>> Also, I've seen that some hadoop users uses cron approach to cleanup
>> jobcache:
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>>  (
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> )
>>
>> Are there other approaches to control jobcache size?
>> What is more correct way to do it?
>>
>> Thanks in advance!
>>
>> P.S. We are using CDH 4.1.1.
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>>
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

The directory name you have provided is
/data?/mapred/local/taskTracker/persona/jobcache/.
This directory is used by the TaskTracker (slave) daemons to localize job
files when the tasks are run on the slaves.

Hence, I don't think this is related to the parameter
"mapreduce.jobtracker.retiredjobs.cache.size",
which is a parameter related to the jobtracker process.

Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

Could you please look at the logs of the tasktracker machine where this has
gotten filled up to see if there are any errors that could give clues ?
Also, since this is a CDH release, it could be a problem specific to that -
and maybe reaching out on the CDH mailing lists will also help

Thanks
hemanth

On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>
> I've found that there is following parameter:
>
> <property>
>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>   <value>1000</value>
>   <description>The number of retired job status to keep in the cache.
>   </description>
> </property>
>
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
>
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>  (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
>
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
>
> Thanks in advance!
>
> P.S. We are using CDH 4.1.1.
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

The directory name you have provided is
/data?/mapred/local/taskTracker/persona/jobcache/.
This directory is used by the TaskTracker (slave) daemons to localize job
files when the tasks are run on the slaves.

Hence, I don't think this is related to the parameter
"mapreduce.jobtracker.retiredjobs.cache.size",
which is a parameter related to the jobtracker process.

Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

Could you please look at the logs of the tasktracker machine where this has
gotten filled up to see if there are any errors that could give clues ?
Also, since this is a CDH release, it could be a problem specific to that -
and maybe reaching out on the CDH mailing lists will also help

Thanks
hemanth

On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>
> I've found that there is following parameter:
>
> <property>
>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>   <value>1000</value>
>   <description>The number of retired job status to keep in the cache.
>   </description>
> </property>
>
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
>
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>  (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
>
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
>
> Thanks in advance!
>
> P.S. We are using CDH 4.1.1.
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

Can you check the job configuration for these ~100 jobs? Do they have
keep.failed.task.files set to true? If so, these files won't be deleted. If
it doesn't, it could be a bug.

Sharing your configs for these jobs will definitely help.

Thanks,
+Vinod

On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>

Re: JobCache directory cleanup

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

Can you check the job configuration for these ~100 jobs? Do they have
keep.failed.task.files set to true? If so, these files won't be deleted. If
it doesn't, it could be a bug.

Sharing your configs for these jobs will definitely help.

Thanks,
+Vinod

On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>

Re: JobCache directory cleanup

Posted by Robert Molina <rm...@hortonworks.com>.

Hi Ivan,
Regarding the mapreduce.jobtracker.retiredjobs.cache.size property, the
jobtracker keeps information about a number of completed jobs in memory.
There's a threshold for this, which is a single day by default - as well as
a certain number of jobs per user. Once these limits are hit, the job is
moved into the retired job cache.  Both are used for the UI as well as to
answer RPC requests from the client - like getJobStatus, or getCounters.
Once a job goes out of the retired job cache, it's not available via RPC.

Hope that helps.

Regards,
Robert

On Wed, Jan 9, 2013 at 7:22 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Thanks a lot Alexander!
>
> What is mapreduce.jobtracker.retiredjobs.cache.size for?
> Does cron approach safe for hadoop? Is that only way at the moment?
>
>
> On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz <
> wget.null@gmail.com> wrote:
>
>> Hi,
>>
>> Per default (and not configurable) the logs will be persist for 30 days.
>> This will be configurable in future (
>> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>>
>> - Alex
>>
>> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
>> wrote:
>>
>> > Hello!
>> >
>> > I've found that jobcache directory became very large on our cluster,
>> e.g.:
>> >
>> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> > 465G    /data1/mapred/local/taskTracker/user/jobcache
>> > 464G    /data2/mapred/local/taskTracker/user/jobcache
>> > 454G    /data3/mapred/local/taskTracker/user/jobcache
>> >
>> > And it stores information for about 100 jobs:
>> >
>> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>> uniq |
>> > wc -l
>> >
>> > I've found that there is following parameter:
>> >
>> > <property>
>> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>> >  <value>1000</value>
>> >  <description>The number of retired job status to keep in the cache.
>> >  </description>
>> > </property>
>> >
>> > So, if I got it right it intended to control job cache size by limiting
>> > number of jobs to store cache for.
>> >
>> > Also, I've seen that some hadoop users uses cron approach to cleanup
>> > jobcache:
>> >
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>> > (
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> > )
>> >
>> > Are there other approaches to control jobcache size?
>> > What is more correct way to do it?
>> >
>> > Thanks in advance!
>> >
>> > P.S. We are using CDH 4.1.1.
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Robert Molina <rm...@hortonworks.com>.

Hi Ivan,
Regarding the mapreduce.jobtracker.retiredjobs.cache.size property, the
jobtracker keeps information about a number of completed jobs in memory.
There's a threshold for this, which is a single day by default - as well as
a certain number of jobs per user. Once these limits are hit, the job is
moved into the retired job cache.  Both are used for the UI as well as to
answer RPC requests from the client - like getJobStatus, or getCounters.
Once a job goes out of the retired job cache, it's not available via RPC.

Hope that helps.

Regards,
Robert

On Wed, Jan 9, 2013 at 7:22 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Thanks a lot Alexander!
>
> What is mapreduce.jobtracker.retiredjobs.cache.size for?
> Does cron approach safe for hadoop? Is that only way at the moment?
>
>
> On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz <
> wget.null@gmail.com> wrote:
>
>> Hi,
>>
>> Per default (and not configurable) the logs will be persist for 30 days.
>> This will be configurable in future (
>> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>>
>> - Alex
>>
>> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
>> wrote:
>>
>> > Hello!
>> >
>> > I've found that jobcache directory became very large on our cluster,
>> e.g.:
>> >
>> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> > 465G    /data1/mapred/local/taskTracker/user/jobcache
>> > 464G    /data2/mapred/local/taskTracker/user/jobcache
>> > 454G    /data3/mapred/local/taskTracker/user/jobcache
>> >
>> > And it stores information for about 100 jobs:
>> >
>> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>> uniq |
>> > wc -l
>> >
>> > I've found that there is following parameter:
>> >
>> > <property>
>> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>> >  <value>1000</value>
>> >  <description>The number of retired job status to keep in the cache.
>> >  </description>
>> > </property>
>> >
>> > So, if I got it right it intended to control job cache size by limiting
>> > number of jobs to store cache for.
>> >
>> > Also, I've seen that some hadoop users uses cron approach to cleanup
>> > jobcache:
>> >
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>> > (
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> > )
>> >
>> > Are there other approaches to control jobcache size?
>> > What is more correct way to do it?
>> >
>> > Thanks in advance!
>> >
>> > P.S. We are using CDH 4.1.1.
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Robert Molina <rm...@hortonworks.com>.

Hi Ivan,
Regarding the mapreduce.jobtracker.retiredjobs.cache.size property, the
jobtracker keeps information about a number of completed jobs in memory.
There's a threshold for this, which is a single day by default - as well as
a certain number of jobs per user. Once these limits are hit, the job is
moved into the retired job cache.  Both are used for the UI as well as to
answer RPC requests from the client - like getJobStatus, or getCounters.
Once a job goes out of the retired job cache, it's not available via RPC.

Hope that helps.

Regards,
Robert

On Wed, Jan 9, 2013 at 7:22 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Thanks a lot Alexander!
>
> What is mapreduce.jobtracker.retiredjobs.cache.size for?
> Does cron approach safe for hadoop? Is that only way at the moment?
>
>
> On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz <
> wget.null@gmail.com> wrote:
>
>> Hi,
>>
>> Per default (and not configurable) the logs will be persist for 30 days.
>> This will be configurable in future (
>> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>>
>> - Alex
>>
>> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
>> wrote:
>>
>> > Hello!
>> >
>> > I've found that jobcache directory became very large on our cluster,
>> e.g.:
>> >
>> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> > 465G    /data1/mapred/local/taskTracker/user/jobcache
>> > 464G    /data2/mapred/local/taskTracker/user/jobcache
>> > 454G    /data3/mapred/local/taskTracker/user/jobcache
>> >
>> > And it stores information for about 100 jobs:
>> >
>> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>> uniq |
>> > wc -l
>> >
>> > I've found that there is following parameter:
>> >
>> > <property>
>> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>> >  <value>1000</value>
>> >  <description>The number of retired job status to keep in the cache.
>> >  </description>
>> > </property>
>> >
>> > So, if I got it right it intended to control job cache size by limiting
>> > number of jobs to store cache for.
>> >
>> > Also, I've seen that some hadoop users uses cron approach to cleanup
>> > jobcache:
>> >
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>> > (
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> > )
>> >
>> > Are there other approaches to control jobcache size?
>> > What is more correct way to do it?
>> >
>> > Thanks in advance!
>> >
>> > P.S. We are using CDH 4.1.1.
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Robert Molina <rm...@hortonworks.com>.

Hi Ivan,
Regarding the mapreduce.jobtracker.retiredjobs.cache.size property, the
jobtracker keeps information about a number of completed jobs in memory.
There's a threshold for this, which is a single day by default - as well as
a certain number of jobs per user. Once these limits are hit, the job is
moved into the retired job cache.  Both are used for the UI as well as to
answer RPC requests from the client - like getJobStatus, or getCounters.
Once a job goes out of the retired job cache, it's not available via RPC.

Hope that helps.

Regards,
Robert

On Wed, Jan 9, 2013 at 7:22 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Thanks a lot Alexander!
>
> What is mapreduce.jobtracker.retiredjobs.cache.size for?
> Does cron approach safe for hadoop? Is that only way at the moment?
>
>
> On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz <
> wget.null@gmail.com> wrote:
>
>> Hi,
>>
>> Per default (and not configurable) the logs will be persist for 30 days.
>> This will be configurable in future (
>> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>>
>> - Alex
>>
>> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
>> wrote:
>>
>> > Hello!
>> >
>> > I've found that jobcache directory became very large on our cluster,
>> e.g.:
>> >
>> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
>> > 465G    /data1/mapred/local/taskTracker/user/jobcache
>> > 464G    /data2/mapred/local/taskTracker/user/jobcache
>> > 454G    /data3/mapred/local/taskTracker/user/jobcache
>> >
>> > And it stores information for about 100 jobs:
>> >
>> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort |
>> uniq |
>> > wc -l
>> >
>> > I've found that there is following parameter:
>> >
>> > <property>
>> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>> >  <value>1000</value>
>> >  <description>The number of retired job status to keep in the cache.
>> >  </description>
>> > </property>
>> >
>> > So, if I got it right it intended to control job cache size by limiting
>> > number of jobs to store cache for.
>> >
>> > Also, I've seen that some hadoop users uses cron approach to cleanup
>> > jobcache:
>> >
>> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>> > (
>> >
>> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
>> > )
>> >
>> > Are there other approaches to control jobcache size?
>> > What is more correct way to do it?
>> >
>> > Thanks in advance!
>> >
>> > P.S. We are using CDH 4.1.1.
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>> --
>> Alexander Alten-Lorenz
>> http://mapredit.blogspot.com
>> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>>
>>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot Alexander!

What is mapreduce.jobtracker.retiredjobs.cache.size for?
Does cron approach safe for hadoop? Is that only way at the moment?


On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz
<wg...@gmail.com>wrote:

> Hi,
>
> Per default (and not configurable) the logs will be persist for 30 days.
> This will be configurable in future (
> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>
> - Alex
>
> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
> wrote:
>
> > Hello!
> >
> > I've found that jobcache directory became very large on our cluster,
> e.g.:
> >
> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
> > 465G    /data1/mapred/local/taskTracker/user/jobcache
> > 464G    /data2/mapred/local/taskTracker/user/jobcache
> > 454G    /data3/mapred/local/taskTracker/user/jobcache
> >
> > And it stores information for about 100 jobs:
> >
> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
> |
> > wc -l
> >
> > I've found that there is following parameter:
> >
> > <property>
> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
> >  <value>1000</value>
> >  <description>The number of retired job status to keep in the cache.
> >  </description>
> > </property>
> >
> > So, if I got it right it intended to control job cache size by limiting
> > number of jobs to store cache for.
> >
> > Also, I've seen that some hadoop users uses cron approach to cleanup
> > jobcache:
> >
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> > (
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> > )
> >
> > Are there other approaches to control jobcache size?
> > What is more correct way to do it?
> >
> > Thanks in advance!
> >
> > P.S. We are using CDH 4.1.1.
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot Alexander!

What is mapreduce.jobtracker.retiredjobs.cache.size for?
Does cron approach safe for hadoop? Is that only way at the moment?


On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz
<wg...@gmail.com>wrote:

> Hi,
>
> Per default (and not configurable) the logs will be persist for 30 days.
> This will be configurable in future (
> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>
> - Alex
>
> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
> wrote:
>
> > Hello!
> >
> > I've found that jobcache directory became very large on our cluster,
> e.g.:
> >
> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
> > 465G    /data1/mapred/local/taskTracker/user/jobcache
> > 464G    /data2/mapred/local/taskTracker/user/jobcache
> > 454G    /data3/mapred/local/taskTracker/user/jobcache
> >
> > And it stores information for about 100 jobs:
> >
> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
> |
> > wc -l
> >
> > I've found that there is following parameter:
> >
> > <property>
> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
> >  <value>1000</value>
> >  <description>The number of retired job status to keep in the cache.
> >  </description>
> > </property>
> >
> > So, if I got it right it intended to control job cache size by limiting
> > number of jobs to store cache for.
> >
> > Also, I've seen that some hadoop users uses cron approach to cleanup
> > jobcache:
> >
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> > (
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> > )
> >
> > Are there other approaches to control jobcache size?
> > What is more correct way to do it?
> >
> > Thanks in advance!
> >
> > P.S. We are using CDH 4.1.1.
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot Alexander!

What is mapreduce.jobtracker.retiredjobs.cache.size for?
Does cron approach safe for hadoop? Is that only way at the moment?


On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz
<wg...@gmail.com>wrote:

> Hi,
>
> Per default (and not configurable) the logs will be persist for 30 days.
> This will be configurable in future (
> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>
> - Alex
>
> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
> wrote:
>
> > Hello!
> >
> > I've found that jobcache directory became very large on our cluster,
> e.g.:
> >
> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
> > 465G    /data1/mapred/local/taskTracker/user/jobcache
> > 464G    /data2/mapred/local/taskTracker/user/jobcache
> > 454G    /data3/mapred/local/taskTracker/user/jobcache
> >
> > And it stores information for about 100 jobs:
> >
> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
> |
> > wc -l
> >
> > I've found that there is following parameter:
> >
> > <property>
> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
> >  <value>1000</value>
> >  <description>The number of retired job status to keep in the cache.
> >  </description>
> > </property>
> >
> > So, if I got it right it intended to control job cache size by limiting
> > number of jobs to store cache for.
> >
> > Also, I've seen that some hadoop users uses cron approach to cleanup
> > jobcache:
> >
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> > (
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> > )
> >
> > Are there other approaches to control jobcache size?
> > What is more correct way to do it?
> >
> > Thanks in advance!
> >
> > P.S. We are using CDH 4.1.1.
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Ivan Tretyakov <it...@griddynamics.com>.

Thanks a lot Alexander!

What is mapreduce.jobtracker.retiredjobs.cache.size for?
Does cron approach safe for hadoop? Is that only way at the moment?


On Wed, Jan 9, 2013 at 6:50 PM, Alexander Alten-Lorenz
<wg...@gmail.com>wrote:

> Hi,
>
> Per default (and not configurable) the logs will be persist for 30 days.
> This will be configurable in future (
> https://issues.apache.org/jira/browse/MAPREDUCE-4643).
>
> - Alex
>
> On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com>
> wrote:
>
> > Hello!
> >
> > I've found that jobcache directory became very large on our cluster,
> e.g.:
> >
> > # du -sh /data?/mapred/local/taskTracker/user/jobcache
> > 465G    /data1/mapred/local/taskTracker/user/jobcache
> > 464G    /data2/mapred/local/taskTracker/user/jobcache
> > 454G    /data3/mapred/local/taskTracker/user/jobcache
> >
> > And it stores information for about 100 jobs:
> >
> > # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq
> |
> > wc -l
> >
> > I've found that there is following parameter:
> >
> > <property>
> >  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
> >  <value>1000</value>
> >  <description>The number of retired job status to keep in the cache.
> >  </description>
> > </property>
> >
> > So, if I got it right it intended to control job cache size by limiting
> > number of jobs to store cache for.
> >
> > Also, I've seen that some hadoop users uses cron approach to cleanup
> > jobcache:
> >
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> > (
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> > )
> >
> > Are there other approaches to control jobcache size?
> > What is more correct way to do it?
> >
> > Thanks in advance!
> >
> > P.S. We are using CDH 4.1.1.
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
> --
> Alexander Alten-Lorenz
> http://mapredit.blogspot.com
> German Hadoop LinkedIn Group: http://goo.gl/N8pCF
>
>


-- 
Best Regards
Ivan Tretyakov

Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com

Re: JobCache directory cleanup

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.

Hi,

Per default (and not configurable) the logs will be persist for 30 days. This will be configurable in future (https://issues.apache.org/jira/browse/MAPREDUCE-4643).

- Alex

On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com> wrote:

> Hello!
> 
> I've found that jobcache directory became very large on our cluster, e.g.:
> 
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
> 
> And it stores information for about 100 jobs:
> 
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
> 
> I've found that there is following parameter:
> 
> <property>
>  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>  <value>1000</value>
>  <description>The number of retired job status to keep in the cache.
>  </description>
> </property>
> 
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
> 
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
> 
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
> 
> Thanks in advance!
> 
> P.S. We are using CDH 4.1.1.
> 
> -- 
> Best Regards
> Ivan Tretyakov
> 
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: JobCache directory cleanup

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.

Hi,

Per default (and not configurable) the logs will be persist for 30 days. This will be configurable in future (https://issues.apache.org/jira/browse/MAPREDUCE-4643).

- Alex

On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com> wrote:

> Hello!
> 
> I've found that jobcache directory became very large on our cluster, e.g.:
> 
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
> 
> And it stores information for about 100 jobs:
> 
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
> 
> I've found that there is following parameter:
> 
> <property>
>  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>  <value>1000</value>
>  <description>The number of retired job status to keep in the cache.
>  </description>
> </property>
> 
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
> 
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
> 
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
> 
> Thanks in advance!
> 
> P.S. We are using CDH 4.1.1.
> 
> -- 
> Best Regards
> Ivan Tretyakov
> 
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: JobCache directory cleanup

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.

Hi,

Per default (and not configurable) the logs will be persist for 30 days. This will be configurable in future (https://issues.apache.org/jira/browse/MAPREDUCE-4643).

- Alex

On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com> wrote:

> Hello!
> 
> I've found that jobcache directory became very large on our cluster, e.g.:
> 
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
> 
> And it stores information for about 100 jobs:
> 
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
> 
> I've found that there is following parameter:
> 
> <property>
>  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>  <value>1000</value>
>  <description>The number of retired job status to keep in the cache.
>  </description>
> </property>
> 
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
> 
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
> 
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
> 
> Thanks in advance!
> 
> P.S. We are using CDH 4.1.1.
> 
> -- 
> Best Regards
> Ivan Tretyakov
> 
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: JobCache directory cleanup

Posted by Alexander Alten-Lorenz <wg...@gmail.com>.

Hi,

Per default (and not configurable) the logs will be persist for 30 days. This will be configurable in future (https://issues.apache.org/jira/browse/MAPREDUCE-4643).

- Alex

On Jan 9, 2013, at 3:41 PM, Ivan Tretyakov <it...@griddynamics.com> wrote:

> Hello!
> 
> I've found that jobcache directory became very large on our cluster, e.g.:
> 
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
> 
> And it stores information for about 100 jobs:
> 
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
> 
> I've found that there is following parameter:
> 
> <property>
>  <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>  <value>1000</value>
>  <description>The number of retired job status to keep in the cache.
>  </description>
> </property>
> 
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
> 
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
> (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
> 
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
> 
> Thanks in advance!
> 
> P.S. We are using CDH 4.1.1.
> 
> -- 
> Best Regards
> Ivan Tretyakov
> 
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

The directory name you have provided is
/data?/mapred/local/taskTracker/persona/jobcache/.
This directory is used by the TaskTracker (slave) daemons to localize job
files when the tasks are run on the slaves.

Hence, I don't think this is related to the parameter
"mapreduce.jobtracker.retiredjobs.cache.size",
which is a parameter related to the jobtracker process.

Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

Could you please look at the logs of the tasktracker machine where this has
gotten filled up to see if there are any errors that could give clues ?
Also, since this is a CDH release, it could be a problem specific to that -
and maybe reaching out on the CDH mailing lists will also help

Thanks
hemanth

On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>
> I've found that there is following parameter:
>
> <property>
>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>   <value>1000</value>
>   <description>The number of retired job status to keep in the cache.
>   </description>
> </property>
>
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
>
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>  (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
>
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
>
> Thanks in advance!
>
> P.S. We are using CDH 4.1.1.
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Hemanth Yamijala <yh...@thoughtworks.com>.

Hi,

The directory name you have provided is
/data?/mapred/local/taskTracker/persona/jobcache/.
This directory is used by the TaskTracker (slave) daemons to localize job
files when the tasks are run on the slaves.

Hence, I don't think this is related to the parameter
"mapreduce.jobtracker.retiredjobs.cache.size",
which is a parameter related to the jobtracker process.

Now, when a job completes, the directories under the jobCache must get
automatically cleaned up. However it doesn't look like this is happening in
your case.

Could you please look at the logs of the tasktracker machine where this has
gotten filled up to see if there are any errors that could give clues ?
Also, since this is a CDH release, it could be a problem specific to that -
and maybe reaching out on the CDH mailing lists will also help

Thanks
hemanth

On Wed, Jan 9, 2013 at 8:11 PM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>
> I've found that there is following parameter:
>
> <property>
>   <name>mapreduce.jobtracker.retiredjobs.cache.size</name>
>   <value>1000</value>
>   <description>The number of retired job status to keep in the cache.
>   </description>
> </property>
>
> So, if I got it right it intended to control job cache size by limiting
> number of jobs to store cache for.
>
> Also, I've seen that some hadoop users uses cron approach to cleanup
> jobcache:
> http://grokbase.com/t/hadoop/common-user/102ax9bze1/cleaning-jobcache-manually
>  (
> http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201002.mbox/%3C99484d561002100143s4404df98qead8f2cf687a76d0@mail.gmail.com%3E
> )
>
> Are there other approaches to control jobcache size?
> What is more correct way to do it?
>
> Thanks in advance!
>
> P.S. We are using CDH 4.1.1.
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>

Re: JobCache directory cleanup

Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.

Can you check the job configuration for these ~100 jobs? Do they have
keep.failed.task.files set to true? If so, these files won't be deleted. If
it doesn't, it could be a bug.

Sharing your configs for these jobs will definitely help.

Thanks,
+Vinod

On Wed, Jan 9, 2013 at 6:41 AM, Ivan Tretyakov
<it...@griddynamics.com>wrote:

> Hello!
>
> I've found that jobcache directory became very large on our cluster, e.g.:
>
> # du -sh /data?/mapred/local/taskTracker/user/jobcache
> 465G    /data1/mapred/local/taskTracker/user/jobcache
> 464G    /data2/mapred/local/taskTracker/user/jobcache
> 454G    /data3/mapred/local/taskTracker/user/jobcache
>
> And it stores information for about 100 jobs:
>
> # ls -1 /data?/mapred/local/taskTracker/persona/jobcache/  | sort | uniq |
> wc -l
>