You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Karthik Kambatla (JIRA)" <ji...@apache.org> on 2012/09/07 01:43:07 UTC
[jira] [Created] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Karthik Kambatla created MAPREDUCE-4643:
-------------------------------------------
Summary: Make job-history cleanup-period configurable
Key: MAPREDUCE-4643
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.0.0-alpha, 1.0.3
Reporter: Karthik Kambatla
Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462220#comment-13462220 ]
Karthik Kambatla commented on MAPREDUCE-4643:
---------------------------------------------
Thanks for your patch, Sandy.
The patch looks like it should work for (max-age > 1 day). How do we handle significantly smaller ages? We should probably address it the same it is addressed in trunk - use the modification time? Thoughts?
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandy Ryza updated MAPREDUCE-4643:
----------------------------------
Attachment: MAPREDUCE-4643-branch1.patch
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471913#comment-13471913 ]
Alejandro Abdelnur commented on MAPREDUCE-4643:
-----------------------------------------------
using ms as unit for a value that will normally be in days seems a bit off. I understand that for testing purposes we want something short enough, can we use seconds instead? (saving 3 digits). Ideally we should be able to specify units on the time interval (ie '2ms', '2d'), but this is completely out of scope of this JIRA. Thx
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471926#comment-13471926 ]
Sandy Ryza commented on MAPREDUCE-4643:
---------------------------------------
trunk already uses milliseconds. Should branch-1 be different?
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462241#comment-13462241 ]
Sandy Ryza commented on MAPREDUCE-4643:
---------------------------------------
It should work for significantly smaller ages as well. It uses the directory names to decide which to look in, but checks the modification times on the files inside them.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463023#comment-13463023 ]
Hadoop QA commented on MAPREDUCE-4643:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12546557/MAPREDUCE-4643-branch1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2877//console
This message is automatically generated.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandy Ryza updated MAPREDUCE-4643:
----------------------------------
Assignee: Sandy Ryza
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3, 2.0.0-alpha
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471934#comment-13471934 ]
Alejandro Abdelnur commented on MAPREDUCE-4643:
-----------------------------------------------
oh well. looks good, one small NIT though
{code}
+ return result.getTimeInMillis() -
+ result.getTimeInMillis() % HistoryCleaner.ONE_DAY_IN_MS;
{code}
Assign getTimeInMillis() to a local var and then use that for the expression above.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13463022#comment-13463022 ]
Sandy Ryza commented on MAPREDUCE-4643:
---------------------------------------
Uploaded a patch that fixes the comments and includes the test
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandy Ryza updated MAPREDUCE-4643:
----------------------------------
Affects Version/s: (was: 2.0.0-alpha)
Status: Patch Available (was: Open)
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471966#comment-13471966 ]
Sandy Ryza commented on MAPREDUCE-4643:
---------------------------------------
I uploaded a patch that makes that revision.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandy Ryza updated MAPREDUCE-4643:
----------------------------------
Attachment: MAPREDUCE-4643-branch1-2.patch
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461585#comment-13461585 ]
Sandy Ryza commented on MAPREDUCE-4643:
---------------------------------------
Ah, didn't see this JIRA until now. The patch I'm working on for MAPREDUCE-4676 leaves the directory structure the same and deletes based on the modification time (only checking files in old enough directories). For the testJobHistoryCleaner test, it would be ideal for it to be configurable to milliseconds. What do you think?
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3, 2.0.0-alpha
> Reporter: Karthik Kambatla
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandy Ryza updated MAPREDUCE-4643:
----------------------------------
Attachment: MAPREDUCE-4643-branch1.patch
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462215#comment-13462215 ]
Hadoop QA commented on MAPREDUCE-4643:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12546397/MAPREDUCE-4643.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2874//console
This message is automatically generated.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462285#comment-13462285 ]
Sandy Ryza commented on MAPREDUCE-4643:
---------------------------------------
I spoke with Karthik and we resolved that the patch does work for smaller ages, but that for directories older than a day, it could work more efficiently by deleting the entire directory.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462408#comment-13462408 ]
Hadoop QA commented on MAPREDUCE-4643:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12546425/MAPREDUCE-4643-branch1.patch
against trunk revision .
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2875//console
This message is automatically generated.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450412#comment-13450412 ]
Karthik Kambatla commented on MAPREDUCE-4643:
---------------------------------------------
The two aspects of making the job-history cleanup period configurable seem to be:
# unit of time to configure it on (minutes vs days)
# directory naming to reflect the creation time
For the directory naming, we should be able to leave it as yyyy/mm/dd format. If the cleanup period is in hours/minutes, we can read the file modification time and use that to do the comparison. Trunk currently uses the modification time even for higher values of cleanup period (days) and has a TODO note that we should use the directory structure instead to reduce the load on HDFS.
With regards to the unit of time for configuration, minutes seems to be the better option.
Thoughts?
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3, 2.0.0-alpha
> Reporter: Karthik Kambatla
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13471968#comment-13471968 ]
Hadoop QA commented on MAPREDUCE-4643:
--------------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12548332/MAPREDUCE-4643-branch1-2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2918//console
This message is automatically generated.
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1-2.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Sandy Ryza (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sandy Ryza updated MAPREDUCE-4643:
----------------------------------
Attachment: MAPREDUCE-4643.patch
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4643) Make job-history cleanup-period
configurable
Posted by "Karthik Kambatla (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462346#comment-13462346 ]
Karthik Kambatla commented on MAPREDUCE-4643:
---------------------------------------------
Thanks for the patch, Sandy. It looks like it should work.
Minor comments:
# In {{JobHistory.directoryTime()}}, the initial comments do not apply anymore and can be removed
# Do you want to add a test to verify the timely deletion of history files? (Now, I see that this test would fix MAPREDUCE-4676 as well). Do you think you can add a test here and mark the other one duplicate?
> Make job-history cleanup-period configurable
> --------------------------------------------
>
> Key: MAPREDUCE-4643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4643
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 1.0.3
> Reporter: Karthik Kambatla
> Assignee: Sandy Ryza
> Attachments: MAPREDUCE-4643-branch1.patch, MAPREDUCE-4643.patch
>
>
> Job history cleanup should be made configurable. Currently, it is set to 1 month by default. The DEBUG_MODE (to be removed, see MAPREDUCE-4629) sets it to 20 minutes, but it should be configurable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira