You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Alejandro Abdelnur (JIRA)" <ji...@apache.org> on 2007/04/20 09:11:15 UTC
[jira] Created: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
list of completed jobs purges jobs based on submission not on completion age
----------------------------------------------------------------------------
Key: HADOOP-1279
URL: https://issues.apache.org/jira/browse/HADOOP-1279
Project: Hadoop
Issue Type: Bug
Components: mapred
Environment: all
Reporter: Alejandro Abdelnur
The JobTracker keeps at most the last 100 completed jobs.
One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12492246 ]
Hadoop QA commented on HADOOP-1279:
-----------------------------------
Integrated in Hadoop-Nightly #71 (See http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Nightly/71/)
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Fix For: 0.13.0
>
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned HADOOP-1279:
-------------------------------------
Assignee: Arun C Murthy
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491742 ]
Arun C Murthy commented on HADOOP-1279:
---------------------------------------
Arkady, all information is persisted already into 'JobTracker History' (link at bottom of JT's web-ui) regardless of whether the in-memory data-structures are purged or not... is there any enhancement you'd like to see there?
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Fix For: 0.13.0
>
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Alejandro Abdelnur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491544 ]
Alejandro Abdelnur commented on HADOOP-1279:
--------------------------------------------
+1
tested the patch and it fixes the issue.
thxs.
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1279:
----------------------------------
Fix Version/s: 0.13.0
Status: Patch Available (was: Open)
Thanks to Alejandro for helping test this...
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Fix For: 0.13.0
>
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1279:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Arun!
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Fix For: 0.13.0
>
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491743 ]
Hadoop QA commented on HADOOP-1279:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12355916/HADOOP-1279_20070420_1.patch applied and successfully tested against trunk revision r532083.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/79/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/79/console
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Fix For: 0.13.0
>
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "arkady borkovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491739 ]
arkady borkovsky commented on HADOOP-1279:
------------------------------------------
It would be very nice if the information about completed jobs is persitent and is NEVER lost.
Once it purged from JobTracker RAM, it should be moved into some persistent storage and still be retrievable.
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Fix For: 0.13.0
>
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1279) list of completed jobs purges jobs
based on submission not on completion age
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1279:
----------------------------------
Attachment: HADOOP-1279_20070420_1.patch
Alejandro - here is a patch which adds only 'completed' jobs to the queue... could you give it a whirl and let me know? Thanks!
> list of completed jobs purges jobs based on submission not on completion age
> ----------------------------------------------------------------------------
>
> Key: HADOOP-1279
> URL: https://issues.apache.org/jira/browse/HADOOP-1279
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Environment: all
> Reporter: Alejandro Abdelnur
> Assigned To: Arun C Murthy
> Attachments: HADOOP-1279_20070420_1.patch
>
>
> The JobTracker keeps at most the last 100 completed jobs.
> One this limit is exceeded and there is a new completed job the old submitted job in completed state is removed regardless if there are jobs that have finished much earlier in the completed list.
> The job should be added at completion time, not a submission time to avoid this problem.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.