You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Thomas Friol (JIRA)" <ji...@apache.org> on 2006/10/27 13:32:18 UTC
[jira] Created: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Jobs without any map and reduce operations seems to be lost after their execution
---------------------------------------------------------------------------------
Key: HADOOP-649
URL: http://issues.apache.org/jira/browse/HADOOP-649
Project: Hadoop
Issue Type: Bug
Affects Versions: 0.6.1
Reporter: Thomas Friol
When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472662 ]
Owen O'Malley commented on HADOOP-649:
--------------------------------------
When a patch is ready to be looked at you need to push the "submit patch" button to make it's state "patch available." We use the the patch available state to find the patches to look at. This looks like a good change assuming it still applies to head.
+1
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Sylvain Wallez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472577 ]
Sylvain Wallez commented on HADOOP-649:
---------------------------------------
Can someone have a look at this one-liner? Thanks
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472744 ]
Hadoop QA commented on HADOOP-649:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12346228/HADOOP-649.patch applied and successfully tested against trunk revision r506778.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-649?page=comments#action_12456335 ]
Thomas Friol commented on HADOOP-649:
-------------------------------------
Has somenone looked to this patch ?
I think it is quiet important to fix this issue since some NPEs are regularly thrown by the JobInProgress class.
This patch has been tested on the 0.9.0 version and works perfectly.
Thanks for attention.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: http://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-649:
--------------------------------
Resolution: Fixed
Fix Version/s: 0.12.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Thomas!
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Fix For: 0.12.0
>
> Attachments: HADOOP-649-0.11.2.patch, HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472675 ]
Thomas Friol commented on HADOOP-649:
-------------------------------------
Sorry. Didn't know about this practice. Will take care of that next time. ;-)
Thanks for your attention.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-649?page=all ]
Thomas Friol updated HADOOP-649:
--------------------------------
Attachment: HADOOP-649.patch
The previous patch is not really good. In fact, the finished time must be setted before the JobStatus in order not to be retire by the job tracker.
Here is new and valid patch. :-)
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: http://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472831 ]
Doug Cutting commented on HADOOP-649:
-------------------------------------
> It would be best to have a unit test
Probably this can be done by simply adding a new test or slightly modifying an existing unit test, rather than constructing an entirely new test for this bug.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-649?page=comments#action_12454867 ]
Thomas Friol commented on HADOOP-649:
-------------------------------------
If my patch is correct and commited (I have tested it), I think that HADOOP-648 issue can also be resolved in the same time as this one because in fact, it is the same issue.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: http://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Attachments: HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472828 ]
Doug Cutting commented on HADOOP-649:
-------------------------------------
This looks good to me. It would be best to have a unit test, to make sure that it really fixes the issue and that we don't break it again in the future. Thomas, are you willing to write one?
Also, please don't include the bug number in the patch itself. We don't want the code to get cluttered with comments documenting its history: we leave history to subversion.
Thanks!
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Friol updated HADOOP-649:
--------------------------------
Attachment: HADOOP-649-0.11.2.patch
Here is the new patch including the unit test.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649-0.11.2.patch, HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Friol updated HADOOP-649:
--------------------------------
Status: Patch Available (was: Open)
Patch available ;-)
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473152 ]
Hadoop QA commented on HADOOP-649:
----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12351118/HADOOP-649-0.11.2.patch applied and successfully tested against trunk revision r507276.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649-0.11.2.patch, HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-649?page=all ]
Thomas Friol updated HADOOP-649:
--------------------------------
Attachment: HADOOP-649.patch
After a long investigation, i finally found why these kind of jobs where retired by the jobtracker. This is because in this case, job finished time is never setted.
So here is a patch to correct this problem.
It could be great if my patch is correct to integrate it in the next 0.9.0 release if possible.
Thanks.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: http://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Attachments: HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-649) Jobs without any map and reduce
operations seems to be lost after their execution
Posted by "Thomas Friol (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473009 ]
Thomas Friol commented on HADOOP-649:
-------------------------------------
I will add a unit test and remove my comment from the previous patch.
> Jobs without any map and reduce operations seems to be lost after their execution
> ---------------------------------------------------------------------------------
>
> Key: HADOOP-649
> URL: https://issues.apache.org/jira/browse/HADOOP-649
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.6.1
> Reporter: Thomas Friol
> Assigned To: Owen O'Malley
> Attachments: HADOOP-649.patch, HADOOP-649.patch
>
>
> When a job does not provide any file splits (because there is no data present), the job successes but then it is lost by the job tracker. Is this normal ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.