You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Feng Peng (JIRA)" <ji...@apache.org> on 2012/06/29 20:49:42 UTC
[jira] [Created] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Feng Peng created PIG-2780:
------------------------------
Summary: MapReduceLauncher should break early when one of the jobs throws an exception
Key: PIG-2780
URL: https://issues.apache.org/jira/browse/PIG-2780
Project: Pig
Issue Type: Bug
Reporter: Feng Peng
Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
{noformat}
jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
...
jcThread.start();
// Now wait, till we are finished.
while(!jc.allFinished()){
...
}
//check for the jobControlException first
//if the job controller fails before launching the jobs then there are
//no jobs to check for failure
if (jobControlException != null) {
...
}
{noformat}
There are two problems with this approach:
1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Feng Peng (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404291#comment-13404291 ]
Feng Peng commented on PIG-2780:
--------------------------------
We have an example where Pig tries to submit 4 jobs simultaneously but only reports the exception after all other three jobs are finished (after 6 hours). Neither problem would cause errors in the results, but fixing them would help debugging errors in large complex jobs.
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li updated PIG-2780:
------------------------
Attachment: PIG-2780.1.patch
Update the patch to log a early warning for the failure when stop_on_failure is not enabled.
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2780.0.patch, PIG-2780.1.patch
>
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2780:
----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Patch committed to trunk. Thanks Jie!
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2780.0.patch, PIG-2780.1.patch
>
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li updated PIG-2780:
------------------------
Status: Patch Available (was: Open)
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2780.0.patch
>
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li updated PIG-2780:
------------------------
Attachment: PIG-2780.0.patch
Attached a patch that checks frequently whether any job has failed, and if so stops immediately.
Also includes a unit test where Pig submits three jobs at the same time and one of them will fail. With this patch, Pig will stop without finishing the other two jobs.
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
> Assignee: Jie Li
> Fix For: 0.11
>
> Attachments: PIG-2780.0.patch
>
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404213#comment-13404213 ]
Jie Li commented on PIG-2780:
-----------------------------
The jobControlException is different from the job exception. Pig submit jobs in batches and check exceptions at the end of each batch. Jobs without inter-dependency can run in parallel in the same batch, otherwise they are in different batches. Usually the number of jobs in one batch is very small, so checking exception at the end looks acceptable.
If you specify "-stop_on_failure" in the command line option, then Pig will stop after the failed batch.
If it doesn't satisfy your need, i.e. you also want Pig to stop on failure within batch, then please keep this jira open.
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2780:
----------------------------
Fix Version/s: 0.11
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
> Fix For: 0.11
>
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (PIG-2780) MapReduceLauncher should break early
when one of the jobs throws an exception
Posted by "Jie Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jie Li reassigned PIG-2780:
---------------------------
Assignee: Jie Li
> MapReduceLauncher should break early when one of the jobs throws an exception
> -----------------------------------------------------------------------------
>
> Key: PIG-2780
> URL: https://issues.apache.org/jira/browse/PIG-2780
> Project: Pig
> Issue Type: Bug
> Reporter: Feng Peng
> Assignee: Jie Li
> Fix For: 0.11
>
>
> Right now MapReduceLauncher caches the job exception in jobControlException and only processes it when all the jobs are done:
> {noformat}
> jcThread.setUncaughtExceptionHandler(jctExceptionHandler);
> ...
> jcThread.start();
> // Now wait, till we are finished.
> while(!jc.allFinished()){
> ...
> }
> //check for the jobControlException first
> //if the job controller fails before launching the jobs then there are
> //no jobs to check for failure
> if (jobControlException != null) {
> ...
> }
> {noformat}
> There are two problems with this approach:
> 1. There is only one jobControlException variable. If two jobs are throwing exceptions, the first one will be lost.
> 2. If there are multiple jobs, the exceptions will not be reported until other jobs are finished, which is a waste of system resource.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira