You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ravi Prakash (Created) (JIRA)" <ji...@apache.org> on 2011/11/29 23:05:39 UTC
[jira] [Created] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Job end notification method should be called before stop() in handle(JobFinishEvent)
------------------------------------------------------------------------------------
Key: MAPREDUCE-3484
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Ravi Prakash
We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3484:
-----------------------------------------------
Description:
We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
was:We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
Hadoop Flags: Reviewed
Summary: JobEndNotifier is getting interrupted before completing all its retries. (was: Job end notification method should be called before stop() in handle(JobFinishEvent))
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Ravi Prakash (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-3484:
------------------------------------
Attachment: MAPREDUCE-3484.branch-0.23.patch
Last patch was wrong. These are the changes. I need to add tests still
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169773#comment-13169773 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3484:
----------------------------------------------------
bq. Does anyone know if I'm seeing the InterruptedException because the NM is trying to reclaim resources?
The interruption is because the dispatcher itself is going to be exited as part of stop.
+1 for moving the notification before stop().
Looking at the patch now.
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169838#comment-13169838 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3484:
----------------------------------------------------
{quote}
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.TestRMNodeTransitions
{quote}
Totally missed this, will investigate and fix it separately. Not related to this patch, though.
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169859#comment-13169859 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1458 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1458/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214563
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169852#comment-13169852 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1507 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1507/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214563
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169858#comment-13169858 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #302 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/302/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
svn merge -c 1214563 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214566
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Ravi Prakash (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-3484:
------------------------------------
Attachment: MAPREDUCE-3484.branch-0.23.patch
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169857#comment-13169857 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #280 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/280/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
svn merge -c 1214563 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214566
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169846#comment-13169846 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #290 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/290/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
svn merge -c 1214563 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214566
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169918#comment-13169918 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #126 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/126/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
svn merge -c 1214563 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214566
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169837#comment-13169837 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1434 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1434/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214563
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169819#comment-13169819 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3484:
----------------------------------------------------
Patch looks good to me. +1.
I was concerned about use cases like oozie which act on the notification url and request JobCounters on notification, but I think we are fine because
- if the notification is received by oozie and the AM is still alive, AM will directly respond to the request for counters
- if the notification is received but the AM is gone by then, it'd mean that history is properly written and JobHistoryServer can respond to the request.
There is a small corner case when AM sends the notification but crashes before completely writing the JobHistory. In this case, the client/oozie will see the job as successful via RPC/UI, but cannot get any information. This is an orthogonal bug, I'll create a separate ticket.
At any rate, I think we are missing an integration test for job-end notification feature. We need to modify/port {{NotificationTestCase}} to work with {{MiniMRYarnCluster}}. Will open a ticket for that too.
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170286#comment-13170286 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #928 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/928/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214563
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169825#comment-13169825 ]
Hadoop QA commented on MAPREDUCE-3484:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12507449/MAPREDUCE-3484.branch-0.23.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.TestRMNodeTransitions
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1454//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1454//console
This message is automatically generated.
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Ravi Prakash (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-3484:
------------------------------------
Status: Patch Available (was: Open)
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170220#comment-13170220 ]
Hudson commented on MAPREDUCE-3484:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #108 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/108/])
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing all its retries. Contributed by Ravi Prakash.
svn merge -c 1214563 --ignore-ancestry ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214566
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestJobEndNotifier.java
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3484) JobEndNotifier is getting
interrupted before completing all its retries.
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3484:
-----------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.1
Release Note: Fixed JobEndNotifier to not get interrupted before completing all its retries.
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Ravi Prakash!
> JobEndNotifier is getting interrupted before completing all its retries.
> ------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
> To fix this, Job end notification method should be called before stop() in handle(JobFinishEvent).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159574#comment-13159574 ]
Ravi Prakash commented on MAPREDUCE-3484:
-----------------------------------------
This is the exception we noticed. Although 2 more retries were left, the InterruptedException caused the notification to not happen.
{noformat}
2011-11-29 20:13:09,259 WARN [AsyncDispatcher event handler] org.mortbay.log: Job end notification to <someURL> failed
java.net.UnknownServiceException: no content-type
at java.net.URLConnection.getContentHandler(URLConnection.java:1192)
at java.net.URLConnection.getContent(URLConnection.java:689)
at org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notifyURLOnce(JobEndNotifier.java:95)
at org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:139)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:397)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:370)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:116)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:619)
2011-11-29 20:13:09,259 WARN [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Job end notification interrupted for jobID : <some jobID>
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.mapreduce.v2.app.JobEndNotifier.notify(JobEndNotifier.java:141)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:397)
at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:370)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:116)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:619)
{noformat}
Does anyone know if I'm seeing the InterruptedException because the NM is trying to reclaim resources? i.e. did it try killing the AM process?
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3484) Job end notification method
should be called before stop() in handle(JobFinishEvent)
Posted by "Ravi Prakash (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ravi Prakash updated MAPREDUCE-3484:
------------------------------------
Attachment: MAPREDUCE-3484.branch-0.23.patch
Could someone please review and commit this patch?
> Job end notification method should be called before stop() in handle(JobFinishEvent)
> ------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3484
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3484
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
> Affects Versions: 0.23.0
> Reporter: Ravi Prakash
> Assignee: Ravi Prakash
> Attachments: MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch, MAPREDUCE-3484.branch-0.23.patch
>
>
> We noticed JobEndNotifier was getting an InterruptedException before completing all its retries.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira