You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (Created) (JIRA)" <ji...@apache.org> on 2011/10/21 16:38:32 UTC
[jira] [Created] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
NM should send a SIGKILL for completed containers also
------------------------------------------------------
Key: MAPREDUCE-3240
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: MR-3240.2.patch
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Status: Patch Available (was: Open)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah reassigned MAPREDUCE-3240:
--------------------------------------
Assignee: Hitesh Shah
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137257#comment-13137257 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #72 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/72/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
Added the new files which I missed earlier.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189815
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137229#comment-13137229 ]
Robert Joseph Evans commented on MAPREDUCE-3240:
------------------------------------------------
Seeing errors in
{noformat}
hadoop-yarn-applications-distributedshell ......... FAILURE
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile (default-compile) on project hadoop-yarn-applications-distributedshell: Fatal error compiling: Error while executing the compiler. InvocationTargetException: PermGen space -> [Help 1]
{noformat}
Which is different from before so it is likely not anything to do with you. Also the error went away the second time I built, so I am not sure what is happening there.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137221#comment-13137221 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3240:
----------------------------------------------------
Just committed the new files which are still lying around in my local workspace.
I just pushed the new files in. Can you try again? Thanks Robert for pointing out.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3240:
-----------------------------------------------
Status: Open (was: Patch Available)
Looks good overall. We don't need the ShellCommandExecutor objects in ContainerExecutor anymore. Will remove that code and attach a patch to avoid one more review cycle.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: MR-3240.wip.patch
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Attachments: MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137212#comment-13137212 ]
Robert Joseph Evans commented on MAPREDUCE-3240:
------------------------------------------------
It looks like this broke 0.23
{noformat}
$git log branch-0.23
commit 3589a79d2057acbdc348ca222756c84c62289f5c
Author: Vinod Kumar Vavilapalli <vi...@apache.org>
Date: Thu Oct 27 12:44:57 2011 +0000
MAPREDUCE-2986. Fixed MiniYARNCluster to support multiple NodeManagers. Contributed by Anupam Seth.
svn merge -c r1189721 --ignore-ancestry ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23@1189723 13f79535-47bb-0310-9956-ffa450edef68
commit cd52508b445ee5c1d00570b5cd9b03f45dd48058
Author: Vinod Kumar Vavilapalli <vi...@apache.org>
Date: Thu Oct 27 12:06:08 2011 +0000
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23@1189713 13f79535-47bb-0310-9956-ffa450edef68
commit 7c01afd599dee540f4d76243a6f40c43edaf1209
Author: Arun Murthy <ac...@apache.org>
Date: Thu Oct 27 08:32:18 2011 +0000
Merge -c 1189670 from trunk to branch-0.23 to fix MAPREDUCE-3279.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.23@1189672 13f79535-47bb-0310-9956-ffa450edef68
$git checkout 7c01afd599dee540f4d76243a6f40c43edaf1209
$mvn clean install -DskipTests
#passed
$git checkout cd52508b445ee5c1d00570b5cd9b03f45dd48058
$mvn clean install -DskipTests
#failed
/home/evans/src/hadoop-git/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java:[39,53] package org.apache.hadoop.yarn.server.nodemanager.util does not exist
{noformat}
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140145#comment-13140145 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #76 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/76/])
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195257
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/Makefile.am
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140110#comment-13140110 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #56 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/56/])
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195257
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/Makefile.am
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138296#comment-13138296 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #53 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/53/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
Added the new files which I missed earlier.
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189815
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189713
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3240:
-------------------------------------
Priority: Blocker (was: Major)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Status: Patch Available (was: Open)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136988#comment-13136988 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1244 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1244/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189711
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: MR-3240.4.wip.patch
Adding more debug statements to pinpoint failure in build.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137218#comment-13137218 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3240:
----------------------------------------------------
Oops, my mistake. Missed adding new files. Am on it.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3240:
-----------------------------------------------
Attachment: MAPREDUCE-3240-20111027.2.txt
Fixing javadoc errors and test issues.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136975#comment-13136975 ]
Hadoop QA commented on MAPREDUCE-3240:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501077/MAPREDUCE-3240-20111027.2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 28 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 170 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-web-proxy.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1171//console
This message is automatically generated.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135703#comment-13135703 ]
Hadoop QA commented on MAPREDUCE-3240:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12500815/MR-3240.3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 22 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 166 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1147//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1147//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1147//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1147//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1147//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1147//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1147//console
This message is automatically generated.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3240:
-----------------------------------------------
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137879#comment-13137879 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #65 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/65/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
Added the new files which I missed earlier.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189815
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137069#comment-13137069 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #873 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/873/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189711
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136718#comment-13136718 ]
Mahadev konar commented on MAPREDUCE-3240:
------------------------------------------
@Hitesh,
Its happening to my patches as well. Looks like something wrong with some of the build machines. Ill ping Giri to see if we can get this fixed.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136711#comment-13136711 ]
Hadoop QA commented on MAPREDUCE-3240:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501001/MR-3240.5.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 170 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-web-proxy.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1159//console
This message is automatically generated.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137228#comment-13137228 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #76 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/76/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
Added the new files which I missed earlier.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189815
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138377#comment-13138377 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #846 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/846/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189711
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136741#comment-13136741 ]
Arun C Murthy commented on MAPREDUCE-3240:
------------------------------------------
Looks like it's due to newer build machines...
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: MR-3240.1.patch
Worked around things slightly now to have setsid be invoked by the executors and not from generated task.sh to address permission issues.
Ran all tests locally on mac/linux dev box. Untested on secure cluster as of now.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Attachments: MR-3240.1.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Status: Open (was: Patch Available)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3240:
-----------------------------------------------
Attachment: MAPREDUCE-3240-20111027.1.txt
Attaching updated patch. This does the following in addition:
- Moved ProcessFileReader into NodeManager module.
- Removed the ShellCommandExecutor objects from ContainerExecutor as we don't need them anymore.
- Moved the starting of SIGKILL thread to after the finish of the SIGTERM processing.
Tested this on a single node secure setup. Verified that the pid generated by LCE matches to that of the running container.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3240:
-------------------------------------
Status: Open (was: Patch Available)
Need to investigate test failure.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137110#comment-13137110 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #64 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/64/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189713
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136266#comment-13136266 ]
Hadoop QA commented on MAPREDUCE-3240:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12500933/MR-3240.4.wip.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 22 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 170 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-web-proxy.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1152//console
This message is automatically generated.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136992#comment-13136992 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1167 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1167/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189711
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136930#comment-13136930 ]
Hadoop QA commented on MAPREDUCE-3240:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12501062/MAPREDUCE-3240-20111027.1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 2 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 170 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-web-proxy.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1170//console
This message is automatically generated.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Status: Patch Available (was: Open)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3240:
-----------------------------------------------
Status: Patch Available (was: Open)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136994#comment-13136994 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #74 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/74/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189713
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Status: Open (was: Patch Available)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3240:
-----------------------------------------------
Status: Open (was: Patch Available)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136717#comment-13136717 ]
Hitesh Shah commented on MAPREDUCE-3240:
----------------------------------------
Build failure related to autoreconf. Will re-submit patch once build issues fixed.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139800#comment-13139800 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #105 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/105/])
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195257
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/Makefile.am
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Status: Patch Available (was: Open)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: MR-3240.wip.patch
Patch does the following:
- introduced sending a sigterm followed by a sigkill when cleaning up a container
- new config settings introduced for the delay between sigterm and sigkill
- introduced activeContainers within the ContainerExecutor. Used by the launcher to set whether a container should be launched or not. If cleanup is called before the process starts, this flag ensures that the process is never started. Addresses race-kill issue in MR-3084
- Getting the pid after the shell executor has completed is unreliable so now task.sh writes the pid into a local file which can be read by the containerlauncher and used to kill the process.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Attachments: MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136744#comment-13136744 ]
Arun C Murthy commented on MAPREDUCE-3240:
------------------------------------------
Looks good, I ran both java MR and streaming jobs on secure clusters and things look good. +1
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: (was: MR-3240.wip.patch)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Attachments: MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139803#comment-13139803 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #105 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/105/])
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195257
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/Makefile.am
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Fix Version/s: 0.23.0
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3240:
-------------------------------------
Status: Open (was: Patch Available)
Looks great, some comments:
# Create a PidFile* class for handling the pid file
# Move pidfile to nmPrivate
# Minor nit: add a doc-comment on use of 'mv pid.tmp pid'.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137003#comment-13137003 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #70 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/70/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189713
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3240:
-----------------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Hitesh!
This was one of the more complicated patches. Thanks for being on top of this!
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136989#comment-13136989 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1183 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1183/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189711
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Status: Patch Available (was: Open)
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Attachments: MR-3240.1.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: MR-3240.3.patch
Re-submitting same patch to get pre-commit build to run
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137232#comment-13137232 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #76 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/76/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
Added the new files which I missed earlier.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189815
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/util/ProcessIdFileReader.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/util/TestProcessIdFileReader.java
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3240:
-----------------------------------
Attachment: MR-3240.5.patch
Patch addressing comments and cleaning up debug info from previous patch.
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13139812#comment-13139812 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #114 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/114/])
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1195257
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/Makefile.am
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3240) NM should send a SIGKILL for
completed containers also
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13136996#comment-13136996 ]
Hudson commented on MAPREDUCE-3240:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #74 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/74/])
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its containers (process-trees) irrespective of whether the container succeeded, or killed. Contributed by Hitesh Shah.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1189713
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/container-executor.h
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/impl/main.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/c/container-executor/test/test-container-executor.c
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutorWithMocks.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/mock-container-executor
> NM should send a SIGKILL for completed containers also
> ------------------------------------------------------
>
> Key: MAPREDUCE-3240
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3240
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Hitesh Shah
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3240-20111027.1.txt, MAPREDUCE-3240-20111027.2.txt, MR-3240.1.patch, MR-3240.2.patch, MR-3240.3.patch, MR-3240.4.wip.patch, MR-3240.5.patch, MR-3240.wip.patch
>
>
> This is to address the containers which exit properly after spawning sub-processes themselves. We don't want to leave these sub-process-tree or else they can pillage the NM's resources.
> Today, we already have code to send SIGKILL to the whole process-trees (because of single sessionId resulting from setsid) when the container is alive. We need to obtain the PID of the containers when they start and use that PID to send signal for completed containers' case also.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira