You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2011/07/07 20:31:16 UTC
[jira] [Created] (MAPREDUCE-2655) MR279: Audit logs for YARN
MR279: Audit logs for YARN
---------------------------
Key: MAPREDUCE-2655
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
Assignee: Thomas Graves
We need audit logs for YARN components:
ResourceManager:
- All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
refreshUserToGroupMappings.
- All app-submissions, app-kills to RM.
- Illegal and successful(?) AM registrations.
- Illegal container allocations/deallocations from AMs
- Successful container allocations/deallocations from AMs too?
NodeManager:
- Illegal container launches from AMs
- Successful container launches from AMs too?
Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092082#comment-13092082 ]
Thomas Graves commented on MAPREDUCE-2655:
------------------------------------------
oops looks like my last patch missed a couple of new files I added.. will upload a new one shortly.
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098529#comment-13098529 ]
Hudson commented on MAPREDUCE-2655:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #918 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/918/])
Adding new files for MAPREDUCE-2655.
MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165950
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAuditLogger.java
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165949
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Status: Patch Available (was: Open)
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Attachment: MAPREDUCE-2655.patch
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183083#comment-13183083 ]
Mahadev konar commented on MAPREDUCE-2655:
------------------------------------------
@Eli,
Looking at the subversion commits it looks like its merged. Where do you see this not being merged?
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MAPREDUCE-2655) MR279: Audit logs
for YARN
Posted by "Mahadev konar (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183084#comment-13183084 ]
Mahadev konar edited comment on MAPREDUCE-2655 at 1/10/12 5:20 AM:
-------------------------------------------------------------------
Here's the commit:
{noformat}
Repository Revision Date User Message
ASF #1165951 Sep 07 acmurthy Merge -r 1165948:1165950 from trunk to BRANCH-0.23 to fix MAPREDUCE-2655.
{noformat}
was (Author: mahadev):
Here's the commit:
{noformat}
Repository Revision Date User Message
ASF #1165949 Sep 07 acmurthy MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. Contributed by Thomas Graves.
{noformat}
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Attachment: nm-audit.log
sample nodemanager audit log. Corresponds to attached rm audit log. was from successful run of simple wordcount.
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Attachments: nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084267#comment-13084267 ]
Mahadev konar commented on MAPREDUCE-2655:
------------------------------------------
Thomas,
Are the NM audit logs for container start only?
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Attachment: MAPREDUCE-2655-v3.patch
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084272#comment-13084272 ]
Thomas Graves commented on MAPREDUCE-2655:
------------------------------------------
yes currently that is all its doing. my understand was that the authentication would block illegal allocations. Is there others we want to log?
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-2655:
-----------------------------------
Target Version/s: 0.23.0
Fix Version/s: (was: 0.23.0)
0.24.0
This was marked with fixVersion 0.23.0 but never merged, should it be?
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2655:
-----------------------------------------------
Fix Version/s: (was: 0.23.1)
0.23.0
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Status: Patch Available (was: Open)
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092833#comment-13092833 ]
Thomas Graves commented on MAPREDUCE-2655:
------------------------------------------
the javadoc and findbugs errors are known issues on trunk and not introduced by this jira.
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087898#comment-13087898 ]
Thomas Graves commented on MAPREDUCE-2655:
------------------------------------------
on the NM side I added logging the following:
* Container allocation successes and failure
* successful container completes (ie done)
* failed containers
* killed containers
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084226#comment-13084226 ]
Thomas Graves commented on MAPREDUCE-2655:
------------------------------------------
I attached 2 samples file of what audit logs would look like from simple wordcount on small file. Note, I noticed cut and paste error in the rm-audit.log. The APPID of a couple of those should be either CONTAINERID OR APPATTEMPTID.
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Attachments: nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2655:
-------------------------------------
Attachment: MAPREDUCE-2655-v4.patch
Minor update to merge with some recent changes.
Patch looks good. +1
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Attachment: MAPREDUCE-2655-v2.patch
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084118#comment-13084118 ]
Thomas Graves commented on MAPREDUCE-2655:
------------------------------------------
Going to add a RMAuditLogger class that has methods to log success/failure similar to jobtracker Auditlogger class.
Certain of these don't make sense to log since the authentication will throw away unauthorized requests.
Right now I plan on logging the following things:
RM:
- all refresh commands - success and failure
- Application submissions
- Application kills
- AM registrations - failure and success
- illegal container allocations from AM - invalid container id or container not owned by App
- successful container allocations and deallocations from AM
NM:
- Container allocation successes
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated MAPREDUCE-2655:
-----------------------------------
Target Version/s: 0.23.1 (was: 0.23.0)
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098524#comment-13098524 ]
Hudson commented on MAPREDUCE-2655:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #841 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/841/])
Adding new files for MAPREDUCE-2655.
MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165950
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAuditLogger.java
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165949
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092109#comment-13092109 ]
Hadoop QA commented on MAPREDUCE-2655:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12491843/MAPREDUCE-2655-v3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 73 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/546//console
This message is automatically generated.
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098888#comment-13098888 ]
Hudson commented on MAPREDUCE-2655:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #786 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/786/])
Adding new files for MAPREDUCE-2655.
MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165950
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAuditLogger.java
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165949
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Status: Open (was: Patch Available)
need to update patch to new directory structure
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084304#comment-13084304 ]
Mahadev konar commented on MAPREDUCE-2655:
------------------------------------------
What abt done containers? killed containers? failed containers?
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2655:
-------------------------------------
Fix Version/s: 0.23.0
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098528#comment-13098528 ]
Hudson commented on MAPREDUCE-2655:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #852 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/852/])
Adding new files for MAPREDUCE-2655.
MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165950
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAuditLogger.java
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165949
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-2655:
-------------------------------------
Attachment: rm-audit.log
sample of what rm audit log would look like. This is running successful wordcount on small input file.
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Attachments: rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2655:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Thomas!
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098912#comment-13098912 ]
Hudson commented on MAPREDUCE-2655:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #809 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/809/])
Adding new files for MAPREDUCE-2655.
MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. Contributed by Thomas Graves.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165950
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMAuditLogger.java
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165949
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/AdminService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084191#comment-13084191 ]
Thomas Graves commented on MAPREDUCE-2655:
------------------------------------------
Example on how you would config for rm and nm audit logs:
rm.audit.logger=INFO,RMAUDIT
log4j.logger.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=${rm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger=false
log4j.appender.RMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.RMAUDIT.File=${hadoop.log.dir}/rm-audit.log
log4j.appender.RMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.RMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.RMAUDIT.DatePattern=.yyyy-MM-dd
nm.audit.logger=INFO,NMAUDIT
log4j.logger.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=${nm.audit.logger}
log4j.additivity.org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger=false
log4j.appender.NMAUDIT=org.apache.log4j.DailyRollingFileAppender
log4j.appender.NMAUDIT.File=${hadoop.log.dir}/nm-audit.log
log4j.appender.NMAUDIT.layout=org.apache.log4j.PatternLayout
log4j.appender.NMAUDIT.layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
log4j.appender.NMAUDIT.DatePattern=.yyyy-MM-dd
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092022#comment-13092022 ]
Hadoop QA commented on MAPREDUCE-2655:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12491678/MAPREDUCE-2655-v2.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 javadoc. The javadoc tool appears to have generated 6 warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/540//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/540//console
This message is automatically generated.
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183084#comment-13183084 ]
Mahadev konar commented on MAPREDUCE-2655:
------------------------------------------
Here's the commit:
{noformat}
Repository Revision Date User Message
ASF #1165949 Sep 07 acmurthy MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. Contributed by Thomas Graves.
{noformat}
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2655) MR279: Audit logs for YARN
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2655:
-------------------------------------
Fix Version/s: 0.23.1
Was merged in 0.23.1
> MR279: Audit logs for YARN
> ---------------------------
>
> Key: MAPREDUCE-2655
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2655
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Fix For: 0.23.1, 0.24.0
>
> Attachments: MAPREDUCE-2655-v2.patch, MAPREDUCE-2655-v3.patch, MAPREDUCE-2655-v4.patch, MAPREDUCE-2655.patch, nm-audit.log, rm-audit.log
>
>
> We need audit logs for YARN components:
> ResourceManager:
> - All the refresh* protocol access points - refreshQueues, refreshNodes, refreshProxyUsers,
> refreshUserToGroupMappings.
> - All app-submissions, app-kills to RM.
> - Illegal and successful(?) AM registrations.
> - Illegal container allocations/deallocations from AMs
> - Successful container allocations/deallocations from AMs too?
> NodeManager:
> - Illegal container launches from AMs
> - Successful container launches from AMs too?
> Not sure if we need audit logs from MR AMs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira