You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Alex Kozlov (JIRA)" <ji...@apache.org> on 2010/05/07 01:13:47 UTC
[jira] Created: (HADOOP-6755) Have a configurable metric reporting
CPU/disk usage per user
Have a configurable metric reporting CPU/disk usage per user
------------------------------------------------------------
Key: HADOOP-6755
URL: https://issues.apache.org/jira/browse/HADOOP-6755
Project: Hadoop Common
Issue Type: New Feature
Components: metrics
Reporter: Alex Kozlov
Many organizations are looking at resource usage per department/group/user for diagnostic and resource allocation purposes. It should be straightforward to implement a metric showing the simple resource usage like CPU time and disk I/O per user and aggregate them using Ganglia.
Eventually, we can create an API for pluggable metrics (there is one for Jobtracker and Tasktracker).
Let me know your thoughts.
Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6755) Have a configurable metric reporting
CPU/disk usage per user
Posted by "Alex Kozlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Kozlov updated HADOOP-6755:
--------------------------------
Status: Patch Available (was: Open)
Tags: monitoring
I looked at MAPREDUCE-220. I think the idea is a bit different here: to be able to monitor the usage per user in Ganglia or some other monitoring tool. I am attaching a simple patch mostly for the demo purposes.
In general, I think there should be two systems: one monitoring, focusing on a few important metrics (cpu time, memory, disk usage per user), and more detailed per task and containing more metrics, which later can be picked up by some more detailed reporting/analysis system.
Alex K
> Have a configurable metric reporting CPU/disk usage per user
> ------------------------------------------------------------
>
> Key: HADOOP-6755
> URL: https://issues.apache.org/jira/browse/HADOOP-6755
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
> Reporter: Alex Kozlov
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Many organizations are looking at resource usage per department/group/user for diagnostic and resource allocation purposes. It should be straightforward to implement a metric showing the simple resource usage like CPU time and disk I/O per user and aggregate them using Ganglia.
> Eventually, we can create an API for pluggable metrics (there is one for Jobtracker and Tasktracker).
> Let me know your thoughts.
> Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6755) Have a configurable metric reporting
CPU/disk usage per user
Posted by "Alex Kozlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex Kozlov updated HADOOP-6755:
--------------------------------
Attachment: HADOOP-6755.patch
The patch to report per-user user time and disk usage.
> Have a configurable metric reporting CPU/disk usage per user
> ------------------------------------------------------------
>
> Key: HADOOP-6755
> URL: https://issues.apache.org/jira/browse/HADOOP-6755
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
> Reporter: Alex Kozlov
> Attachments: HADOOP-6755.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Many organizations are looking at resource usage per department/group/user for diagnostic and resource allocation purposes. It should be straightforward to implement a metric showing the simple resource usage like CPU time and disk I/O per user and aggregate them using Ganglia.
> Eventually, we can create an API for pluggable metrics (there is one for Jobtracker and Tasktracker).
> Let me know your thoughts.
> Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6755) Have a configurable metric
reporting CPU/disk usage per user
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865004#action_12865004 ]
Hemanth Yamijala commented on HADOOP-6755:
------------------------------------------
This seems to be very closely related to work being discussed in MAPREDUCE-220. Can you please coordinate ?
> Have a configurable metric reporting CPU/disk usage per user
> ------------------------------------------------------------
>
> Key: HADOOP-6755
> URL: https://issues.apache.org/jira/browse/HADOOP-6755
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
> Reporter: Alex Kozlov
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Many organizations are looking at resource usage per department/group/user for diagnostic and resource allocation purposes. It should be straightforward to implement a metric showing the simple resource usage like CPU time and disk I/O per user and aggregate them using Ganglia.
> Eventually, we can create an API for pluggable metrics (there is one for Jobtracker and Tasktracker).
> Let me know your thoughts.
> Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6755) Have a configurable metric
reporting CPU/disk usage per user
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867672#action_12867672 ]
Hadoop QA commented on HADOOP-6755:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12444513/HADOOP-6755.patch
against trunk revision 944397.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/520/console
This message is automatically generated.
> Have a configurable metric reporting CPU/disk usage per user
> ------------------------------------------------------------
>
> Key: HADOOP-6755
> URL: https://issues.apache.org/jira/browse/HADOOP-6755
> Project: Hadoop Common
> Issue Type: New Feature
> Components: metrics
> Reporter: Alex Kozlov
> Attachments: HADOOP-6755.patch
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> Many organizations are looking at resource usage per department/group/user for diagnostic and resource allocation purposes. It should be straightforward to implement a metric showing the simple resource usage like CPU time and disk I/O per user and aggregate them using Ganglia.
> Eventually, we can create an API for pluggable metrics (there is one for Jobtracker and Tasktracker).
> Let me know your thoughts.
> Alex K
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.