You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jason (JIRA)" <ji...@apache.org> on 2008/05/20 19:35:55 UTC
[jira] Created: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
------------------------------------------------------------------------------------------------------------
Key: HADOOP-3422
URL: https://issues.apache.org/jira/browse/HADOOP-3422
Project: Hadoop Core
Issue Type: Bug
Components: metrics
Affects Versions: 0.16.4
Reporter: Jason
Fix For: 0.17.0
The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nigel Daley updated HADOOP-3422:
--------------------------------
Fix Version/s: (was: 0.17.0)
0.18.0
This is not a blocker for 0.17.0 (for which a release candidate is being voted on). Moving to 0.18
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Fix For: 0.18.0
>
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Jason (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason updated HADOOP-3422:
--------------------------
Attachment: ganglia-patch-3422-qand-4137-1.patch
This patch removes the thread local code and uses StringBuilder instead of StringBuffer.
The core ganglia pieces are unchanged.
At this point an outstanding issue is that it is not possible to distinguish the metrics from different jvms.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638717#action_12638717 ]
stack commented on HADOOP-3422:
-------------------------------
Good stuff Brian.
1). Just by way of FYI, see MetricsUtil in same package for how to add logging. Import two classes, then just inside your class opening, add something like:
{code}
private static final Log LOG =
LogFactory.getLog("org.apache.hadoop.util.GangliaContext");
{code}
In code, do something like this: LOG.warn("Type is null"); ... and so on.
2). Let me write Jason and see what he was up to with the ThreadLocal. I don't understand. Its not keeping any state so why bother with it.
3). Doesn't your patch also address HADOOP-4137 whereas diff-20080520-1025.txt does not?
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616619#action_12616619 ]
Lohit Vijayarenu commented on HADOOP-3422:
------------------------------------------
Trunk has changed, would it be possible to regenerate this patch? We can make it PA once done.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12636863#action_12636863 ]
Hadoop QA commented on HADOOP-3422:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12391475/ganglia-patch-3422-and-4137.patch
against trunk revision 701476.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3433/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3433/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3433/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3433/console
This message is automatically generated.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-3422:
--------------------------
Attachment: ganglia-patch-3422-updated-v2.patch
Version of patch that addresses the above comments (Uses StringBuilder instead of StringBuffer, removes extraneous setLength(0), etc.). Also adds overload of JVMMetrics#init method so can pass name of record to use. Default is to use the old 'metrics' name. Overload makes it so can pass in alternate record name. This makes it so can have metrics per jvm rather than have all jvm metrics lumped together. For example, in hbase, we can now have graphs for both the master and regionserver's jvms show in ganglia. Doing similar for rpc didn't make as much sense since method names are shown (and override of RpcMetrics creation is not possible currently).
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-3422:
------------------------------------
Attachment: ganglia-patch-3422-updated.patch
The attached patch does the following:
1) Removes the TODO lines by adding the logger.
2) Incorporates HADOOP-4137. I'll mark that one as a duplicate of this.
3) Includes Jason's updated patch.
Builds cleanly against trunk.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3422:
------------------------------------
Release Note: Changed names of ganglia metrics to avoid conflicts and to better identify source function. (was: Produce ganglia metrics with names of the form:
jvm.metrics.*
mapred.job.Job Counters .*
mapred.job.Map-Reduce Framework.*
mapred.job.Map-Reduce Framework.*
mapred.job.Map-Reduce Framework.*
mapred.jobtracker.*
mapred.shuffleInput.*
mapred.tasktracker.*
instead of just using the last token name, or in the case of Per Job Counters, the string value, for all counters.
The net result of this change is that the metrics names make it clear which part of hadoop is reporting, and the counter values each have their own RRD file instead of all the counters being aggregated under the name "value")
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed])
Edit release note for publication.
jvm.metrics.*
mapred.job.Job Counters .*
mapred.job.Map-Reduce Framework.*
mapred.job.Map-Reduce Framework.*
mapred.job.Map-Reduce Framework.*
mapred.jobtracker.*
mapred.shuffleInput.*
mapred.tasktracker.*
instead of just using the last token name, or in the case of Per Job Counters, the string value, for all counters.
The net result of this change is that the metrics names make it clear which part of hadoop is reporting, and the counter values each have their own RRD file instead of all the counters being aggregated under the name "value"
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638491#action_12638491 ]
stack commented on HADOOP-3422:
-------------------------------
I took a look at the patch. You have this in a few places '// TODO: Log this.'. Why not log the exception?
You keep a ThreadLocal StringBuffer to save on construction cost? Could you use a StringBuilder instead of a StringBuffer?
Otherwise, this patch is great. I tried it. Ganglia graphs work with this in place.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Jason (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598652#action_12598652 ]
Jason commented on HADOOP-3422:
-------------------------------
The names being produced for ganglia are long and tend to overflow the bounding box in the graph.
mapred.job.FQCN.COUNTER_NAME
where FQCN is the fully qualified class that contains the counter ie: org.apache.hadoop.mapred.XXX.YYY
And it would be nice to get this from the individual machines as well as the cluster agreggate so you could tell at a glance how the individual machines were doing. I suspect that will be annother patch for that new feature.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Fix For: 0.18.0
>
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638708#action_12638708 ]
Brian Bockelman commented on HADOOP-3422:
-----------------------------------------
Hey stack,
1) I didn't add the log statements because I don't know the "preferred" way to add a logger to classes for this project. I hoped someone who knew would come along and help out.
2) I don't know the logic behind the ThreadLocal StringBuffer stuff. Adoption from the original patch.
3) The update to the patch which maps Long to float is bust. The class doesn't have a way to serialize floats with XDR, so I'm not sure what would happen.
For now, use the original, not the updated patch - having the buffer overflow is better than nothing at all with floats.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649534#action_12649534 ]
stack commented on HADOOP-3422:
-------------------------------
I agree the test failure is unrelated but would like to post another patch anyways to address comments above.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12616506#action_12616506 ]
Lohit Vijayarenu commented on HADOOP-3422:
------------------------------------------
bq. Are there any unit tests that tests the HadoopMetrics API?
Not as of now. Have opened a JIRA HADOOP-3634 for this. Should be easy if we model something along the lines of FileContext.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-3422:
--------------------------
Status: In Progress (was: Patch Available)
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.2, 0.18.1, 0.18.0, 0.17.2
> Reporter: Jason
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650279#action_12650279 ]
stack commented on HADOOP-3422:
-------------------------------
Thanks for the review Brian.
Unless objection, I'll commit to TRUNK in the next day or so.
Should I commit to branch-0.19 too? I'd like to but its only a 'major' bug, not critical nor a blocker.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Assignee: stack
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Jason (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason updated HADOOP-3422:
--------------------------
Attachment: diff-20080520-1025.txt
A patch against trunk to provide this.
Modifies JobInProgress and GangliaContext
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Fix For: 0.17.0
>
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600845#action_12600845 ]
dhruba borthakur commented on HADOOP-3422:
------------------------------------------
+1. This patch is very helpful. Are there any unit tests that tests the HadoopMetrics API?
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Fix For: 0.18.0
>
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3422:
----------------------------------
Status: Open (was: Patch Available)
What is the status of this issue and HADOOP-4137? The latter patch has been absorbed into the one posted here, but the two fixes appear independent (and the latest conflict). I'm canceling both patches for now, since it is unclear whether either is to be committed.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-3422:
------------------------------------
Fix Version/s: 0.20.0
0.19.1
hudson
Affects Version/s: 0.17.2
0.18.0
0.18.2
Status: Patch Available (was: Open)
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.2, 0.18.1, 0.18.0, 0.17.2
> Reporter: Jason
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Jason (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598492#action_12598492 ]
Jason commented on HADOOP-3422:
-------------------------------
They did not work correctly in 0.15.3 or in 0.16.0. It is a very minor change and I think people would like it.
The only other issue i currently have is that i have to set up my hadoop-metrics.properties files such that the report to node:
jvm.servers=HOST:8649 ..., where HOST is the head node, if i use localhost, the metrics end up all in a bucket under the psuedo machine localhost.localdomain
If the HOST is a real host, they show up under the node's statistics
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Fix For: 0.18.0
>
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650215#action_12650215 ]
Brian Bockelman commented on HADOOP-3422:
-----------------------------------------
I set HADOOP-4675 to depend on this, as we can't start supporting Ganglia 3.1 until these basic Ganglia issues are resolved.
I like stack's refinements. Am I allowed to vote +1 on it if I helped push the patch along? I guess probably not...
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Assignee: stack
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598482#action_12598482 ]
Doug Cutting commented on HADOOP-3422:
--------------------------------------
Did these counters work with Ganglia in a prior release? If so, then this is a regression and might be a candidate for 0.17.1...
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Fix For: 0.18.0
>
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HADOOP-3422:
-----------------------------
Assignee: stack
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Assignee: stack
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649453#action_12649453 ]
Hadoop QA commented on HADOOP-3422:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394071/ganglia-patch-3422-updated.patch
against trunk revision 719152.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3611/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3611/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3611/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3611/console
This message is automatically generated.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650255#action_12650255 ]
Doug Cutting commented on HADOOP-3422:
--------------------------------------
> Am I allowed to vote +1 on it if I helped push the patch along? I guess probably not...
Yes, of course you are allowed to vote. In the case of a dispute, PMC votes trump contributors, but everyone is encouraged to vote, and their opinions are taken seriously.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Assignee: stack
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-3422:
------------------------------------
Attachment: ganglia-patch-3422-and-4137.patch
Updated patch: Silly me, Ganglia has no long data type - and I mapped it to int32, which overflows quickly (especially when reporting bytes transferred!).
This now maps Long to float in Ganglia.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649531#action_12649531 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3422:
------------------------------------------------
Seems that the failed test, TestFsck, is not robust. There was another failing in HADOOP-4647. I believe both of the failing cases are not related to the patches posted.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-3422:
--------------------------
Resolution: Fixed
Fix Version/s: (was: 0.19.1)
(was: hudson)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks for the patch Jason and Brian.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649991#action_12649991 ]
Hadoop QA commented on HADOOP-3422:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12394440/ganglia-patch-3422-updated-v2.patch
against trunk revision 719787.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3635/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3635/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3635/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3635/console
This message is automatically generated.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Assignee: stack
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-3422:
------------------------------------
Attachment: ganglia-patch-3422-and-4137.patch
Patch against more recent trunk (rev 701680). Combined the existing patch with the one submitter for #HADOOP-4137; without both, Ganglia metrics appear to be broken.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HADOOP-3422:
--------------------------
Status: Patch Available (was: In Progress)
Try hudson w/ new version of patch.
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.2, 0.18.1, 0.18.0, 0.17.2
> Reporter: Jason
> Assignee: stack
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated-v2.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Mukund Madhugiri (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mukund Madhugiri updated HADOOP-3422:
-------------------------------------
Fix Version/s: (was: 0.18.0)
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.16.4
> Reporter: Jason
> Attachments: diff-20080520-1025.txt
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-3422:
------------------------------------
Affects Version/s: (was: 0.16.4)
0.18.1
Status: Patch Available (was: Open)
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.18.1
> Reporter: Jason
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3422) Ganglia counter metrics are all
reported with the metric name "value", so the counter values can not be
seen
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648700#action_12648700 ]
stack commented on HADOOP-3422:
-------------------------------
Couple of nitpicks: Use StringBuilder instead of StringBuffer and the set of StringBuffer length to zero on line #47 is not needed since you just created a new instance. Also, make comment on line #45 < 80 characters. Otherwise, I tried the patch against 0.19.0. All works.
What you think we should do about rpc and jvm 'record' names? They both use 'metrics'. Its kinda useless. Means all rpc and jvm metrics get lumped together. Other metrics records are better named: e.g. 'datanode', 'namenode', etc. Should we fix this as part of this patch or in a new issue?
The jvm metrics record might be named for the application its hosting. And rpc the same?
> Ganglia counter metrics are all reported with the metric name "value", so the counter values can not be seen
> ------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-3422
> URL: https://issues.apache.org/jira/browse/HADOOP-3422
> Project: Hadoop Core
> Issue Type: Bug
> Components: metrics
> Affects Versions: 0.17.2, 0.18.0, 0.18.1, 0.18.2
> Reporter: Jason
> Fix For: hudson, 0.19.1, 0.20.0
>
> Attachments: diff-20080520-1025.txt, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-and-4137.patch, ganglia-patch-3422-qand-4137-1.patch, ganglia-patch-3422-updated.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The JobInProgress class reports all metrics with the name "value". The FileMetrics class puts all of the tags into the name when reporting the individual values, but the Ganglia Context does not put the tags into the name..
> This patch modifies the context to build names for the counter metrics out of the tag values. This enables the user to see the indivdual counter values with the ganglia web tool, on a per job basis
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.