You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2009/01/15 18:58:59 UTC
[jira] Created: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
hdfs throughput calculation is incorrect in chukwa database
-----------------------------------------------------------
Key: HADOOP-5051
URL: https://issues.apache.org/jira/browse/HADOOP-5051
Project: Hadoop Core
Issue Type: Bug
Environment: Redhat 5.1, Java 6
Reporter: Eric Yang
The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677448#action_12677448 ]
Jerome Boulon commented on HADOOP-5051:
---------------------------------------
+1
If the goal is to extend the macro language it may be good to look at a template engine like Velocity for example.
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/chukwa
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5051:
----------------------------------
Component/s: contrib/chukwa
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/chukwa
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5051:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/chukwa
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang reassigned HADOOP-5051:
---------------------------------
Assignee: Eric Yang
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681794#action_12681794 ]
Hudson commented on HADOOP-5051:
--------------------------------
Integrated in Hadoop-trunk #778 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/778/])
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/chukwa
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676848#action_12676848 ]
Ari Rabkin commented on HADOOP-5051:
------------------------------------
I don't feel qualified to review this; my SQL experience is quite limited.
Is there a way to reduce the line lengths in aggregator.sql?
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/chukwa
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5051:
------------------------------
Attachment: HADOOP-5051.patch
- Added ability to expand sum(table) macro into SQL statement.
- Change SQL statement to calculate rate for individual datanode, then aggregate the cluster throughput from datanode rates.
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676853#action_12676853 ]
Eric Yang commented on HADOOP-5051:
-----------------------------------
Every iine in the file is a SQL query. There is no way to reduce the line length for now.
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/chukwa
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5051:
------------------------------
Attachment: HADOOP-5051-1.patch
Added extra time window to dfs throughput aggregation.
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5051:
------------------------------
Attachment: (was: HADOOP-5051.patch)
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5051:
------------------------------
Status: Open (was: Patch Available)
Hdfs throughput data calculation should process data for one extra time window in case the dfs datanode metrics arrives late.
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5051:
------------------------------
Status: Patch Available (was: Open)
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051-1.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5051) hdfs throughput calculation is
incorrect in chukwa database
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5051:
------------------------------
Release Note:
What is new in HADOOP-5051:
- Added macro token subsitution for sum(table_name)
- Added correct hdfs throughput aggregation SQL macros.
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> hdfs throughput calculation is incorrect in chukwa database
> -----------------------------------------------------------
>
> Key: HADOOP-5051
> URL: https://issues.apache.org/jira/browse/HADOOP-5051
> Project: Hadoop Core
> Issue Type: Bug
> Environment: Redhat 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: HADOOP-5051.patch
>
>
> The SQL statement to calculate hdfs throughput is incorrect. The correct algorithm is to calculate metrics rate for individual datanode then sum up of all datanode's rate to get the total throughput for the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.