You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rashmi Dixit <ra...@accelerite.com> on 2016/05/12 04:42:24 UTC
Metrics UI - Incorrect metrics data for zone/cluster
Hi,
I came across an issue in the metrics for zone and cluster. I have logged a bug for the same. https://issues.apache.org/jira/browse/CLOUDSTACK-9377
To quickly summarize the issue: APIs have a pagesize restriction - the default.page.size global setting is used as the max result count - value is typically 500. In this case, listClusters and listHosts APIS are used to gather metrics data for zone and clusters. If there are 600 hosts, then one call to listHosts will only return 500 hosts. The usage data is summed up for 500 hosts, however average is calculated based the total hosts which is 600.
To correct this, UI should fetch ALL hosts for ALL clusters to get the right result. Ideally this kind of calculation should be done at the API level - large number of hosts will result in performance degradation. The right way to fix this would be to add a new API for metrics calculations at zones/cluster level.
@Rohit, since you are the original contributor, can you help me with the background on this? Any comments/suggestions on doing this differently are most welcome.
Thanks!
Rashmi
Rashmi Dixit
Principal Product Engineer | CloudPlatform | www.accelerite.com
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: Metrics UI - Incorrect metrics data for zone/cluster
Posted by Rashmi Dixit <ra...@accelerite.com>.
Thank you Rohit. The workaround you have provided works.
-----Original Message-----
From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Sent: Thursday, May 12, 2016 12:32 PM
To: dev@cloudstack.apache.org
Subject: RE: Metrics UI - Incorrect metrics data for zone/cluster
Rashmi, I've left comments on the JIRA ticket you've created. Can you follow them and let me know if the workaround (with pagesize=-1) works for you.
Since we're doing the calculations and performing HTTP requests on UI/client side, it's a known limitation that it will perform O(m*n) http requests where m is number of clusters and n is avg hosts per cluster.
Regards.
Regards,
Rohit Yadav
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Rashmi Dixit [mailto:rashmi_dixit@accelerite.com]
Sent: Thursday, May 12, 2016 11:59 AM
To: dev@cloudstack.apache.org
Subject: RE: Metrics UI - Incorrect metrics data for zone/cluster
Thanks Rohit.
Once you play with the page size - set it to say 40, then the issues show up. This also affects the number of cluster calculations as well.
-----Original Message-----
From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Sent: Thursday, May 12, 2016 11:44 AM
To: dev@cloudstack.apache.org
Subject: RE: Metrics UI - Incorrect metrics data for zone/cluster
Rashmi,
Originally, the feature was implemented using all the available list APIs in future we may have a backend metrics API instead of aggregation in the UI/client side (done right now).
This feature assumes that no single cluster would have 500 hosts (also not supported by many hypervisors to have 500+ hosts in a single cluster), the calculations are done for each cluster in the cluster metrics view; similarly in zones metrics view calculations are done looping through clusters per zone.
Feel free to assign the issue to me, I'll get back to you later.
Regards.
Regards,
Rohit Yadav
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Rashmi Dixit [mailto:rashmi_dixit@accelerite.com]
Sent: Thursday, May 12, 2016 10:12 AM
To: dev@cloudstack.apache.org
Subject: Metrics UI - Incorrect metrics data for zone/cluster
Hi,
I came across an issue in the metrics for zone and cluster. I have logged a bug for the same. https://issues.apache.org/jira/browse/CLOUDSTACK-9377
To quickly summarize the issue: APIs have a pagesize restriction - the default.page.size global setting is used as the max result count - value is typically 500. In this case, listClusters and listHosts APIS are used to gather metrics data for zone and clusters. If there are 600 hosts, then one call to listHosts will only return 500 hosts. The usage data is summed up for 500 hosts, however average is calculated based the total hosts which is 600.
To correct this, UI should fetch ALL hosts for ALL clusters to get the right result. Ideally this kind of calculation should be done at the API level - large number of hosts will result in performance degradation. The right way to fix this would be to add a new API for metrics calculations at zones/cluster level.
@Rohit, since you are the original contributor, can you help me with the background on this? Any comments/suggestions on doing this differently are most welcome.
Thanks!
Rashmi
Rashmi Dixit
Principal Product Engineer | CloudPlatform | www.accelerite.com
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: Metrics UI - Incorrect metrics data for zone/cluster
Posted by Rohit Yadav <ro...@shapeblue.com>.
Rashmi, I've left comments on the JIRA ticket you've created. Can you follow them and let me know if the workaround (with pagesize=-1) works for you.
Since we're doing the calculations and performing HTTP requests on UI/client side, it's a known limitation that it will perform O(m*n) http requests where m is number of clusters and n is avg hosts per cluster.
Regards.
Regards,
Rohit Yadav
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Rashmi Dixit [mailto:rashmi_dixit@accelerite.com]
Sent: Thursday, May 12, 2016 11:59 AM
To: dev@cloudstack.apache.org
Subject: RE: Metrics UI - Incorrect metrics data for zone/cluster
Thanks Rohit.
Once you play with the page size - set it to say 40, then the issues show up. This also affects the number of cluster calculations as well.
-----Original Message-----
From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Sent: Thursday, May 12, 2016 11:44 AM
To: dev@cloudstack.apache.org
Subject: RE: Metrics UI - Incorrect metrics data for zone/cluster
Rashmi,
Originally, the feature was implemented using all the available list APIs in future we may have a backend metrics API instead of aggregation in the UI/client side (done right now).
This feature assumes that no single cluster would have 500 hosts (also not supported by many hypervisors to have 500+ hosts in a single cluster), the calculations are done for each cluster in the cluster metrics view; similarly in zones metrics view calculations are done looping through clusters per zone.
Feel free to assign the issue to me, I'll get back to you later.
Regards.
Regards,
Rohit Yadav
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Rashmi Dixit [mailto:rashmi_dixit@accelerite.com]
Sent: Thursday, May 12, 2016 10:12 AM
To: dev@cloudstack.apache.org
Subject: Metrics UI - Incorrect metrics data for zone/cluster
Hi,
I came across an issue in the metrics for zone and cluster. I have logged a bug for the same. https://issues.apache.org/jira/browse/CLOUDSTACK-9377
To quickly summarize the issue: APIs have a pagesize restriction - the default.page.size global setting is used as the max result count - value is typically 500. In this case, listClusters and listHosts APIS are used to gather metrics data for zone and clusters. If there are 600 hosts, then one call to listHosts will only return 500 hosts. The usage data is summed up for 500 hosts, however average is calculated based the total hosts which is 600.
To correct this, UI should fetch ALL hosts for ALL clusters to get the right result. Ideally this kind of calculation should be done at the API level - large number of hosts will result in performance degradation. The right way to fix this would be to add a new API for metrics calculations at zones/cluster level.
@Rohit, since you are the original contributor, can you help me with the background on this? Any comments/suggestions on doing this differently are most welcome.
Thanks!
Rashmi
Rashmi Dixit
Principal Product Engineer | CloudPlatform | www.accelerite.com
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: Metrics UI - Incorrect metrics data for zone/cluster
Posted by Rashmi Dixit <ra...@accelerite.com>.
Thanks Rohit.
Once you play with the page size - set it to say 40, then the issues show up. This also affects the number of cluster calculations as well.
-----Original Message-----
From: Rohit Yadav [mailto:rohit.yadav@shapeblue.com]
Sent: Thursday, May 12, 2016 11:44 AM
To: dev@cloudstack.apache.org
Subject: RE: Metrics UI - Incorrect metrics data for zone/cluster
Rashmi,
Originally, the feature was implemented using all the available list APIs in future we may have a backend metrics API instead of aggregation in the UI/client side (done right now).
This feature assumes that no single cluster would have 500 hosts (also not supported by many hypervisors to have 500+ hosts in a single cluster), the calculations are done for each cluster in the cluster metrics view; similarly in zones metrics view calculations are done looping through clusters per zone.
Feel free to assign the issue to me, I'll get back to you later.
Regards.
Regards,
Rohit Yadav
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue
-----Original Message-----
From: Rashmi Dixit [mailto:rashmi_dixit@accelerite.com]
Sent: Thursday, May 12, 2016 10:12 AM
To: dev@cloudstack.apache.org
Subject: Metrics UI - Incorrect metrics data for zone/cluster
Hi,
I came across an issue in the metrics for zone and cluster. I have logged a bug for the same. https://issues.apache.org/jira/browse/CLOUDSTACK-9377
To quickly summarize the issue: APIs have a pagesize restriction - the default.page.size global setting is used as the max result count - value is typically 500. In this case, listClusters and listHosts APIS are used to gather metrics data for zone and clusters. If there are 600 hosts, then one call to listHosts will only return 500 hosts. The usage data is summed up for 500 hosts, however average is calculated based the total hosts which is 600.
To correct this, UI should fetch ALL hosts for ALL clusters to get the right result. Ideally this kind of calculation should be done at the API level - large number of hosts will result in performance degradation. The right way to fix this would be to add a new API for metrics calculations at zones/cluster level.
@Rohit, since you are the original contributor, can you help me with the background on this? Any comments/suggestions on doing this differently are most welcome.
Thanks!
Rashmi
Rashmi Dixit
Principal Product Engineer | CloudPlatform | www.accelerite.com
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
RE: Metrics UI - Incorrect metrics data for zone/cluster
Posted by Rohit Yadav <ro...@shapeblue.com>.
Rashmi,
Originally, the feature was implemented using all the available list APIs in future we may have a backend metrics API instead of aggregation in the UI/client side (done right now).
This feature assumes that no single cluster would have 500 hosts (also not supported by many hypervisors to have 500+ hosts in a single cluster), the calculations are done for each cluster in the cluster metrics view; similarly in zones metrics view calculations are done looping through clusters per zone.
Feel free to assign the issue to me, I'll get back to you later.
Regards.
Regards,
Rohit Yadav
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
-----Original Message-----
From: Rashmi Dixit [mailto:rashmi_dixit@accelerite.com]
Sent: Thursday, May 12, 2016 10:12 AM
To: dev@cloudstack.apache.org
Subject: Metrics UI - Incorrect metrics data for zone/cluster
Hi,
I came across an issue in the metrics for zone and cluster. I have logged a bug for the same. https://issues.apache.org/jira/browse/CLOUDSTACK-9377
To quickly summarize the issue: APIs have a pagesize restriction - the default.page.size global setting is used as the max result count - value is typically 500. In this case, listClusters and listHosts APIS are used to gather metrics data for zone and clusters. If there are 600 hosts, then one call to listHosts will only return 500 hosts. The usage data is summed up for 500 hosts, however average is calculated based the total hosts which is 600.
To correct this, UI should fetch ALL hosts for ALL clusters to get the right result. Ideally this kind of calculation should be done at the API level - large number of hosts will result in performance degradation. The right way to fix this would be to add a new API for metrics calculations at zones/cluster level.
@Rohit, since you are the original contributor, can you help me with the background on this? Any comments/suggestions on doing this differently are most welcome.
Thanks!
Rashmi
Rashmi Dixit
Principal Product Engineer | CloudPlatform | www.accelerite.com
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.