You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Tamas Monos <ta...@veber.co.uk> on 2012/06/01 20:19:01 UTC
RE: API usage records and SQL
Hi,
I have used git to get the cloudstack source from github to have a look at this problem.
I could not find that part of code so I run a "grep -R usageRecordGeneric ./*" in the source folder and got no results.
I can't find the code you've pasted earlier. Am I missing something?
Regards
-----Original Message-----
From: Nitin Mehta [mailto:Nitin.Mehta@citrix.com]
Sent: 31 May 2012 17:58
To: cloudstack-dev@incubator.apache.org
Subject: RE: API usage records and SQL
Hi Tamas,
Asking for 500 records takes 5-6 seconds and 1000 records taking 12 seconds shows a linear increase and so doesn't seem like there is any performance degradation of any sort here.
>From a quick glance at the code it seems that your analysis " It seems there is a conversion/parsing algorithm somewhere in CS that struggles with the data " might be correct.
I see that after fetching the data from DB, CS iterates over each record and makes a DB call for setting the usage id (conversion from ID to UUID) depending on the usage type which Indicates the DB table to look for. Example below
for (Object usageRecordGeneric : usageRecords) {
if(usageRecord.getUsageType() == UsageTypes.RUNNING_VM || usageRecord.getUsageType() == UsageTypes.ALLOCATED_VM){
//VM Instance ID
usageRecResponse.setUsageId(identityDao.getIdentityUuid("vm_instance", usageRecord.getUsageId().toString()));
} else if(usageRecord.getUsageType() == UsageTypes.IP_ADDRESS){
//IP Address ID
usageRecResponse.setUsageId(identityDao.getIdentityUuid("user_ip_address", usageRecord.getUsageId().toString()));
} else if .......// So on for different usage type.
So for fetching 1000 records its making 1 +1000 Db queries.
Thanks,
-Nitin
-----Original Message-----
From: Tamas Monos [mailto:tamasm@veber.co.uk]
Sent: Thursday, May 31, 2012 7:50 PM
To: cloudstack-dev@incubator.apache.org
Subject: API usage records and SQL
Hi,
I've spent lots of time trying to debug why does it take so long for the management server to answer the: "listUsageRecords&pagesize=1000&page=X" query.
When I ask for 500 records I get results in about 5-6 seconds. If I increase the pagesize to 1000 I need to wait 12 seconds to start getting any response.
This means to me somewhere there is a massive bottleneck either in mysql or the way cs generates the pages for response.
I have tcpdumped and got proof that the sql server responds and delivers 1000 records in 300ms and after the management "hesitates" for about 10 seconds before sends the respond down to the client.
It seems there is a conversion/parsing algorithm somewhere in CS that struggles with the data. I have and account with a month worth usage records (35k, 35 pages) and takes about 7 minutes (408sec) to get the all the data via "listUsageRecords&pagesize=1000&page=X".
Every page adds a ~12 seconds delay (408/35=11.6). I have checked the server load and cpu utilization but nothing. Java is around 10% of CPU and the load is 0.06.
This is just one account...
Any advice or suggestion is appreciated.
Regards
Tamas Monos DDI +44(0)2034687012
Chief Technical Office +44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk<http://www.veber.co.uk/>
Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>
RE: API usage records and SQL
Posted by Tamas Monos <ta...@veber.co.uk>.
Hi,
Thanks it worked!
-----Original Message-----
From: Sheng Yang [mailto:sheng@yasker.org]
Sent: 01 June 2012 19:33
To: cloudstack-dev@incubator.apache.org
Subject: Re: API usage records and SQL
Hi Tamas,
I remember we got some trouble with github repo?
Please try the current official repo:
http://git-wip-us.apache.org/repos/asf/incubator-cloudstack.git
Thanks!
-- Sheng
On Fri, Jun 1, 2012 at 11:19 AM, Tamas Monos <ta...@veber.co.uk> wrote:
> Hi,
>
> I have used git to get the cloudstack source from github to have a look at this problem.
> I could not find that part of code so I run a "grep -R usageRecordGeneric ./*" in the source folder and got no results.
> I can't find the code you've pasted earlier. Am I missing something?
>
> Regards
>
>
> -----Original Message-----
> From: Nitin Mehta [mailto:Nitin.Mehta@citrix.com]
> Sent: 31 May 2012 17:58
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: API usage records and SQL
>
> Hi Tamas,
> Asking for 500 records takes 5-6 seconds and 1000 records taking 12 seconds shows a linear increase and so doesn't seem like there is any performance degradation of any sort here.
> From a quick glance at the code it seems that your analysis " It seems there is a conversion/parsing algorithm somewhere in CS that struggles with the data " might be correct.
> I see that after fetching the data from DB, CS iterates over each
> record and makes a DB call for setting the usage id (conversion from
> ID to UUID) depending on the usage type which Indicates the DB table
> to look for. Example below
>
> for (Object usageRecordGeneric : usageRecords) {
> if(usageRecord.getUsageType() == UsageTypes.RUNNING_VM ||
> usageRecord.getUsageType() == UsageTypes.ALLOCATED_VM){
> //VM Instance ID
>
> usageRecResponse.setUsageId(identityDao.getIdentityUuid("vm_instance",
> usageRecord.getUsageId().toString()));
> } else if(usageRecord.getUsageType() ==
> UsageTypes.IP_ADDRESS){
> //IP Address ID
>
> usageRecResponse.setUsageId(identityDao.getIdentityUuid("user_ip_addre
> ss", usageRecord.getUsageId().toString()));
> } else if .......// So on for different usage type.
>
> So for fetching 1000 records its making 1 +1000 Db queries.
>
>
> Thanks,
> -Nitin
>
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Thursday, May 31, 2012 7:50 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: API usage records and SQL
>
> Hi,
>
> I've spent lots of time trying to debug why does it take so long for the management server to answer the: "listUsageRecords&pagesize=1000&page=X" query.
> When I ask for 500 records I get results in about 5-6 seconds. If I increase the pagesize to 1000 I need to wait 12 seconds to start getting any response.
>
> This means to me somewhere there is a massive bottleneck either in mysql or the way cs generates the pages for response.
> I have tcpdumped and got proof that the sql server responds and delivers 1000 records in 300ms and after the management "hesitates" for about 10 seconds before sends the respond down to the client.
> It seems there is a conversion/parsing algorithm somewhere in CS that struggles with the data. I have and account with a month worth usage records (35k, 35 pages) and takes about 7 minutes (408sec) to get the all the data via "listUsageRecords&pagesize=1000&page=X".
> Every page adds a ~12 seconds delay (408/35=11.6). I have checked the server load and cpu utilization but nothing. Java is around 10% of CPU and the load is 0.06.
>
> This is just one account...
> Any advice or suggestion is appreciated.
>
> Regards
>
> Tamas Monos DDI
> +44(0)2034687012 Chief Technical
> Office +44(0)2034687000
> Veber: The Hosting Specialists Fax +44(0)871 522
> 7057 http://www.veber.co.uk<http://www.veber.co.uk/>
>
> Follow us on Twitter:
> www.twitter.com/veberhost<http://www.twitter.com/veberhost>
> Follow us on Facebook:
> www.facebook.com/veberhost<http://www.facebook.com/veberhost>
>
>
>
Re: API usage records and SQL
Posted by Sheng Yang <sh...@yasker.org>.
Hi Tamas,
I remember we got some trouble with github repo?
Please try the current official repo:
http://git-wip-us.apache.org/repos/asf/incubator-cloudstack.git
Thanks!
-- Sheng
On Fri, Jun 1, 2012 at 11:19 AM, Tamas Monos <ta...@veber.co.uk> wrote:
> Hi,
>
> I have used git to get the cloudstack source from github to have a look at this problem.
> I could not find that part of code so I run a "grep -R usageRecordGeneric ./*" in the source folder and got no results.
> I can't find the code you've pasted earlier. Am I missing something?
>
> Regards
>
>
> -----Original Message-----
> From: Nitin Mehta [mailto:Nitin.Mehta@citrix.com]
> Sent: 31 May 2012 17:58
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: API usage records and SQL
>
> Hi Tamas,
> Asking for 500 records takes 5-6 seconds and 1000 records taking 12 seconds shows a linear increase and so doesn't seem like there is any performance degradation of any sort here.
> From a quick glance at the code it seems that your analysis " It seems there is a conversion/parsing algorithm somewhere in CS that struggles with the data " might be correct.
> I see that after fetching the data from DB, CS iterates over each record and makes a DB call for setting the usage id (conversion from ID to UUID) depending on the usage type which Indicates the DB table to look for. Example below
>
> for (Object usageRecordGeneric : usageRecords) {
> if(usageRecord.getUsageType() == UsageTypes.RUNNING_VM || usageRecord.getUsageType() == UsageTypes.ALLOCATED_VM){
> //VM Instance ID
> usageRecResponse.setUsageId(identityDao.getIdentityUuid("vm_instance", usageRecord.getUsageId().toString()));
> } else if(usageRecord.getUsageType() == UsageTypes.IP_ADDRESS){
> //IP Address ID
> usageRecResponse.setUsageId(identityDao.getIdentityUuid("user_ip_address", usageRecord.getUsageId().toString()));
> } else if .......// So on for different usage type.
>
> So for fetching 1000 records its making 1 +1000 Db queries.
>
>
> Thanks,
> -Nitin
>
> -----Original Message-----
> From: Tamas Monos [mailto:tamasm@veber.co.uk]
> Sent: Thursday, May 31, 2012 7:50 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: API usage records and SQL
>
> Hi,
>
> I've spent lots of time trying to debug why does it take so long for the management server to answer the: "listUsageRecords&pagesize=1000&page=X" query.
> When I ask for 500 records I get results in about 5-6 seconds. If I increase the pagesize to 1000 I need to wait 12 seconds to start getting any response.
>
> This means to me somewhere there is a massive bottleneck either in mysql or the way cs generates the pages for response.
> I have tcpdumped and got proof that the sql server responds and delivers 1000 records in 300ms and after the management "hesitates" for about 10 seconds before sends the respond down to the client.
> It seems there is a conversion/parsing algorithm somewhere in CS that struggles with the data. I have and account with a month worth usage records (35k, 35 pages) and takes about 7 minutes (408sec) to get the all the data via "listUsageRecords&pagesize=1000&page=X".
> Every page adds a ~12 seconds delay (408/35=11.6). I have checked the server load and cpu utilization but nothing. Java is around 10% of CPU and the load is 0.06.
>
> This is just one account...
> Any advice or suggestion is appreciated.
>
> Regards
>
> Tamas Monos DDI +44(0)2034687012
> Chief Technical Office +44(0)2034687000
> Veber: The Hosting Specialists Fax +44(0)871 522 7057
> http://www.veber.co.uk<http://www.veber.co.uk/>
>
> Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
> Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>
>
>
>