You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Nicolas Vazquez <Ni...@shapeblue.com> on 2020/11/03 14:03:17 UTC
Re: listUsageRecords Update frequency
Hi Hean,
Looks like no data is being processed at all, the usage server takes the raw data from usage_event table and parses it into helper tables (such as usage_network). The data from the usage_event at the cloud_usage database is copied from the usage_event table on the cloud database. Can you check the records from those tables and see if there is a discrepancy? Also, can you verify if the usage db.properties is configured properly?
Regards,
Nicolas Vazquez
________________________________
From: Hean Seng <he...@gmail.com>
Sent: Saturday, October 31, 2020 5:41 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: listUsageRecords Update frequency
I think selec the wrong id, but still is return empty set:
mysql> select * from usage_network where account_id=59;
Empty set (0.00 sec)
root@mgmt01 usage]# more usage.log | grep "accountId: 59"
2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl]
(Usage-Job-1:null) (logid:) getting current accounted bytes for... accountId:
59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
2020-10-31 02
Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <he...@gmail.com> wrote:
> Hi Nicolas
>
> The cloudstacl_usage is running . confirm from log
>
> 20-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
> (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, will
> send statement in next cycle
>
> 2020-10-30 18:45:11,107 INFO [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) usage job complete
>
> ^C
>
> It looks like it did not write to usage table as you say:
>
> mysql> select * from usage_network where
> account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ;
>
> Empty set, 1 warning (0.00 sec)
>
>
>
>
> Where is process to update the usage helper?
>
>
>
>
> On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
>> Hi Hean,
>>
>> The listVirtualMachineMetrics show the "current" usage of the virtual
>> machine, and the usage records display the aggregated information (hourly,
>> daily) per account. If the records are not updating can you please check
>> the usage server logs if there is any error when the usage jobs run? Can
>> you also check if data is being added on the helper table for network usage
>> (usage_network table)? The information from this table is summarized into
>> the resulting usage records when the usage jobs run.
>>
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>> ________________________________
>> From: Hean Seng <he...@gmail.com>
>> Sent: Friday, October 30, 2020 4:18 AM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Re: listUsageRecords Update frequency
>>
>> Hi Nicolas
>>
>> Thanks for answering.
>>
>> Some questions puzzle me, what is different between listUsageRecords and
>> listVirtualMachineMetrics , in terms of Network Read and Write data.
>> (other
>> then one is per account basis, and another one is per VM basis) , the
>> data
>> in listVirtualMachineMetrics almost instant update .
>>
>> Is listUsageReocrds (type 4 and type 5) only keep metric with Public IP ,
>> and not calculated the SharedNetwork ?
>>
>> I facing issue of AdvanceNetwork withSG, and listVirtualMachineMetrics
>> network sent and receive update almost instant, and listUsageRecords never
>> update , I do not know where to check if there is error when updating this
>> Account. Or if the account have setting/options to choose if process
>> usage
>> by Cloudstack-Usage ?
>>
>>
>>
>>
>>
>>
>> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez <
>> Nicolas.Vazquez@shapeblue.com> wrote:
>>
>> > Yes, the usage jobs information is stored in the cloud_usage database on
>> > the usage_job table.
>> >
>> > The start_date and end_date columns indicate which usage range was
>> parsed
>> > and the start_millis and end_millis the jobs execution start and end
>> time
>> > in milliseconds, which can be converted to date.
>> >
>> >
>> > Regards,
>> >
>> > Nicolas Vazquez
>> >
>> > ________________________________
>> > From: Hean Seng <he...@gmail.com>
>> > Sent: Thursday, October 29, 2020 4:58 PM
>> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> > Subject: Re: listUsageRecords Update frequency
>> >
>> > Hi NIC
>> >
>> > Do know where can confirm the last run for usage server ?
>> >
>> > It seems my counter is not running , my setup is AdvanceZone Security
>> > Group.
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez <
>> > Nicolas.Vazquez@shapeblue.com> wrote:
>> >
>> > > Hi Hean,
>> > >
>> > > You are looking at the right configuration, if you set
>> > > 'usage.stats.job.aggregation.range' = 60 then you only need to restart
>> > the
>> > > usage server to propagate the changes.
>> > >
>> > > Also please check the 'usage.stats.job.exec.time' configuration, you
>> can
>> > > set that exec time to a previous time in the day so after the usage
>> > server
>> > > restarts, the usage jobs start running, and then the interval remains
>> > > hourly (by the previous configuration)
>> > >
>> > >
>> > > Regards,
>> > >
>> > > Nicolas Vazquez
>> > >
>> > > ________________________________
>> > > From: Hean Seng <he...@gmail.com>
>> > > Sent: Wednesday, October 28, 2020 3:10 PM
>> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> > > Subject: listUsageRecords Update frequency
>> > >
>> > > Hi
>> > >
>> > > I notice the usage record update once in a day .
>> > >
>> > > Is that possible to update the Usage data more frequently? May be
>> hourly
>> > > etc.
>> > >
>> > > I try to update usage.stats.job.aggregation.range but seem that is
>> not I
>> > > am looking for, it only affect startdate and enddate of the record,
>> but
>> > > not update usage record interval.
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Hean Seng
>> > >
>> > > Nicolas.Vazquez@shapeblue.com
>> > > www.shapeblue.com<http://www.shapeblue.com>
>> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> > > @shapeblue
>> > >
>> > >
>> > >
>> > >
>> >
>> > --
>> > Regards,
>> > Hean Seng
>> >
>> > Nicolas.Vazquez@shapeblue.com
>> > www.shapeblue.com<http://www.shapeblue.com>
>> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>>
>> --
>> Regards,
>> Hean Seng
>>
>> Nicolas.Vazquez@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: listUsageRecords Update frequency
Posted by Hean Seng <he...@gmail.com>.
Hi Nicolas,
I try ing to get VM to download something, and I notice the there is
nothing update to user_statistics table . Do you know which prcess is
suppose respobsible to update the user statistic table. I saw the
listvmmetrcs, and the data is is updated correctly , while there is nohting
update to user_statistics table.
On Wed, Nov 18, 2020 at 11:27 AM Nicolas Vazquez <
Nicolas.Vazquez@shapeblue.com> wrote:
> Hi Hean,
>
> Apologies, in this case it looks like data is obtained from
> user_statistics table as Andrija mentioned in the other thread.
>
>
> Regards,
>
> Nicolas Vazquez
>
> ________________________________
> From: Hean Seng <he...@gmail.com>
> Sent: Friday, November 13, 2020 3:24 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: listUsageRecords Update frequency
>
> is that any parameter that In network offering etc that do not let it count
> the data sent and receive ?
>
>
> Nicolas.Vazquez@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
> On Sat, Nov 14, 2020 at 2:22 AM Hean Seng <he...@gmail.com> wrote:
>
> > Hi Nicolas,
> >
> > i checked both, the data is tally , it has following type in
> usage_event:
> >
> > | NET.IPASSIGN |
> >
> > | VOLUME.CREATE |
> >
> > | VM.CREATE |
> >
> > | NETWORK.OFFERING.ASSIGN |
> >
> > | VM.START |
> >
> > | SG.ASSIGN |
> >
> > | VMSNAPSHOT.CREATE |
> >
> > | VMSNAPSHOT.ON_PRIMARY
> >
> >
> >
> > I check the ListUsageRecord , it has usage type of 3, 6, 13 m , just
> > do not have type of type4 and 5 which for network bytes
> >
> >
> >
> > On Tue, Nov 3, 2020 at 10:03 PM Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> >> Hi Hean,
> >>
> >> Looks like no data is being processed at all, the usage server takes the
> >> raw data from usage_event table and parses it into helper tables (such
> as
> >> usage_network). The data from the usage_event at the cloud_usage
> database
> >> is copied from the usage_event table on the cloud database. Can you
> check
> >> the records from those tables and see if there is a discrepancy? Also,
> can
> >> you verify if the usage db.properties is configured properly?
> >>
> >>
> >> Regards,
> >>
> >> Nicolas Vazquez
> >>
> >> ________________________________
> >> From: Hean Seng <he...@gmail.com>
> >> Sent: Saturday, October 31, 2020 5:41 AM
> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> Subject: Re: listUsageRecords Update frequency
> >>
> >> I think selec the wrong id, but still is return empty set:
> >>
> >>
> >> mysql> select * from usage_network where account_id=59;
> >>
> >> Empty set (0.00 sec)
> >>
> >> root@mgmt01 usage]# more usage.log | grep "accountId: 59"
> >>
> >> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59
> in
> >> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
> >> abr:
> >> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
> >>
> >> 2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl]
> >> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> >> accountId:
> >> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
> >>
> >> 2020-10-31 02
> >>
> >>
> >>
> >>
> >> Nicolas.Vazquez@shapeblue.com
> >> www.shapeblue.com<http://www.shapeblue.com>
> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> >> @shapeblue
> >>
> >>
> >>
> >> On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <he...@gmail.com> wrote:
> >>
> >> > Hi Nicolas
> >> >
> >> > The cloudstacl_usage is running . confirm from log
> >> >
> >> > 20-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
> >> > (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days,
> will
> >> > send statement in next cycle
> >> >
> >> > 2020-10-30 18:45:11,107 INFO [cloud.usage.UsageManagerImpl]
> >> > (Usage-Job-1:null) (logid:) usage job complete
> >> >
> >> > ^C
> >> >
> >> > It looks like it did not write to usage table as you say:
> >> >
> >> > mysql> select * from usage_network where
> >> > account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ;
> >> >
> >> > Empty set, 1 warning (0.00 sec)
> >> >
> >> >
> >> >
> >> >
> >> > Where is process to update the usage helper?
> >> >
> >> >
> >> >
> >> >
> >> > On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez <
> >> > Nicolas.Vazquez@shapeblue.com> wrote:
> >> >
> >> >> Hi Hean,
> >> >>
> >> >> The listVirtualMachineMetrics show the "current" usage of the virtual
> >> >> machine, and the usage records display the aggregated information
> >> (hourly,
> >> >> daily) per account. If the records are not updating can you please
> >> check
> >> >> the usage server logs if there is any error when the usage jobs run?
> >> Can
> >> >> you also check if data is being added on the helper table for network
> >> usage
> >> >> (usage_network table)? The information from this table is summarized
> >> into
> >> >> the resulting usage records when the usage jobs run.
> >> >>
> >> >>
> >> >> Regards,
> >> >>
> >> >> Nicolas Vazquez
> >> >>
> >> >> ________________________________
> >> >> From: Hean Seng <he...@gmail.com>
> >> >> Sent: Friday, October 30, 2020 4:18 AM
> >> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> >> Subject: Re: listUsageRecords Update frequency
> >> >>
> >> >> Hi Nicolas
> >> >>
> >> >> Thanks for answering.
> >> >>
> >> >> Some questions puzzle me, what is different between listUsageRecords
> >> and
> >> >> listVirtualMachineMetrics , in terms of Network Read and Write data.
> >> >> (other
> >> >> then one is per account basis, and another one is per VM basis) ,
> the
> >> >> data
> >> >> in listVirtualMachineMetrics almost instant update .
> >> >>
> >> >> Is listUsageReocrds (type 4 and type 5) only keep metric with Public
> >> IP ,
> >> >> and not calculated the SharedNetwork ?
> >> >>
> >> >> I facing issue of AdvanceNetwork withSG, and
> listVirtualMachineMetrics
> >> >> network sent and receive update almost instant, and listUsageRecords
> >> never
> >> >> update , I do not know where to check if there is error when updating
> >> this
> >> >> Account. Or if the account have setting/options to choose if process
> >> >> usage
> >> >> by Cloudstack-Usage ?
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez <
> >> >> Nicolas.Vazquez@shapeblue.com> wrote:
> >> >>
> >> >> > Yes, the usage jobs information is stored in the cloud_usage
> >> database on
> >> >> > the usage_job table.
> >> >> >
> >> >> > The start_date and end_date columns indicate which usage range was
> >> >> parsed
> >> >> > and the start_millis and end_millis the jobs execution start and
> end
> >> >> time
> >> >> > in milliseconds, which can be converted to date.
> >> >> >
> >> >> >
> >> >> > Regards,
> >> >> >
> >> >> > Nicolas Vazquez
> >> >> >
> >> >> > ________________________________
> >> >> > From: Hean Seng <he...@gmail.com>
> >> >> > Sent: Thursday, October 29, 2020 4:58 PM
> >> >> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> >> > Subject: Re: listUsageRecords Update frequency
> >> >> >
> >> >> > Hi NIC
> >> >> >
> >> >> > Do know where can confirm the last run for usage server ?
> >> >> >
> >> >> > It seems my counter is not running , my setup is AdvanceZone
> Security
> >> >> > Group.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez <
> >> >> > Nicolas.Vazquez@shapeblue.com> wrote:
> >> >> >
> >> >> > > Hi Hean,
> >> >> > >
> >> >> > > You are looking at the right configuration, if you set
> >> >> > > 'usage.stats.job.aggregation.range' = 60 then you only need to
> >> restart
> >> >> > the
> >> >> > > usage server to propagate the changes.
> >> >> > >
> >> >> > > Also please check the 'usage.stats.job.exec.time' configuration,
> >> you
> >> >> can
> >> >> > > set that exec time to a previous time in the day so after the
> usage
> >> >> > server
> >> >> > > restarts, the usage jobs start running, and then the interval
> >> remains
> >> >> > > hourly (by the previous configuration)
> >> >> > >
> >> >> > >
> >> >> > > Regards,
> >> >> > >
> >> >> > > Nicolas Vazquez
> >> >> > >
> >> >> > > ________________________________
> >> >> > > From: Hean Seng <he...@gmail.com>
> >> >> > > Sent: Wednesday, October 28, 2020 3:10 PM
> >> >> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> >> > > Subject: listUsageRecords Update frequency
> >> >> > >
> >> >> > > Hi
> >> >> > >
> >> >> > > I notice the usage record update once in a day .
> >> >> > >
> >> >> > > Is that possible to update the Usage data more frequently? May be
> >> >> hourly
> >> >> > > etc.
> >> >> > >
> >> >> > > I try to update usage.stats.job.aggregation.range but seem that
> is
> >> >> not I
> >> >> > > am looking for, it only affect startdate and enddate of the
> >> record,
> >> >> but
> >> >> > > not update usage record interval.
> >> >> > >
> >> >> > >
> >> >> > > --
> >> >> > > Regards,
> >> >> > > Hean Seng
> >> >> > >
> >> >> > > Nicolas.Vazquez@shapeblue.com
> >> >> > > www.shapeblue.com<http://www.shapeblue.com>
> >> >> > > 3 London Bridge Street, 3rd floor, News Building, London SE1
> >> 9SGUK
> >> >> > > @shapeblue
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> >
> >> >> > --
> >> >> > Regards,
> >> >> > Hean Seng
> >> >> >
> >> >> > Nicolas.Vazquez@shapeblue.com
> >> >> > www.shapeblue.com<http://www.shapeblue.com>
> >> >> > 3 London Bridge Street, 3rd floor, News Building, London SE1
> 9SGUK
> >> >> > @shapeblue
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >>
> >> >> --
> >> >> Regards,
> >> >> Hean Seng
> >> >>
> >> >> Nicolas.Vazquez@shapeblue.com
> >> >> www.shapeblue.com<http://www.shapeblue.com>
> >> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> >> >> @shapeblue
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >> > --
> >> > Regards,
> >> > Hean Seng
> >> >
> >>
> >>
> >> --
> >> Regards,
> >> Hean Seng
> >>
> >
> >
> > --
> > Regards,
> > Hean Seng
> >
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: listUsageRecords Update frequency
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Hi Hean,
Apologies, in this case it looks like data is obtained from user_statistics table as Andrija mentioned in the other thread.
Regards,
Nicolas Vazquez
________________________________
From: Hean Seng <he...@gmail.com>
Sent: Friday, November 13, 2020 3:24 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: listUsageRecords Update frequency
is that any parameter that In network offering etc that do not let it count
the data sent and receive ?
Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
@shapeblue
On Sat, Nov 14, 2020 at 2:22 AM Hean Seng <he...@gmail.com> wrote:
> Hi Nicolas,
>
> i checked both, the data is tally , it has following type in usage_event:
>
> | NET.IPASSIGN |
>
> | VOLUME.CREATE |
>
> | VM.CREATE |
>
> | NETWORK.OFFERING.ASSIGN |
>
> | VM.START |
>
> | SG.ASSIGN |
>
> | VMSNAPSHOT.CREATE |
>
> | VMSNAPSHOT.ON_PRIMARY
>
>
>
> I check the ListUsageRecord , it has usage type of 3, 6, 13 m , just
> do not have type of type4 and 5 which for network bytes
>
>
>
> On Tue, Nov 3, 2020 at 10:03 PM Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
>> Hi Hean,
>>
>> Looks like no data is being processed at all, the usage server takes the
>> raw data from usage_event table and parses it into helper tables (such as
>> usage_network). The data from the usage_event at the cloud_usage database
>> is copied from the usage_event table on the cloud database. Can you check
>> the records from those tables and see if there is a discrepancy? Also, can
>> you verify if the usage db.properties is configured properly?
>>
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>> ________________________________
>> From: Hean Seng <he...@gmail.com>
>> Sent: Saturday, October 31, 2020 5:41 AM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Re: listUsageRecords Update frequency
>>
>> I think selec the wrong id, but still is return empty set:
>>
>>
>> mysql> select * from usage_network where account_id=59;
>>
>> Empty set (0.00 sec)
>>
>> root@mgmt01 usage]# more usage.log | grep "accountId: 59"
>>
>> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 02
>>
>>
>>
>>
>> Nicolas.Vazquez@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>> On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <he...@gmail.com> wrote:
>>
>> > Hi Nicolas
>> >
>> > The cloudstacl_usage is running . confirm from log
>> >
>> > 20-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,107 INFO [cloud.usage.UsageManagerImpl]
>> > (Usage-Job-1:null) (logid:) usage job complete
>> >
>> > ^C
>> >
>> > It looks like it did not write to usage table as you say:
>> >
>> > mysql> select * from usage_network where
>> > account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ;
>> >
>> > Empty set, 1 warning (0.00 sec)
>> >
>> >
>> >
>> >
>> > Where is process to update the usage helper?
>> >
>> >
>> >
>> >
>> > On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez <
>> > Nicolas.Vazquez@shapeblue.com> wrote:
>> >
>> >> Hi Hean,
>> >>
>> >> The listVirtualMachineMetrics show the "current" usage of the virtual
>> >> machine, and the usage records display the aggregated information
>> (hourly,
>> >> daily) per account. If the records are not updating can you please
>> check
>> >> the usage server logs if there is any error when the usage jobs run?
>> Can
>> >> you also check if data is being added on the helper table for network
>> usage
>> >> (usage_network table)? The information from this table is summarized
>> into
>> >> the resulting usage records when the usage jobs run.
>> >>
>> >>
>> >> Regards,
>> >>
>> >> Nicolas Vazquez
>> >>
>> >> ________________________________
>> >> From: Hean Seng <he...@gmail.com>
>> >> Sent: Friday, October 30, 2020 4:18 AM
>> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >> Subject: Re: listUsageRecords Update frequency
>> >>
>> >> Hi Nicolas
>> >>
>> >> Thanks for answering.
>> >>
>> >> Some questions puzzle me, what is different between listUsageRecords
>> and
>> >> listVirtualMachineMetrics , in terms of Network Read and Write data.
>> >> (other
>> >> then one is per account basis, and another one is per VM basis) , the
>> >> data
>> >> in listVirtualMachineMetrics almost instant update .
>> >>
>> >> Is listUsageReocrds (type 4 and type 5) only keep metric with Public
>> IP ,
>> >> and not calculated the SharedNetwork ?
>> >>
>> >> I facing issue of AdvanceNetwork withSG, and listVirtualMachineMetrics
>> >> network sent and receive update almost instant, and listUsageRecords
>> never
>> >> update , I do not know where to check if there is error when updating
>> this
>> >> Account. Or if the account have setting/options to choose if process
>> >> usage
>> >> by Cloudstack-Usage ?
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez <
>> >> Nicolas.Vazquez@shapeblue.com> wrote:
>> >>
>> >> > Yes, the usage jobs information is stored in the cloud_usage
>> database on
>> >> > the usage_job table.
>> >> >
>> >> > The start_date and end_date columns indicate which usage range was
>> >> parsed
>> >> > and the start_millis and end_millis the jobs execution start and end
>> >> time
>> >> > in milliseconds, which can be converted to date.
>> >> >
>> >> >
>> >> > Regards,
>> >> >
>> >> > Nicolas Vazquez
>> >> >
>> >> > ________________________________
>> >> > From: Hean Seng <he...@gmail.com>
>> >> > Sent: Thursday, October 29, 2020 4:58 PM
>> >> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >> > Subject: Re: listUsageRecords Update frequency
>> >> >
>> >> > Hi NIC
>> >> >
>> >> > Do know where can confirm the last run for usage server ?
>> >> >
>> >> > It seems my counter is not running , my setup is AdvanceZone Security
>> >> > Group.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez <
>> >> > Nicolas.Vazquez@shapeblue.com> wrote:
>> >> >
>> >> > > Hi Hean,
>> >> > >
>> >> > > You are looking at the right configuration, if you set
>> >> > > 'usage.stats.job.aggregation.range' = 60 then you only need to
>> restart
>> >> > the
>> >> > > usage server to propagate the changes.
>> >> > >
>> >> > > Also please check the 'usage.stats.job.exec.time' configuration,
>> you
>> >> can
>> >> > > set that exec time to a previous time in the day so after the usage
>> >> > server
>> >> > > restarts, the usage jobs start running, and then the interval
>> remains
>> >> > > hourly (by the previous configuration)
>> >> > >
>> >> > >
>> >> > > Regards,
>> >> > >
>> >> > > Nicolas Vazquez
>> >> > >
>> >> > > ________________________________
>> >> > > From: Hean Seng <he...@gmail.com>
>> >> > > Sent: Wednesday, October 28, 2020 3:10 PM
>> >> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >> > > Subject: listUsageRecords Update frequency
>> >> > >
>> >> > > Hi
>> >> > >
>> >> > > I notice the usage record update once in a day .
>> >> > >
>> >> > > Is that possible to update the Usage data more frequently? May be
>> >> hourly
>> >> > > etc.
>> >> > >
>> >> > > I try to update usage.stats.job.aggregation.range but seem that is
>> >> not I
>> >> > > am looking for, it only affect startdate and enddate of the
>> record,
>> >> but
>> >> > > not update usage record interval.
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Regards,
>> >> > > Hean Seng
>> >> > >
>> >> > > Nicolas.Vazquez@shapeblue.com
>> >> > > www.shapeblue.com<http://www.shapeblue.com>
>> >> > > 3 London Bridge Street, 3rd floor, News Building, London SE1
>> 9SGUK
>> >> > > @shapeblue
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Hean Seng
>> >> >
>> >> > Nicolas.Vazquez@shapeblue.com
>> >> > www.shapeblue.com<http://www.shapeblue.com>
>> >> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> >> > @shapeblue
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >> --
>> >> Regards,
>> >> Hean Seng
>> >>
>> >> Nicolas.Vazquez@shapeblue.com
>> >> www.shapeblue.com<http://www.shapeblue.com>
>> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> >> @shapeblue
>> >>
>> >>
>> >>
>> >>
>> >
>> > --
>> > Regards,
>> > Hean Seng
>> >
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: listUsageRecords Update frequency
Posted by Hean Seng <he...@gmail.com>.
is that any parameter that In network offering etc that do not let it count
the data sent and receive ?
On Sat, Nov 14, 2020 at 2:22 AM Hean Seng <he...@gmail.com> wrote:
> Hi Nicolas,
>
> i checked both, the data is tally , it has following type in usage_event:
>
> | NET.IPASSIGN |
>
> | VOLUME.CREATE |
>
> | VM.CREATE |
>
> | NETWORK.OFFERING.ASSIGN |
>
> | VM.START |
>
> | SG.ASSIGN |
>
> | VMSNAPSHOT.CREATE |
>
> | VMSNAPSHOT.ON_PRIMARY
>
>
>
> I check the ListUsageRecord , it has usage type of 3, 6, 13 m , just
> do not have type of type4 and 5 which for network bytes
>
>
>
> On Tue, Nov 3, 2020 at 10:03 PM Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
>> Hi Hean,
>>
>> Looks like no data is being processed at all, the usage server takes the
>> raw data from usage_event table and parses it into helper tables (such as
>> usage_network). The data from the usage_event at the cloud_usage database
>> is copied from the usage_event table on the cloud database. Can you check
>> the records from those tables and see if there is a discrepancy? Also, can
>> you verify if the usage db.properties is configured properly?
>>
>>
>> Regards,
>>
>> Nicolas Vazquez
>>
>> ________________________________
>> From: Hean Seng <he...@gmail.com>
>> Sent: Saturday, October 31, 2020 5:41 AM
>> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> Subject: Re: listUsageRecords Update frequency
>>
>> I think selec the wrong id, but still is return empty set:
>>
>>
>> mysql> select * from usage_network where account_id=59;
>>
>> Empty set (0.00 sec)
>>
>> root@mgmt01 usage]# more usage.log | grep "accountId: 59"
>>
>> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
>> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0;
>> abr:
>> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>>
>> 2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl]
>> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
>> accountId:
>> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>>
>> 2020-10-31 02
>>
>>
>>
>>
>> Nicolas.Vazquez@shapeblue.com
>> www.shapeblue.com
>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> @shapeblue
>>
>>
>>
>> On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <he...@gmail.com> wrote:
>>
>> > Hi Nicolas
>> >
>> > The cloudstacl_usage is running . confirm from log
>> >
>> > 20-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
>> > (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, will
>> > send statement in next cycle
>> >
>> > 2020-10-30 18:45:11,107 INFO [cloud.usage.UsageManagerImpl]
>> > (Usage-Job-1:null) (logid:) usage job complete
>> >
>> > ^C
>> >
>> > It looks like it did not write to usage table as you say:
>> >
>> > mysql> select * from usage_network where
>> > account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ;
>> >
>> > Empty set, 1 warning (0.00 sec)
>> >
>> >
>> >
>> >
>> > Where is process to update the usage helper?
>> >
>> >
>> >
>> >
>> > On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez <
>> > Nicolas.Vazquez@shapeblue.com> wrote:
>> >
>> >> Hi Hean,
>> >>
>> >> The listVirtualMachineMetrics show the "current" usage of the virtual
>> >> machine, and the usage records display the aggregated information
>> (hourly,
>> >> daily) per account. If the records are not updating can you please
>> check
>> >> the usage server logs if there is any error when the usage jobs run?
>> Can
>> >> you also check if data is being added on the helper table for network
>> usage
>> >> (usage_network table)? The information from this table is summarized
>> into
>> >> the resulting usage records when the usage jobs run.
>> >>
>> >>
>> >> Regards,
>> >>
>> >> Nicolas Vazquez
>> >>
>> >> ________________________________
>> >> From: Hean Seng <he...@gmail.com>
>> >> Sent: Friday, October 30, 2020 4:18 AM
>> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >> Subject: Re: listUsageRecords Update frequency
>> >>
>> >> Hi Nicolas
>> >>
>> >> Thanks for answering.
>> >>
>> >> Some questions puzzle me, what is different between listUsageRecords
>> and
>> >> listVirtualMachineMetrics , in terms of Network Read and Write data.
>> >> (other
>> >> then one is per account basis, and another one is per VM basis) , the
>> >> data
>> >> in listVirtualMachineMetrics almost instant update .
>> >>
>> >> Is listUsageReocrds (type 4 and type 5) only keep metric with Public
>> IP ,
>> >> and not calculated the SharedNetwork ?
>> >>
>> >> I facing issue of AdvanceNetwork withSG, and listVirtualMachineMetrics
>> >> network sent and receive update almost instant, and listUsageRecords
>> never
>> >> update , I do not know where to check if there is error when updating
>> this
>> >> Account. Or if the account have setting/options to choose if process
>> >> usage
>> >> by Cloudstack-Usage ?
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez <
>> >> Nicolas.Vazquez@shapeblue.com> wrote:
>> >>
>> >> > Yes, the usage jobs information is stored in the cloud_usage
>> database on
>> >> > the usage_job table.
>> >> >
>> >> > The start_date and end_date columns indicate which usage range was
>> >> parsed
>> >> > and the start_millis and end_millis the jobs execution start and end
>> >> time
>> >> > in milliseconds, which can be converted to date.
>> >> >
>> >> >
>> >> > Regards,
>> >> >
>> >> > Nicolas Vazquez
>> >> >
>> >> > ________________________________
>> >> > From: Hean Seng <he...@gmail.com>
>> >> > Sent: Thursday, October 29, 2020 4:58 PM
>> >> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >> > Subject: Re: listUsageRecords Update frequency
>> >> >
>> >> > Hi NIC
>> >> >
>> >> > Do know where can confirm the last run for usage server ?
>> >> >
>> >> > It seems my counter is not running , my setup is AdvanceZone Security
>> >> > Group.
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez <
>> >> > Nicolas.Vazquez@shapeblue.com> wrote:
>> >> >
>> >> > > Hi Hean,
>> >> > >
>> >> > > You are looking at the right configuration, if you set
>> >> > > 'usage.stats.job.aggregation.range' = 60 then you only need to
>> restart
>> >> > the
>> >> > > usage server to propagate the changes.
>> >> > >
>> >> > > Also please check the 'usage.stats.job.exec.time' configuration,
>> you
>> >> can
>> >> > > set that exec time to a previous time in the day so after the usage
>> >> > server
>> >> > > restarts, the usage jobs start running, and then the interval
>> remains
>> >> > > hourly (by the previous configuration)
>> >> > >
>> >> > >
>> >> > > Regards,
>> >> > >
>> >> > > Nicolas Vazquez
>> >> > >
>> >> > > ________________________________
>> >> > > From: Hean Seng <he...@gmail.com>
>> >> > > Sent: Wednesday, October 28, 2020 3:10 PM
>> >> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
>> >> > > Subject: listUsageRecords Update frequency
>> >> > >
>> >> > > Hi
>> >> > >
>> >> > > I notice the usage record update once in a day .
>> >> > >
>> >> > > Is that possible to update the Usage data more frequently? May be
>> >> hourly
>> >> > > etc.
>> >> > >
>> >> > > I try to update usage.stats.job.aggregation.range but seem that is
>> >> not I
>> >> > > am looking for, it only affect startdate and enddate of the
>> record,
>> >> but
>> >> > > not update usage record interval.
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Regards,
>> >> > > Hean Seng
>> >> > >
>> >> > > Nicolas.Vazquez@shapeblue.com
>> >> > > www.shapeblue.com<http://www.shapeblue.com>
>> >> > > 3 London Bridge Street, 3rd floor, News Building, London SE1
>> 9SGUK
>> >> > > @shapeblue
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Hean Seng
>> >> >
>> >> > Nicolas.Vazquez@shapeblue.com
>> >> > www.shapeblue.com<http://www.shapeblue.com>
>> >> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> >> > @shapeblue
>> >> >
>> >> >
>> >> >
>> >> >
>> >>
>> >> --
>> >> Regards,
>> >> Hean Seng
>> >>
>> >> Nicolas.Vazquez@shapeblue.com
>> >> www.shapeblue.com<http://www.shapeblue.com>
>> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
>> >> @shapeblue
>> >>
>> >>
>> >>
>> >>
>> >
>> > --
>> > Regards,
>> > Hean Seng
>> >
>>
>>
>> --
>> Regards,
>> Hean Seng
>>
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng
Re: listUsageRecords Update frequency
Posted by Hean Seng <he...@gmail.com>.
Hi Nicolas,
i checked both, the data is tally , it has following type in usage_event:
| NET.IPASSIGN |
| VOLUME.CREATE |
| VM.CREATE |
| NETWORK.OFFERING.ASSIGN |
| VM.START |
| SG.ASSIGN |
| VMSNAPSHOT.CREATE |
| VMSNAPSHOT.ON_PRIMARY
I check the ListUsageRecord , it has usage type of 3, 6, 13 m , just do
not have type of type4 and 5 which for network bytes
On Tue, Nov 3, 2020 at 10:03 PM Nicolas Vazquez <
Nicolas.Vazquez@shapeblue.com> wrote:
> Hi Hean,
>
> Looks like no data is being processed at all, the usage server takes the
> raw data from usage_event table and parses it into helper tables (such as
> usage_network). The data from the usage_event at the cloud_usage database
> is copied from the usage_event table on the cloud database. Can you check
> the records from those tables and see if there is a discrepancy? Also, can
> you verify if the usage db.properties is configured properly?
>
>
> Regards,
>
> Nicolas Vazquez
>
> ________________________________
> From: Hean Seng <he...@gmail.com>
> Sent: Saturday, October 31, 2020 5:41 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: listUsageRecords Update frequency
>
> I think selec the wrong id, but still is return empty set:
>
>
> mysql> select * from usage_network where account_id=59;
>
> Empty set (0.00 sec)
>
> root@mgmt01 usage]# more usage.log | grep "accountId: 59"
>
> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:00:00,945 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:15:00,780 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:30:00,972 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 00:45:00,973 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:00:00,837 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:15:00,809 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:30:00,983 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 01:45:00,799 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) creating vmDiskHelperEntry... accountId: 59 in
> zone: 7; aiw: 0; air: 0; curAIR: 0; curAIW: 0; uir: 0; uiw: 0; abw: 0; abr:
> 0; curABR: 0; curABW: 0; ubr: 0; ubw: 0
>
> 2020-10-31 02:00:00,854 DEBUG [cloud.usage.UsageManagerImpl]
> (Usage-Job-1:null) (logid:) getting current accounted bytes for...
> accountId:
> 59 in zone: 7; aiw: 0; air: 0; abw: 0; abr: 0
>
> 2020-10-31 02
>
>
>
>
> Nicolas.Vazquez@shapeblue.com
> www.shapeblue.com
> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> @shapeblue
>
>
>
> On Sat, Oct 31, 2020 at 2:57 AM Hean Seng <he...@gmail.com> wrote:
>
> > Hi Nicolas
> >
> > The cloudstacl_usage is running . confirm from log
> >
> > 20-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 5 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 10 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 11 it is already more than 59 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 12 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 13 it is already more than 58 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,016 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 14 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 23 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,017 INFO [cloudstack.quota.QuotaStatementImpl]
> > (Usage-Job-1:null) (logid:) For 24 it is already more than 29 days, will
> > send statement in next cycle
> >
> > 2020-10-30 18:45:11,107 INFO [cloud.usage.UsageManagerImpl]
> > (Usage-Job-1:null) (logid:) usage job complete
> >
> > ^C
> >
> > It looks like it did not write to usage table as you say:
> >
> > mysql> select * from usage_network where
> > account_id="a40e9892-3051-41b6-baf2-fabe3af06c92" ;
> >
> > Empty set, 1 warning (0.00 sec)
> >
> >
> >
> >
> > Where is process to update the usage helper?
> >
> >
> >
> >
> > On Sat, Oct 31, 2020 at 1:36 AM Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> >> Hi Hean,
> >>
> >> The listVirtualMachineMetrics show the "current" usage of the virtual
> >> machine, and the usage records display the aggregated information
> (hourly,
> >> daily) per account. If the records are not updating can you please check
> >> the usage server logs if there is any error when the usage jobs run? Can
> >> you also check if data is being added on the helper table for network
> usage
> >> (usage_network table)? The information from this table is summarized
> into
> >> the resulting usage records when the usage jobs run.
> >>
> >>
> >> Regards,
> >>
> >> Nicolas Vazquez
> >>
> >> ________________________________
> >> From: Hean Seng <he...@gmail.com>
> >> Sent: Friday, October 30, 2020 4:18 AM
> >> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> Subject: Re: listUsageRecords Update frequency
> >>
> >> Hi Nicolas
> >>
> >> Thanks for answering.
> >>
> >> Some questions puzzle me, what is different between listUsageRecords
> and
> >> listVirtualMachineMetrics , in terms of Network Read and Write data.
> >> (other
> >> then one is per account basis, and another one is per VM basis) , the
> >> data
> >> in listVirtualMachineMetrics almost instant update .
> >>
> >> Is listUsageReocrds (type 4 and type 5) only keep metric with Public IP
> ,
> >> and not calculated the SharedNetwork ?
> >>
> >> I facing issue of AdvanceNetwork withSG, and listVirtualMachineMetrics
> >> network sent and receive update almost instant, and listUsageRecords
> never
> >> update , I do not know where to check if there is error when updating
> this
> >> Account. Or if the account have setting/options to choose if process
> >> usage
> >> by Cloudstack-Usage ?
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Oct 30, 2020 at 5:55 AM Nicolas Vazquez <
> >> Nicolas.Vazquez@shapeblue.com> wrote:
> >>
> >> > Yes, the usage jobs information is stored in the cloud_usage database
> on
> >> > the usage_job table.
> >> >
> >> > The start_date and end_date columns indicate which usage range was
> >> parsed
> >> > and the start_millis and end_millis the jobs execution start and end
> >> time
> >> > in milliseconds, which can be converted to date.
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Nicolas Vazquez
> >> >
> >> > ________________________________
> >> > From: Hean Seng <he...@gmail.com>
> >> > Sent: Thursday, October 29, 2020 4:58 PM
> >> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> > Subject: Re: listUsageRecords Update frequency
> >> >
> >> > Hi NIC
> >> >
> >> > Do know where can confirm the last run for usage server ?
> >> >
> >> > It seems my counter is not running , my setup is AdvanceZone Security
> >> > Group.
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Oct 29, 2020 at 4:34 AM Nicolas Vazquez <
> >> > Nicolas.Vazquez@shapeblue.com> wrote:
> >> >
> >> > > Hi Hean,
> >> > >
> >> > > You are looking at the right configuration, if you set
> >> > > 'usage.stats.job.aggregation.range' = 60 then you only need to
> restart
> >> > the
> >> > > usage server to propagate the changes.
> >> > >
> >> > > Also please check the 'usage.stats.job.exec.time' configuration, you
> >> can
> >> > > set that exec time to a previous time in the day so after the usage
> >> > server
> >> > > restarts, the usage jobs start running, and then the interval
> remains
> >> > > hourly (by the previous configuration)
> >> > >
> >> > >
> >> > > Regards,
> >> > >
> >> > > Nicolas Vazquez
> >> > >
> >> > > ________________________________
> >> > > From: Hean Seng <he...@gmail.com>
> >> > > Sent: Wednesday, October 28, 2020 3:10 PM
> >> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> >> > > Subject: listUsageRecords Update frequency
> >> > >
> >> > > Hi
> >> > >
> >> > > I notice the usage record update once in a day .
> >> > >
> >> > > Is that possible to update the Usage data more frequently? May be
> >> hourly
> >> > > etc.
> >> > >
> >> > > I try to update usage.stats.job.aggregation.range but seem that is
> >> not I
> >> > > am looking for, it only affect startdate and enddate of the record,
> >> but
> >> > > not update usage record interval.
> >> > >
> >> > >
> >> > > --
> >> > > Regards,
> >> > > Hean Seng
> >> > >
> >> > > Nicolas.Vazquez@shapeblue.com
> >> > > www.shapeblue.com<http://www.shapeblue.com>
> >> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> >> > > @shapeblue
> >> > >
> >> > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > Regards,
> >> > Hean Seng
> >> >
> >> > Nicolas.Vazquez@shapeblue.com
> >> > www.shapeblue.com<http://www.shapeblue.com>
> >> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> >> > @shapeblue
> >> >
> >> >
> >> >
> >> >
> >>
> >> --
> >> Regards,
> >> Hean Seng
> >>
> >> Nicolas.Vazquez@shapeblue.com
> >> www.shapeblue.com<http://www.shapeblue.com>
> >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK
> >> @shapeblue
> >>
> >>
> >>
> >>
> >
> > --
> > Regards,
> > Hean Seng
> >
>
>
> --
> Regards,
> Hean Seng
>
--
Regards,
Hean Seng