You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Vivek Kumar <vi...@indiqus.com.INVALID> on 2022/07/10 06:50:09 UTC
Re: Usage is not working | ACS 4.15
Hello Folks,
Can someone please help me on this ?
Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies
vivek.kumar@indiqus.com <ma...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>
> On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vi...@indiqus.com> wrote:
>
> Hello Guys,
>
> I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We have recently observed that cloud_usage is not being generated since the moment we had upgrade. Here is the details and logs from usage.log. Can someone help me out what else do we need to change ot check. We have restarted both management and usage service multiple times.
>
> 1- Global settings -
>
> mysql> select * from configuration where name like '%usage%' and component = 'management-server';
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> | category | instance | component | name | value | description | default_value | updated | scope | is_dynamic |
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> | Usage | DEFAUL. | management-server | enable.usage.server | true | Flag for enabling usage | true | NULL | NULL | 0 |
> | Advanced | DEFAULT | management-server | publish.usage.events | true | enable or disable publishing of usage events on the event bus | true | NULL | NULL | 0 |
> | Usage | DEFAULT | management-server | usage.aggregation.timezone | UTC-3 | The timezone to use for usage stats aggregation | GMT | NULL | NULL | 0 |
> | Usage | DEFAULT | management-server | usage.execution.timezone | UTC-3 | The timezone to use for usage job execution time | NULL | NULL | NULL | 0 |
> | Usage | DEFAULT | management-server | usage.sanity.check.interval | NULL | Interval (in days) to check sanity of usage data | NULL | NULL | NULL | 0 |
> | Usage | DEFAULT | management-server | usage.stats.job.aggregation.range | 60 | The range of time for aggregating the user statistics specified in minutes (e.g. 1440 for daily, 60 for hourly. | 1440 | NULL | NULL | 0 |
> | Usage | DEFAULT | management-server | usage.stats.job.exec.time | 00:15 | The time at which the usage statistics aggregation job will run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 | NULL | NULL | 0 |
> +----------+----------+-------------------+-----------------------------------+-------+—————————————————————————————————————————————————————————
>
> 2- Rpm installed -
>
>
>
> [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack"
> cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64
> cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64
> cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64
> cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64
> cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64
>
>
> 3- I am getting below logs in all usage.log -
>
>
> 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl] (main:null) (logid:) Checking to see if usage.vmops.pid exists.
> 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl] (main:null) (logid:) Implementation Version is 4.15.1.0
> 2022-03-29 13:21:45,059 DEBUG [utils.crypt.EncryptionSecretKeyChecker] (main:null) (logid:) Encryption Type: file
> 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null) (logid:) Is Data Base High Availiability enabled? Ans : true
> 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null) (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8
> 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null) (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
> 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext] (main:null) (logid:) Setup Spring Application context
> 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef
> 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9
> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855
> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263
> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306
> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699
> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910
> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734
> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab
> 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a
> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55
> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6
> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a
> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9
> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59
> 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb
> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877
> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c
> 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc
> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16
> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147
> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d
> 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5
> 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f
> 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f
> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42
> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86
> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da
> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e
> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e
> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e
> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6
> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8
> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24
> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6
> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495
> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717
> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be
> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89
> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8
> 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl] (main:null) (logid:) Checking to see if usage.vmops.pid exists.
> 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl] (main:null) (logid:) Implementation Version is 4.15.1.0
>
>
>
> Regards,
> Vivek Kumar
>
>
>
--
This message is intended only for the use of the individual or entity to
which it is addressed and may contain confidential and/or privileged
information. If you are not the intended recipient, please delete the
original message and any copy of it from your computer system. You are
hereby notified that any dissemination, distribution or copying of this
communication is strictly prohibited unless proper authorization has been
obtained for such action. If you have received this communication in error,
please notify the sender immediately. Although IndiQus attempts to sweep
e-mail and attachments for viruses, it does not guarantee that both are
virus-free and accepts no liability for any damage sustained as a result of
viruses.
Re: Usage is not working | ACS 4.15
Posted by Daan Hoogland <da...@gmail.com>.
this is another matter entirely Vivek. I'm guessing for each of those you
have jobs with the same start_date/end_date that are successful. only
<null> matters. A new job should be inserted when you start a new
usageserver. So that you have an empty set for that is the issue. The usage
server didn´t insert it's record (and can´t reach the DB) this is where you
should look for a remedy.
On Fri, Jul 15, 2022 at 6:05 AM Vivek Kumar <vi...@indiqus.com.invalid>
wrote:
> Hello Daan.
>
> So we have multiple entries where success is zero -
>
> mysql> select * from usage_job where success is null order by id desc
> limit 10;
> Empty set (0.00 sec)
>
> mysql> select * from usage_job where success = 0 order by id desc limit 10;
>
> +------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
> | id | host | pid | job_type | scheduled |
> start_millis | end_millis | exec_time | start_date | end_date
> | success | heartbeat |
>
> +------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
> | 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 |
> 2021-08-26 23:59:59 | 0 | 2021-08-27 02:52:15 |
> | 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 |
> 2021-08-26 23:59:59 | 0 | 2021-08-27 02:52:15 |
> | 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
> 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:26 |
> | 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
> 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:25 |
> | 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 |
> 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:25 |
> | 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 |
> 2021-08-24 23:59:59 | 0 | 2021-08-25 02:51:45 |
> | 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 |
> 2021-08-24 23:59:59 | 0 | 2021-08-25 02:51:45 |
> | 1993 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629763200000 | 1629763199999 | 2 | 2021-08-24 00:00:00 |
> 2021-08-23 23:59:59 | 0 | 2021-08-24 02:51:46 |
> | 1992 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629763200000 | 1629763199999 | 1 | 2021-08-24 00:00:00 |
> 2021-08-23 23:59:59 | 0 | 2021-08-24 02:51:46 |
> | 1990 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629676800000 | 1629676799999 | 1 | 2021-08-23 00:00:00 |
> 2021-08-22 23:59:59 | 0 | 2021-08-23 02:51:43 |
>
> +------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
> 10 rows in set (0.00 sec)
>
> ——
>
>
> select count(id) from usage_job where success = 0;
> +-----------+
> | count(id) |
> +-----------+
> | 999 |
> +-----------+
> 1 row in set (0.00 sec)
>
>
> Vivek Kumar
> Sr. Manager - Cloud & DevOps
> TechOps | Indiqus Technologies
>
> vivek.kumar@indiqus.com <ma...@indiqus.com>
> www.indiqus.com <https://www.indiqus.com/>
>
>
>
>
> > On 14-Jul-2022, at 3:03 PM, Daan Hoogland <da...@gmail.com>
> wrote:
> >
> > On Wed, Jul 13, 2022 at 3:51 AM Vivek Kumar <vivek.kumar@indiqus.com.invalid
> <ma...@indiqus.com.invalid>>
> > wrote:
> >
> >> Hello Daan,
> >>
> >> Thanks for the response.
> >>
> >> So we have deployed a replica of our management servers and databases on
> >> a complete isolated environment, Tried a work around which you have
> >> suggested but PID is not getting updated in the usage_job. Usage.logs
> gets
> >> stuck on below logs - We have also tried to upgrade on 4.16 on the
> replica
> >> setup but same situation with 4.16 as well.
> >>
> >>
> >> 2022-07-04 10:23:11,977 INFO [cloud.usage.UsageManagerImpl] (main:null)
> >> (logid:) Implementation Version is 4.16.1.0
> >> 2022-07-11 04:26:23,905 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
> >> (main:null) (logid:) Encryption Type: file
> >> 2022-07-11 04:26:23,927 INFO [db.Transaction.Transaction] (main:null)
> >> (logid:) Is Data Base High Availiability enabled? Ans : true
> >> 2022-07-11 04:26:23,981 INFO [db.Transaction.Transaction] (main:null)
> >> (logid:) The replicas configured for Cloud Data base is/are : 198.18.8.8
> >> 2022-07-11 04:26:24,014 DEBUG [utils.db.DriverLoader] (main:null)
> (logid:)
> >> Successfully loaded DB driver com.mysql.cj.jdbc.Driver
> >> 2022-07-11 04:26:24,063 DEBUG [utils.db.DriverLoader] (main:null)
> (logid:)
> >> DB driver com.mysql.cj.jdbc.Driver was already loaded.
> >> 2022-07-11 04:26:24,070 DEBUG [utils.db.DriverLoader] (main:null)
> (logid:)
> >> DB driver com.mysql.cj.jdbc.Driver was already loaded.
> >> 2022-07-11 04:26:25,172 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Setup Spring Application context
> >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_b5109665
> >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_42da784f
> >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_bf461acb
> >> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_a66084d9
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_35dea57c
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_7ec9190f
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_98c3bb86
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_a3d9d9aa
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_c06bb621
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_82eb64e0
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_95b19ecb
> >> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6458895c
> >> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_d545c0a0
> >> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_ef2ae44f
> >> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_39357dcf
> >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_421ab841
> >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_3d9daaed
> >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_a3dbe792
> >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_9f537032
> >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_5b9ccd8c
> >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_e67e53bd
> >> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_6e25b0b3
> >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_91eaad5
> >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_5bdd1db5
> >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_e36e4eb8
> >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_d51f0ffc
> >> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_ae698650
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_a22b67b4
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_42e796e4
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_84083314
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_fc57081c
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3532324e
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_bf4cdf9a
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_3c5a442c
> >> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_6e84e70b
> >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6150198d
> >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_a8a8c734
> >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_58bc9eff
> >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ad0cdf5e
> >> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_87f6b7d5
> >> 2022-07-11 04:26:25,625 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
> >> (logid:) Checking to see if usage.vmops.pid exists.
> >> 2022-07-11 04:26:25,625 INFO [cloud.usage.UsageManagerImpl] (main:null)
> >> (logid:) Implementation Version is 4.16.1.0
> >>
> >
> > I see no errors here.
> >
> >>
> >>
> >>
> >> — Also we have seen in the usage_job - there are multiple jobs which are
> >> showing as ‘zero’ in success, So my question is - Is last job has to be
> set
> >> to success true to all the jobs has to be success true ?
> >>
> >>
> >> mysql> select
> >>
> id,host,pid,job_type,scheduled,start_millis,end_millis,exec_time,start_date,end_date,success
> >> from usage_job order by id desc limit 10;
> >>
> >>
> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
> >> | id | host | pid | job_type | scheduled |
> >> start_millis | end_millis | exec_time | start_date | end_date
> >> | success |
> >>
> >>
> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
> >> | 2004 | acs-mgmt-02/198.18.8.6 | 30073 | 0 | 0 |
> >> 1630022400000 | 1630108799999 | 194785 | 2021-08-27 00:00:00 |
> >> 2021-08-27 23:59:59 | 1 |
> >> | 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 |
> >> 2021-08-26 23:59:59. | 0 |
> >> | 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 |
> >> 2021-08-26 23:59:59. | 0 |
> >> | 2001 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1629936000000 | 1630022399999 | 189715 | 2021-08-26 00:00:00 |
> >> 2021-08-26 23:59:59. | 1 |
> >> | 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
> >> 2021-08-25 23:59:59. | 0 |
> >> | 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
> >> 2021-08-25 23:59:59. | 0 |
> >> | 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 |
> >> 2021-08-25 23:59:59. | 0 |
> >> | 1997 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1629849600000 | 1629935999999 | 200641 | 2021-08-25 00:00:00 |
> >> 2021-08-25 23:59:59. | 1 |
> >> | 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 |
> >> 2021-08-24 23:59:59. | 0 |
> >> | 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> >> 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 |
> >> 2021-08-24 23:59:59. | 0 |
> >>
> >>
> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
> >>
> >
> > What I would expect, if the usage server is running is to see a reord
> where
> > none of start_millis, end_millis and success are filled. Your DB shows
> > multiple jobs with the same time stamps which is wrong. As if multiple
> > processes with the same PID are running, which should be impossible on
> any
> > *nis-like system.
> > A record like
> > |2005|acs-mgmt-02/198.18.8.6|30073|| 0| 0| 0| 0| 0|
> > [NULL]| [NULL] | [NULL]| <heartbeat-timestamp>|
> > is expected.
> > can you try
> >> select * from usage_job where success is null order by id desc limit 10;
> >
> >
> >>
> >> — I have also observed while checking the usage_service - It’s saying
> >> something related to mysql jdbc driver. However same driver is working
> >> everywhere -
> >>
> >>
> >> # rpm -qa | grep “connector"
> >>
> >> mysql-connector-python-1.1.6-1.el7.noarch
> >> mysql-connector-java-5.1.25-3.el7.noarch
> >>
> >> — usage service status
> >>
> >> [root@acs-mgmt-01 ~]# systemctl status cloudstack-usage
> >> ● cloudstack-usage.service - CloudStack Usage Server
> >> Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service;
> >> enabled; vendor preset: disabled)
> >> Active: active (running) since Mon 2022-07-11 04:26:20 -03; 4s ago
> >> Docs: http://www.cloudstack.org/
> >> Main PID: 1079 (java)
> >> Tasks: 24
> >> CGroup: /system.slice/cloudstack-usage.service
> >> └─1079 /usr/bin/java -Dpid=1079 -Xms256m -Xmx2048m -cp
> >>
> /usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/cloudstack-mysql-ha/lib/*:/etc/cl...
> >>
> >> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: Defaulting to no-operation
> >> (NOP) logger implementation
> >> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: See
> >> http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: An illegal reflective
> >> access operation has occurred
> >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Illegal reflective access
> >> by net.sf.cglib.core.ReflectUtils$1
> >> (file:/usr/share/cloudstack-usage/lib/cglib-nodep-3.3....tionDomain)
> >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Please consider reporting
> >> this to the maintainers of net.sf.cglib.core.ReflectUtils$1
> >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Use --illegal-access=warn
> >> to enable warnings of further illegal reflective access operations
> >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: All illegal access
> >> operations will be denied in a future release
> >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,927 INFO Transaction:1027
> >> - Is Data Base High Availiability enabled? Ans : true
> >> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,981 INFO Transaction:1070
> >> - The replicas configured for Cloud Data base is/are : 198.18.8.8
> >> Jul 11 04:26:24 acs-mgmt-01 sh[1079]: Loading class
> >> `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is
> >> `com.mysql.cj.jdbc.Driver'. The driver is ...nnecessary.
> >>
> >>
> >> — I am also concerned about the time zone, is there any specific time
> >> zones which needs to be placed in global setting - I have tried putting
> UTC
> >> and other near Time Zone as well but nothing worked.
> >>
> >> mysql> select name,value from configuration where name like '%usage%'
> and
> >> component = 'management-server';
> >> +-----------------------------------+----------------+
> >> | name | value |
> >> +-----------------------------------+----------------+
> >> | enable.usage.server | true |
> >> | publish.usage.events | true |
> >> | usage.aggregation.timezone | America/Cuiaba |
> >> | usage.execution.timezone | America/Cuiaba |
> >> | usage.sanity.check.interval | 1 |
> >> | usage.stats.job.aggregation.range | 60 |
> >> | usage.stats.job.exec.time | 00:10 |
> >> +-----------------------------------+----------------+
> >>
> >
> > Time zones need to be defined on your system to be used in mysql. you'll
> > have to look into your zones dir (see google for howto) 'System',
> > 'Europe/Amsterdam' or '+01:00' should work depending on your system
> > configuration.
> >
> >
> >
> >>
> >>
> >>
> >>
> >> Vivek Kumar
> >> Sr. Manager - Cloud & DevOps
> >> TechOps | Indiqus Technologies
> >>
> >> vivek.kumar@indiqus.com <ma...@indiqus.com> <mailto:
> vivek.kumar@indiqus.com <ma...@indiqus.com>>
> >> www.indiqus.com <http://www.indiqus.com/> <https://www.indiqus.com/ <
> https://www.indiqus.com/>>
> >>
> >>
> >>
> >>
> >>> On 11-Jul-2022, at 12:15 PM, Daan Hoogland <daan.hoogland@gmail.com
> <ma...@gmail.com>>
> >> wrote:
> >>>
> >>> vivek,
> >>> *take care with this and make a backup of the cloud_usage.usage_job
> >> first*
> >>> check in the usage_job table to see if there is a record for a not yet
> >>> completed job.
> >>> get the pid from that job and check if that is the same as the
> currently
> >>> running usage server.
> >>> If not, stop the usage server, remove the record and restart the usage
> >>> server.
> >>>
> >>> if the pids match you'll need to investigate further.
> >>>
> >>> On Sun, Jul 10, 2022 at 8:50 AM Vivek Kumar <vivek.kumar@indiqus.com
> >> .invalid>
> >>> wrote:
> >>>
> >>>> Hello Folks,
> >>>>
> >>>> Can someone please help me on this ?
> >>>>
> >>>>
> >>>> Vivek Kumar
> >>>> Sr. Manager - Cloud & DevOps
> >>>> TechOps | Indiqus Technologies
> >>>>
> >>>> vivek.kumar@indiqus.com <ma...@indiqus.com>
> >>>> www.indiqus.com <https://www.indiqus.com/>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vi...@indiqus.com>
> >> wrote:
> >>>>>
> >>>>> Hello Guys,
> >>>>>
> >>>>> I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We
> >>>> have recently observed that cloud_usage is not being generated since
> the
> >>>> moment we had upgrade. Here is the details and logs from usage.log.
> Can
> >>>> someone help me out what else do we need to change ot check. We have
> >>>> restarted both management and usage service multiple times.
> >>>>>
> >>>>> 1- Global settings -
> >>>>>
> >>>>> mysql> select * from configuration where name like '%usage%' and
> >>>> component = 'management-server';
> >>>>>
> >>>>
> >>
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> >>>>> | category | instance | component | name
> >>>> | value | description
> >>>> | default_value |
> >>>> updated | scope | is_dynamic |
> >>>>>
> >>>>
> >>
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> >>>>> | Usage | DEFAUL. | management-server | enable.usage.server
> >>>> | true | Flag for enabling usage
> >>>> | true |
> >>>> NULL | NULL | 0 |
> >>>>> | Advanced | DEFAULT | management-server | publish.usage.events
> >>>> | true | enable or disable publishing of usage events on the event
> >>>> bus | true
> >> |
> >>>> NULL | NULL | 0 |
> >>>>> | Usage | DEFAULT | management-server | usage.aggregation.timezone
> >>>> | UTC-3 | The timezone to use for usage stats aggregation
> >>>> | GMT |
> >>>> NULL | NULL | 0 |
> >>>>> | Usage | DEFAULT | management-server | usage.execution.timezone
> >>>> | UTC-3 | The timezone to use for usage job execution time
> >>>> | NULL |
> >>>> NULL | NULL | 0 |
> >>>>> | Usage | DEFAULT | management-server | usage.sanity.check.interval
> >>>> | NULL | Interval (in days) to check sanity of usage data
> >>>> | NULL |
> >>>> NULL | NULL | 0 |
> >>>>> | Usage | DEFAULT | management-server |
> >>>> usage.stats.job.aggregation.range | 60 | The range of time for
> >>>> aggregating the user statistics specified in minutes (e.g. 1440 for
> >> daily,
> >>>> 60 for hourly. | 1440 | NULL | NULL | 0 |
> >>>>> | Usage | DEFAULT | management-server | usage.stats.job.exec.time
> >>>> | 00:15 | The time at which the usage statistics aggregation job
> >> will
> >>>> run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 |
> >>>> NULL | NULL | 0 |
> >>>>>
> >>>>
> >>
> +----------+----------+-------------------+-----------------------------------+-------+—————————————————————————————————————————————————————————
> >>>>>
> >>>>> 2- Rpm installed -
> >>>>>
> >>>>>
> >>>>>
> >>>>> [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack"
> >>>>> cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64
> >>>>> cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64
> >>>>> cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64
> >>>>> cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64
> >>>>> cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64
> >>>>>
> >>>>>
> >>>>> 3- I am getting below logs in all usage.log -
> >>>>>
> >>>>>
> >>>>> 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> >>>>> 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl]
> >> (main:null)
> >>>> (logid:) Checking to see if usage.vmops.pid exists.
> >>>>> 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl]
> >> (main:null)
> >>>> (logid:) Implementation Version is 4.15.1.0
> >>>>> 2022-03-29 13:21:45,059 DEBUG
> [utils.crypt.EncryptionSecretKeyChecker]
> >>>> (main:null) (logid:) Encryption Type: file
> >>>>> 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null)
> >>>> (logid:) Is Data Base High Availiability enabled? Ans : true
> >>>>> 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null)
> >>>> (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8
> >>>>> 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null)
> >>>> (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
> >>>>> 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null)
> >>>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> >>>>> 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null)
> >>>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> >>>>> 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Setup Spring Application context
> >>>>> 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef
> >>>>> 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9
> >>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855
> >>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263
> >>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306
> >>>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699
> >>>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910
> >>>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734
> >>>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab
> >>>>> 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a
> >>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55
> >>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6
> >>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a
> >>>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9
> >>>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59
> >>>>> 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb
> >>>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877
> >>>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c
> >>>>> 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc
> >>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16
> >>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147
> >>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d
> >>>>> 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5
> >>>>> 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f
> >>>>> 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f
> >>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42
> >>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86
> >>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da
> >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e
> >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e
> >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e
> >>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6
> >>>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8
> >>>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24
> >>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6
> >>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495
> >>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717
> >>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be
> >>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>>
> >>
> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89
> >>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8
> >>>>> 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext]
> >>>> (main:null) (logid:) Configuring
> >>>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> >>>>> 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl]
> >> (main:null)
> >>>> (logid:) Checking to see if usage.vmops.pid exists.
> >>>>> 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl]
> >> (main:null)
> >>>> (logid:) Implementation Version is 4.15.1.0
> >>>>>
> >>>>>
> >>>>>
> >>>>> Regards,
> >>>>> Vivek Kumar
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> This message is intended only for the use of the individual or entity
> to
> >>>> which it is addressed and may contain confidential and/or privileged
> >>>> information. If you are not the intended recipient, please delete the
> >>>> original message and any copy of it from your computer system. You are
> >>>> hereby notified that any dissemination, distribution or copying of
> this
> >>>> communication is strictly prohibited unless proper authorization has
> >> been
> >>>> obtained for such action. If you have received this communication in
> >>>> error,
> >>>> please notify the sender immediately. Although IndiQus attempts to
> sweep
> >>>> e-mail and attachments for viruses, it does not guarantee that both
> are
> >>>> virus-free and accepts no liability for any damage sustained as a
> result
> >>>> of
> >>>> viruses.
> >>>>
> >>>
> >>>
> >>> --
> >>> Daan
> >>
> >>
> >> --
> >> This message is intended only for the use of the individual or entity to
> >> which it is addressed and may contain confidential and/or privileged
> >> information. If you are not the intended recipient, please delete the
> >> original message and any copy of it from your computer system. You are
> >> hereby notified that any dissemination, distribution or copying of this
> >> communication is strictly prohibited unless proper authorization has
> been
> >> obtained for such action. If you have received this communication in
> >> error,
> >> please notify the sender immediately. Although IndiQus attempts to sweep
> >> e-mail and attachments for viruses, it does not guarantee that both are
> >> virus-free and accepts no liability for any damage sustained as a result
> >> of
> >> viruses.
> >>
> >
> >
> > --
> > Daan
>
>
> --
> This message is intended only for the use of the individual or entity to
> which it is addressed and may contain confidential and/or privileged
> information. If you are not the intended recipient, please delete the
> original message and any copy of it from your computer system. You are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited unless proper authorization has been
> obtained for such action. If you have received this communication in
> error,
> please notify the sender immediately. Although IndiQus attempts to sweep
> e-mail and attachments for viruses, it does not guarantee that both are
> virus-free and accepts no liability for any damage sustained as a result
> of
> viruses.
>
--
Daan
Re: Usage is not working | ACS 4.15
Posted by Vivek Kumar <vi...@indiqus.com.INVALID>.
Hello Daan.
So we have multiple entries where success is zero -
mysql> select * from usage_job where success is null order by id desc limit 10;
Empty set (0.00 sec)
mysql> select * from usage_job where success = 0 order by id desc limit 10;
+------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| id | host | pid | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat |
+------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 | 2021-08-26 23:59:59 | 0 | 2021-08-27 02:52:15 |
| 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 | 2021-08-26 23:59:59 | 0 | 2021-08-27 02:52:15 |
| 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:26 |
| 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:25 |
| 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 | 2021-08-25 23:59:59 | 0 | 2021-08-26 02:52:25 |
| 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 | 2021-08-24 23:59:59 | 0 | 2021-08-25 02:51:45 |
| 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 | 2021-08-24 23:59:59 | 0 | 2021-08-25 02:51:45 |
| 1993 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629763200000 | 1629763199999 | 2 | 2021-08-24 00:00:00 | 2021-08-23 23:59:59 | 0 | 2021-08-24 02:51:46 |
| 1992 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629763200000 | 1629763199999 | 1 | 2021-08-24 00:00:00 | 2021-08-23 23:59:59 | 0 | 2021-08-24 02:51:46 |
| 1990 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629676800000 | 1629676799999 | 1 | 2021-08-23 00:00:00 | 2021-08-22 23:59:59 | 0 | 2021-08-23 02:51:43 |
+------+------------------------+------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
10 rows in set (0.00 sec)
——
select count(id) from usage_job where success = 0;
+-----------+
| count(id) |
+-----------+
| 999 |
+-----------+
1 row in set (0.00 sec)
Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies
vivek.kumar@indiqus.com <ma...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>
> On 14-Jul-2022, at 3:03 PM, Daan Hoogland <da...@gmail.com> wrote:
>
> On Wed, Jul 13, 2022 at 3:51 AM Vivek Kumar <vivek.kumar@indiqus.com.invalid <ma...@indiqus.com.invalid>>
> wrote:
>
>> Hello Daan,
>>
>> Thanks for the response.
>>
>> So we have deployed a replica of our management servers and databases on
>> a complete isolated environment, Tried a work around which you have
>> suggested but PID is not getting updated in the usage_job. Usage.logs gets
>> stuck on below logs - We have also tried to upgrade on 4.16 on the replica
>> setup but same situation with 4.16 as well.
>>
>>
>> 2022-07-04 10:23:11,977 INFO [cloud.usage.UsageManagerImpl] (main:null)
>> (logid:) Implementation Version is 4.16.1.0
>> 2022-07-11 04:26:23,905 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
>> (main:null) (logid:) Encryption Type: file
>> 2022-07-11 04:26:23,927 INFO [db.Transaction.Transaction] (main:null)
>> (logid:) Is Data Base High Availiability enabled? Ans : true
>> 2022-07-11 04:26:23,981 INFO [db.Transaction.Transaction] (main:null)
>> (logid:) The replicas configured for Cloud Data base is/are : 198.18.8.8
>> 2022-07-11 04:26:24,014 DEBUG [utils.db.DriverLoader] (main:null) (logid:)
>> Successfully loaded DB driver com.mysql.cj.jdbc.Driver
>> 2022-07-11 04:26:24,063 DEBUG [utils.db.DriverLoader] (main:null) (logid:)
>> DB driver com.mysql.cj.jdbc.Driver was already loaded.
>> 2022-07-11 04:26:24,070 DEBUG [utils.db.DriverLoader] (main:null) (logid:)
>> DB driver com.mysql.cj.jdbc.Driver was already loaded.
>> 2022-07-11 04:26:25,172 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Setup Spring Application context
>> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_b5109665
>> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_42da784f
>> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_bf461acb
>> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_a66084d9
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_35dea57c
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_7ec9190f
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_98c3bb86
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_a3d9d9aa
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_c06bb621
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_82eb64e0
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_95b19ecb
>> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6458895c
>> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_d545c0a0
>> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_ef2ae44f
>> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_39357dcf
>> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_421ab841
>> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_3d9daaed
>> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_a3dbe792
>> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_9f537032
>> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_5b9ccd8c
>> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_e67e53bd
>> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_6e25b0b3
>> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_91eaad5
>> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_5bdd1db5
>> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_e36e4eb8
>> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_d51f0ffc
>> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_ae698650
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_a22b67b4
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_42e796e4
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_84083314
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_fc57081c
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3532324e
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_bf4cdf9a
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_3c5a442c
>> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_6e84e70b
>> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6150198d
>> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_a8a8c734
>> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_58bc9eff
>> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ad0cdf5e
>> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_87f6b7d5
>> 2022-07-11 04:26:25,625 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
>> (logid:) Checking to see if usage.vmops.pid exists.
>> 2022-07-11 04:26:25,625 INFO [cloud.usage.UsageManagerImpl] (main:null)
>> (logid:) Implementation Version is 4.16.1.0
>>
>
> I see no errors here.
>
>>
>>
>>
>> — Also we have seen in the usage_job - there are multiple jobs which are
>> showing as ‘zero’ in success, So my question is - Is last job has to be set
>> to success true to all the jobs has to be success true ?
>>
>>
>> mysql> select
>> id,host,pid,job_type,scheduled,start_millis,end_millis,exec_time,start_date,end_date,success
>> from usage_job order by id desc limit 10;
>>
>> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
>> | id | host | pid | job_type | scheduled |
>> start_millis | end_millis | exec_time | start_date | end_date
>> | success |
>>
>> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
>> | 2004 | acs-mgmt-02/198.18.8.6 | 30073 | 0 | 0 |
>> 1630022400000 | 1630108799999 | 194785 | 2021-08-27 00:00:00 |
>> 2021-08-27 23:59:59 | 1 |
>> | 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 |
>> 2021-08-26 23:59:59. | 0 |
>> | 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 |
>> 2021-08-26 23:59:59. | 0 |
>> | 2001 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1629936000000 | 1630022399999 | 189715 | 2021-08-26 00:00:00 |
>> 2021-08-26 23:59:59. | 1 |
>> | 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
>> 2021-08-25 23:59:59. | 0 |
>> | 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
>> 2021-08-25 23:59:59. | 0 |
>> | 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 |
>> 2021-08-25 23:59:59. | 0 |
>> | 1997 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1629849600000 | 1629935999999 | 200641 | 2021-08-25 00:00:00 |
>> 2021-08-25 23:59:59. | 1 |
>> | 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 |
>> 2021-08-24 23:59:59. | 0 |
>> | 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
>> 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 |
>> 2021-08-24 23:59:59. | 0 |
>>
>> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
>>
>
> What I would expect, if the usage server is running is to see a reord where
> none of start_millis, end_millis and success are filled. Your DB shows
> multiple jobs with the same time stamps which is wrong. As if multiple
> processes with the same PID are running, which should be impossible on any
> *nis-like system.
> A record like
> |2005|acs-mgmt-02/198.18.8.6|30073|| 0| 0| 0| 0| 0|
> [NULL]| [NULL] | [NULL]| <heartbeat-timestamp>|
> is expected.
> can you try
>> select * from usage_job where success is null order by id desc limit 10;
>
>
>>
>> — I have also observed while checking the usage_service - It’s saying
>> something related to mysql jdbc driver. However same driver is working
>> everywhere -
>>
>>
>> # rpm -qa | grep “connector"
>>
>> mysql-connector-python-1.1.6-1.el7.noarch
>> mysql-connector-java-5.1.25-3.el7.noarch
>>
>> — usage service status
>>
>> [root@acs-mgmt-01 ~]# systemctl status cloudstack-usage
>> ● cloudstack-usage.service - CloudStack Usage Server
>> Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service;
>> enabled; vendor preset: disabled)
>> Active: active (running) since Mon 2022-07-11 04:26:20 -03; 4s ago
>> Docs: http://www.cloudstack.org/
>> Main PID: 1079 (java)
>> Tasks: 24
>> CGroup: /system.slice/cloudstack-usage.service
>> └─1079 /usr/bin/java -Dpid=1079 -Xms256m -Xmx2048m -cp
>> /usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/cloudstack-mysql-ha/lib/*:/etc/cl...
>>
>> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: Defaulting to no-operation
>> (NOP) logger implementation
>> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: See
>> http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
>> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: An illegal reflective
>> access operation has occurred
>> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Illegal reflective access
>> by net.sf.cglib.core.ReflectUtils$1
>> (file:/usr/share/cloudstack-usage/lib/cglib-nodep-3.3....tionDomain)
>> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Please consider reporting
>> this to the maintainers of net.sf.cglib.core.ReflectUtils$1
>> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Use --illegal-access=warn
>> to enable warnings of further illegal reflective access operations
>> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: All illegal access
>> operations will be denied in a future release
>> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,927 INFO Transaction:1027
>> - Is Data Base High Availiability enabled? Ans : true
>> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,981 INFO Transaction:1070
>> - The replicas configured for Cloud Data base is/are : 198.18.8.8
>> Jul 11 04:26:24 acs-mgmt-01 sh[1079]: Loading class
>> `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is
>> `com.mysql.cj.jdbc.Driver'. The driver is ...nnecessary.
>>
>>
>> — I am also concerned about the time zone, is there any specific time
>> zones which needs to be placed in global setting - I have tried putting UTC
>> and other near Time Zone as well but nothing worked.
>>
>> mysql> select name,value from configuration where name like '%usage%' and
>> component = 'management-server';
>> +-----------------------------------+----------------+
>> | name | value |
>> +-----------------------------------+----------------+
>> | enable.usage.server | true |
>> | publish.usage.events | true |
>> | usage.aggregation.timezone | America/Cuiaba |
>> | usage.execution.timezone | America/Cuiaba |
>> | usage.sanity.check.interval | 1 |
>> | usage.stats.job.aggregation.range | 60 |
>> | usage.stats.job.exec.time | 00:10 |
>> +-----------------------------------+----------------+
>>
>
> Time zones need to be defined on your system to be used in mysql. you'll
> have to look into your zones dir (see google for howto) 'System',
> 'Europe/Amsterdam' or '+01:00' should work depending on your system
> configuration.
>
>
>
>>
>>
>>
>>
>> Vivek Kumar
>> Sr. Manager - Cloud & DevOps
>> TechOps | Indiqus Technologies
>>
>> vivek.kumar@indiqus.com <ma...@indiqus.com> <mailto:vivek.kumar@indiqus.com <ma...@indiqus.com>>
>> www.indiqus.com <http://www.indiqus.com/> <https://www.indiqus.com/ <https://www.indiqus.com/>>
>>
>>
>>
>>
>>> On 11-Jul-2022, at 12:15 PM, Daan Hoogland <daan.hoogland@gmail.com <ma...@gmail.com>>
>> wrote:
>>>
>>> vivek,
>>> *take care with this and make a backup of the cloud_usage.usage_job
>> first*
>>> check in the usage_job table to see if there is a record for a not yet
>>> completed job.
>>> get the pid from that job and check if that is the same as the currently
>>> running usage server.
>>> If not, stop the usage server, remove the record and restart the usage
>>> server.
>>>
>>> if the pids match you'll need to investigate further.
>>>
>>> On Sun, Jul 10, 2022 at 8:50 AM Vivek Kumar <vivek.kumar@indiqus.com
>> .invalid>
>>> wrote:
>>>
>>>> Hello Folks,
>>>>
>>>> Can someone please help me on this ?
>>>>
>>>>
>>>> Vivek Kumar
>>>> Sr. Manager - Cloud & DevOps
>>>> TechOps | Indiqus Technologies
>>>>
>>>> vivek.kumar@indiqus.com <ma...@indiqus.com>
>>>> www.indiqus.com <https://www.indiqus.com/>
>>>>
>>>>
>>>>
>>>>
>>>>> On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vi...@indiqus.com>
>> wrote:
>>>>>
>>>>> Hello Guys,
>>>>>
>>>>> I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We
>>>> have recently observed that cloud_usage is not being generated since the
>>>> moment we had upgrade. Here is the details and logs from usage.log. Can
>>>> someone help me out what else do we need to change ot check. We have
>>>> restarted both management and usage service multiple times.
>>>>>
>>>>> 1- Global settings -
>>>>>
>>>>> mysql> select * from configuration where name like '%usage%' and
>>>> component = 'management-server';
>>>>>
>>>>
>> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
>>>>> | category | instance | component | name
>>>> | value | description
>>>> | default_value |
>>>> updated | scope | is_dynamic |
>>>>>
>>>>
>> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
>>>>> | Usage | DEFAUL. | management-server | enable.usage.server
>>>> | true | Flag for enabling usage
>>>> | true |
>>>> NULL | NULL | 0 |
>>>>> | Advanced | DEFAULT | management-server | publish.usage.events
>>>> | true | enable or disable publishing of usage events on the event
>>>> bus | true
>> |
>>>> NULL | NULL | 0 |
>>>>> | Usage | DEFAULT | management-server | usage.aggregation.timezone
>>>> | UTC-3 | The timezone to use for usage stats aggregation
>>>> | GMT |
>>>> NULL | NULL | 0 |
>>>>> | Usage | DEFAULT | management-server | usage.execution.timezone
>>>> | UTC-3 | The timezone to use for usage job execution time
>>>> | NULL |
>>>> NULL | NULL | 0 |
>>>>> | Usage | DEFAULT | management-server | usage.sanity.check.interval
>>>> | NULL | Interval (in days) to check sanity of usage data
>>>> | NULL |
>>>> NULL | NULL | 0 |
>>>>> | Usage | DEFAULT | management-server |
>>>> usage.stats.job.aggregation.range | 60 | The range of time for
>>>> aggregating the user statistics specified in minutes (e.g. 1440 for
>> daily,
>>>> 60 for hourly. | 1440 | NULL | NULL | 0 |
>>>>> | Usage | DEFAULT | management-server | usage.stats.job.exec.time
>>>> | 00:15 | The time at which the usage statistics aggregation job
>> will
>>>> run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 |
>>>> NULL | NULL | 0 |
>>>>>
>>>>
>> +----------+----------+-------------------+-----------------------------------+-------+—————————————————————————————————————————————————————————
>>>>>
>>>>> 2- Rpm installed -
>>>>>
>>>>>
>>>>>
>>>>> [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack"
>>>>> cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64
>>>>> cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64
>>>>> cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64
>>>>> cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64
>>>>> cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64
>>>>>
>>>>>
>>>>> 3- I am getting below logs in all usage.log -
>>>>>
>>>>>
>>>>> 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
>>>>> 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl]
>> (main:null)
>>>> (logid:) Checking to see if usage.vmops.pid exists.
>>>>> 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl]
>> (main:null)
>>>> (logid:) Implementation Version is 4.15.1.0
>>>>> 2022-03-29 13:21:45,059 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
>>>> (main:null) (logid:) Encryption Type: file
>>>>> 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null)
>>>> (logid:) Is Data Base High Availiability enabled? Ans : true
>>>>> 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null)
>>>> (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8
>>>>> 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null)
>>>> (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
>>>>> 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null)
>>>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
>>>>> 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null)
>>>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
>>>>> 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Setup Spring Application context
>>>>> 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef
>>>>> 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9
>>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855
>>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263
>>>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306
>>>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699
>>>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910
>>>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734
>>>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab
>>>>> 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a
>>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55
>>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6
>>>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a
>>>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9
>>>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59
>>>>> 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb
>>>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877
>>>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c
>>>>> 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc
>>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16
>>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147
>>>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d
>>>>> 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5
>>>>> 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f
>>>>> 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f
>>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42
>>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86
>>>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da
>>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e
>>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e
>>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e
>>>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6
>>>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8
>>>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24
>>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6
>>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495
>>>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717
>>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be
>>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>>
>> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89
>>>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8
>>>>> 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext]
>>>> (main:null) (logid:) Configuring
>>>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
>>>>> 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl]
>> (main:null)
>>>> (logid:) Checking to see if usage.vmops.pid exists.
>>>>> 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl]
>> (main:null)
>>>> (logid:) Implementation Version is 4.15.1.0
>>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Vivek Kumar
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> This message is intended only for the use of the individual or entity to
>>>> which it is addressed and may contain confidential and/or privileged
>>>> information. If you are not the intended recipient, please delete the
>>>> original message and any copy of it from your computer system. You are
>>>> hereby notified that any dissemination, distribution or copying of this
>>>> communication is strictly prohibited unless proper authorization has
>> been
>>>> obtained for such action. If you have received this communication in
>>>> error,
>>>> please notify the sender immediately. Although IndiQus attempts to sweep
>>>> e-mail and attachments for viruses, it does not guarantee that both are
>>>> virus-free and accepts no liability for any damage sustained as a result
>>>> of
>>>> viruses.
>>>>
>>>
>>>
>>> --
>>> Daan
>>
>>
>> --
>> This message is intended only for the use of the individual or entity to
>> which it is addressed and may contain confidential and/or privileged
>> information. If you are not the intended recipient, please delete the
>> original message and any copy of it from your computer system. You are
>> hereby notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited unless proper authorization has been
>> obtained for such action. If you have received this communication in
>> error,
>> please notify the sender immediately. Although IndiQus attempts to sweep
>> e-mail and attachments for viruses, it does not guarantee that both are
>> virus-free and accepts no liability for any damage sustained as a result
>> of
>> viruses.
>>
>
>
> --
> Daan
--
This message is intended only for the use of the individual or entity to
which it is addressed and may contain confidential and/or privileged
information. If you are not the intended recipient, please delete the
original message and any copy of it from your computer system. You are
hereby notified that any dissemination, distribution or copying of this
communication is strictly prohibited unless proper authorization has been
obtained for such action. If you have received this communication in error,
please notify the sender immediately. Although IndiQus attempts to sweep
e-mail and attachments for viruses, it does not guarantee that both are
virus-free and accepts no liability for any damage sustained as a result of
viruses.
Re: Usage is not working | ACS 4.15
Posted by Daan Hoogland <da...@gmail.com>.
On Wed, Jul 13, 2022 at 3:51 AM Vivek Kumar <vi...@indiqus.com.invalid>
wrote:
> Hello Daan,
>
> Thanks for the response.
>
> So we have deployed a replica of our management servers and databases on
> a complete isolated environment, Tried a work around which you have
> suggested but PID is not getting updated in the usage_job. Usage.logs gets
> stuck on below logs - We have also tried to upgrade on 4.16 on the replica
> setup but same situation with 4.16 as well.
>
>
> 2022-07-04 10:23:11,977 INFO [cloud.usage.UsageManagerImpl] (main:null)
> (logid:) Implementation Version is 4.16.1.0
> 2022-07-11 04:26:23,905 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
> (main:null) (logid:) Encryption Type: file
> 2022-07-11 04:26:23,927 INFO [db.Transaction.Transaction] (main:null)
> (logid:) Is Data Base High Availiability enabled? Ans : true
> 2022-07-11 04:26:23,981 INFO [db.Transaction.Transaction] (main:null)
> (logid:) The replicas configured for Cloud Data base is/are : 198.18.8.8
> 2022-07-11 04:26:24,014 DEBUG [utils.db.DriverLoader] (main:null) (logid:)
> Successfully loaded DB driver com.mysql.cj.jdbc.Driver
> 2022-07-11 04:26:24,063 DEBUG [utils.db.DriverLoader] (main:null) (logid:)
> DB driver com.mysql.cj.jdbc.Driver was already loaded.
> 2022-07-11 04:26:24,070 DEBUG [utils.db.DriverLoader] (main:null) (logid:)
> DB driver com.mysql.cj.jdbc.Driver was already loaded.
> 2022-07-11 04:26:25,172 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Setup Spring Application context
> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_b5109665
> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_42da784f
> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_bf461acb
> 2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_a66084d9
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_35dea57c
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_7ec9190f
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_98c3bb86
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_a3d9d9aa
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_c06bb621
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_82eb64e0
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_95b19ecb
> 2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6458895c
> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_d545c0a0
> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_ef2ae44f
> 2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_39357dcf
> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_421ab841
> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_3d9daaed
> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_a3dbe792
> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_9f537032
> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_5b9ccd8c
> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_e67e53bd
> 2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_6e25b0b3
> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_91eaad5
> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_5bdd1db5
> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_e36e4eb8
> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_d51f0ffc
> 2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_ae698650
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_a22b67b4
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_42e796e4
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_84083314
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_fc57081c
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3532324e
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_bf4cdf9a
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_3c5a442c
> 2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_6e84e70b
> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6150198d
> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_a8a8c734
> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_58bc9eff
> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ad0cdf5e
> 2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_87f6b7d5
> 2022-07-11 04:26:25,625 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
> (logid:) Checking to see if usage.vmops.pid exists.
> 2022-07-11 04:26:25,625 INFO [cloud.usage.UsageManagerImpl] (main:null)
> (logid:) Implementation Version is 4.16.1.0
>
I see no errors here.
>
>
>
> — Also we have seen in the usage_job - there are multiple jobs which are
> showing as ‘zero’ in success, So my question is - Is last job has to be set
> to success true to all the jobs has to be success true ?
>
>
> mysql> select
> id,host,pid,job_type,scheduled,start_millis,end_millis,exec_time,start_date,end_date,success
> from usage_job order by id desc limit 10;
>
> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
> | id | host | pid | job_type | scheduled |
> start_millis | end_millis | exec_time | start_date | end_date
> | success |
>
> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
> | 2004 | acs-mgmt-02/198.18.8.6 | 30073 | 0 | 0 |
> 1630022400000 | 1630108799999 | 194785 | 2021-08-27 00:00:00 |
> 2021-08-27 23:59:59 | 1 |
> | 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 |
> 2021-08-26 23:59:59. | 0 |
> | 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 |
> 2021-08-26 23:59:59. | 0 |
> | 2001 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629936000000 | 1630022399999 | 189715 | 2021-08-26 00:00:00 |
> 2021-08-26 23:59:59. | 1 |
> | 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
> 2021-08-25 23:59:59. | 0 |
> | 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 |
> 2021-08-25 23:59:59. | 0 |
> | 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 |
> 2021-08-25 23:59:59. | 0 |
> | 1997 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629849600000 | 1629935999999 | 200641 | 2021-08-25 00:00:00 |
> 2021-08-25 23:59:59. | 1 |
> | 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 |
> 2021-08-24 23:59:59. | 0 |
> | 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 |
> 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 |
> 2021-08-24 23:59:59. | 0 |
>
> +------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
>
What I would expect, if the usage server is running is to see a reord where
none of start_millis, end_millis and success are filled. Your DB shows
multiple jobs with the same time stamps which is wrong. As if multiple
processes with the same PID are running, which should be impossible on any
*nis-like system.
A record like
|2005|acs-mgmt-02/198.18.8.6|30073|| 0| 0| 0| 0| 0|
[NULL]| [NULL] | [NULL]| <heartbeat-timestamp>|
is expected.
can you try
> select * from usage_job where success is null order by id desc limit 10;
>
> — I have also observed while checking the usage_service - It’s saying
> something related to mysql jdbc driver. However same driver is working
> everywhere -
>
>
> # rpm -qa | grep “connector"
>
> mysql-connector-python-1.1.6-1.el7.noarch
> mysql-connector-java-5.1.25-3.el7.noarch
>
> — usage service status
>
> [root@acs-mgmt-01 ~]# systemctl status cloudstack-usage
> ● cloudstack-usage.service - CloudStack Usage Server
> Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service;
> enabled; vendor preset: disabled)
> Active: active (running) since Mon 2022-07-11 04:26:20 -03; 4s ago
> Docs: http://www.cloudstack.org/
> Main PID: 1079 (java)
> Tasks: 24
> CGroup: /system.slice/cloudstack-usage.service
> └─1079 /usr/bin/java -Dpid=1079 -Xms256m -Xmx2048m -cp
> /usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/cloudstack-mysql-ha/lib/*:/etc/cl...
>
> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: Defaulting to no-operation
> (NOP) logger implementation
> Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: See
> http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: An illegal reflective
> access operation has occurred
> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Illegal reflective access
> by net.sf.cglib.core.ReflectUtils$1
> (file:/usr/share/cloudstack-usage/lib/cglib-nodep-3.3....tionDomain)
> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Please consider reporting
> this to the maintainers of net.sf.cglib.core.ReflectUtils$1
> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Use --illegal-access=warn
> to enable warnings of further illegal reflective access operations
> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: All illegal access
> operations will be denied in a future release
> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,927 INFO Transaction:1027
> - Is Data Base High Availiability enabled? Ans : true
> Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,981 INFO Transaction:1070
> - The replicas configured for Cloud Data base is/are : 198.18.8.8
> Jul 11 04:26:24 acs-mgmt-01 sh[1079]: Loading class
> `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is
> `com.mysql.cj.jdbc.Driver'. The driver is ...nnecessary.
>
>
> — I am also concerned about the time zone, is there any specific time
> zones which needs to be placed in global setting - I have tried putting UTC
> and other near Time Zone as well but nothing worked.
>
> mysql> select name,value from configuration where name like '%usage%' and
> component = 'management-server';
> +-----------------------------------+----------------+
> | name | value |
> +-----------------------------------+----------------+
> | enable.usage.server | true |
> | publish.usage.events | true |
> | usage.aggregation.timezone | America/Cuiaba |
> | usage.execution.timezone | America/Cuiaba |
> | usage.sanity.check.interval | 1 |
> | usage.stats.job.aggregation.range | 60 |
> | usage.stats.job.exec.time | 00:10 |
> +-----------------------------------+----------------+
>
Time zones need to be defined on your system to be used in mysql. you'll
have to look into your zones dir (see google for howto) 'System',
'Europe/Amsterdam' or '+01:00' should work depending on your system
configuration.
>
>
>
>
> Vivek Kumar
> Sr. Manager - Cloud & DevOps
> TechOps | Indiqus Technologies
>
> vivek.kumar@indiqus.com <ma...@indiqus.com>
> www.indiqus.com <https://www.indiqus.com/>
>
>
>
>
> > On 11-Jul-2022, at 12:15 PM, Daan Hoogland <da...@gmail.com>
> wrote:
> >
> > vivek,
> > *take care with this and make a backup of the cloud_usage.usage_job
> first*
> > check in the usage_job table to see if there is a record for a not yet
> > completed job.
> > get the pid from that job and check if that is the same as the currently
> > running usage server.
> > If not, stop the usage server, remove the record and restart the usage
> > server.
> >
> > if the pids match you'll need to investigate further.
> >
> > On Sun, Jul 10, 2022 at 8:50 AM Vivek Kumar <vivek.kumar@indiqus.com
> .invalid>
> > wrote:
> >
> >> Hello Folks,
> >>
> >> Can someone please help me on this ?
> >>
> >>
> >> Vivek Kumar
> >> Sr. Manager - Cloud & DevOps
> >> TechOps | Indiqus Technologies
> >>
> >> vivek.kumar@indiqus.com <ma...@indiqus.com>
> >> www.indiqus.com <https://www.indiqus.com/>
> >>
> >>
> >>
> >>
> >>> On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vi...@indiqus.com>
> wrote:
> >>>
> >>> Hello Guys,
> >>>
> >>> I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We
> >> have recently observed that cloud_usage is not being generated since the
> >> moment we had upgrade. Here is the details and logs from usage.log. Can
> >> someone help me out what else do we need to change ot check. We have
> >> restarted both management and usage service multiple times.
> >>>
> >>> 1- Global settings -
> >>>
> >>> mysql> select * from configuration where name like '%usage%' and
> >> component = 'management-server';
> >>>
> >>
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> >>> | category | instance | component | name
> >> | value | description
> >> | default_value |
> >> updated | scope | is_dynamic |
> >>>
> >>
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> >>> | Usage | DEFAUL. | management-server | enable.usage.server
> >> | true | Flag for enabling usage
> >> | true |
> >> NULL | NULL | 0 |
> >>> | Advanced | DEFAULT | management-server | publish.usage.events
> >> | true | enable or disable publishing of usage events on the event
> >> bus | true
> |
> >> NULL | NULL | 0 |
> >>> | Usage | DEFAULT | management-server | usage.aggregation.timezone
> >> | UTC-3 | The timezone to use for usage stats aggregation
> >> | GMT |
> >> NULL | NULL | 0 |
> >>> | Usage | DEFAULT | management-server | usage.execution.timezone
> >> | UTC-3 | The timezone to use for usage job execution time
> >> | NULL |
> >> NULL | NULL | 0 |
> >>> | Usage | DEFAULT | management-server | usage.sanity.check.interval
> >> | NULL | Interval (in days) to check sanity of usage data
> >> | NULL |
> >> NULL | NULL | 0 |
> >>> | Usage | DEFAULT | management-server |
> >> usage.stats.job.aggregation.range | 60 | The range of time for
> >> aggregating the user statistics specified in minutes (e.g. 1440 for
> daily,
> >> 60 for hourly. | 1440 | NULL | NULL | 0 |
> >>> | Usage | DEFAULT | management-server | usage.stats.job.exec.time
> >> | 00:15 | The time at which the usage statistics aggregation job
> will
> >> run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 |
> >> NULL | NULL | 0 |
> >>>
> >>
> +----------+----------+-------------------+-----------------------------------+-------+—————————————————————————————————————————————————————————
> >>>
> >>> 2- Rpm installed -
> >>>
> >>>
> >>>
> >>> [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack"
> >>> cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64
> >>> cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64
> >>> cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64
> >>> cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64
> >>> cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64
> >>>
> >>>
> >>> 3- I am getting below logs in all usage.log -
> >>>
> >>>
> >>> 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> >>> 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl]
> (main:null)
> >> (logid:) Checking to see if usage.vmops.pid exists.
> >>> 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl]
> (main:null)
> >> (logid:) Implementation Version is 4.15.1.0
> >>> 2022-03-29 13:21:45,059 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
> >> (main:null) (logid:) Encryption Type: file
> >>> 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null)
> >> (logid:) Is Data Base High Availiability enabled? Ans : true
> >>> 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null)
> >> (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8
> >>> 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null)
> >> (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
> >>> 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null)
> >> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> >>> 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null)
> >> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> >>> 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Setup Spring Application context
> >>> 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef
> >>> 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9
> >>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855
> >>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263
> >>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306
> >>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699
> >>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910
> >>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734
> >>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab
> >>> 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a
> >>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55
> >>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6
> >>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a
> >>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9
> >>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59
> >>> 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb
> >>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877
> >>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c
> >>> 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc
> >>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16
> >>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147
> >>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d
> >>> 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5
> >>> 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f
> >>> 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f
> >>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42
> >>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86
> >>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da
> >>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e
> >>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e
> >>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e
> >>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6
> >>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8
> >>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24
> >>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6
> >>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495
> >>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717
> >>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be
> >>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >>
> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89
> >>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8
> >>> 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext]
> >> (main:null) (logid:) Configuring
> >> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> >>> 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl]
> (main:null)
> >> (logid:) Checking to see if usage.vmops.pid exists.
> >>> 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl]
> (main:null)
> >> (logid:) Implementation Version is 4.15.1.0
> >>>
> >>>
> >>>
> >>> Regards,
> >>> Vivek Kumar
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> This message is intended only for the use of the individual or entity to
> >> which it is addressed and may contain confidential and/or privileged
> >> information. If you are not the intended recipient, please delete the
> >> original message and any copy of it from your computer system. You are
> >> hereby notified that any dissemination, distribution or copying of this
> >> communication is strictly prohibited unless proper authorization has
> been
> >> obtained for such action. If you have received this communication in
> >> error,
> >> please notify the sender immediately. Although IndiQus attempts to sweep
> >> e-mail and attachments for viruses, it does not guarantee that both are
> >> virus-free and accepts no liability for any damage sustained as a result
> >> of
> >> viruses.
> >>
> >
> >
> > --
> > Daan
>
>
> --
> This message is intended only for the use of the individual or entity to
> which it is addressed and may contain confidential and/or privileged
> information. If you are not the intended recipient, please delete the
> original message and any copy of it from your computer system. You are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited unless proper authorization has been
> obtained for such action. If you have received this communication in
> error,
> please notify the sender immediately. Although IndiQus attempts to sweep
> e-mail and attachments for viruses, it does not guarantee that both are
> virus-free and accepts no liability for any damage sustained as a result
> of
> viruses.
>
--
Daan
Re: Usage is not working | ACS 4.15
Posted by Vivek Kumar <vi...@indiqus.com.INVALID>.
Hello Daan,
Thanks for the response.
So we have deployed a replica of our management servers and databases on a complete isolated environment, Tried a work around which you have suggested but PID is not getting updated in the usage_job. Usage.logs gets stuck on below logs - We have also tried to upgrade on 4.16 on the replica setup but same situation with 4.16 as well.
2022-07-04 10:23:11,977 INFO [cloud.usage.UsageManagerImpl] (main:null) (logid:) Implementation Version is 4.16.1.0
2022-07-11 04:26:23,905 DEBUG [utils.crypt.EncryptionSecretKeyChecker] (main:null) (logid:) Encryption Type: file
2022-07-11 04:26:23,927 INFO [db.Transaction.Transaction] (main:null) (logid:) Is Data Base High Availiability enabled? Ans : true
2022-07-11 04:26:23,981 INFO [db.Transaction.Transaction] (main:null) (logid:) The replicas configured for Cloud Data base is/are : 198.18.8.8
2022-07-11 04:26:24,014 DEBUG [utils.db.DriverLoader] (main:null) (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
2022-07-11 04:26:24,063 DEBUG [utils.db.DriverLoader] (main:null) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
2022-07-11 04:26:24,070 DEBUG [utils.db.DriverLoader] (main:null) (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
2022-07-11 04:26:25,172 INFO [utils.component.ComponentContext] (main:null) (logid:) Setup Spring Application context
2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_b5109665
2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_42da784f
2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_bf461acb
2022-07-11 04:26:25,616 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_a66084d9
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_35dea57c
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_7ec9190f
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_98c3bb86
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_a3d9d9aa
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_c06bb621
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_82eb64e0
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_95b19ecb
2022-07-11 04:26:25,617 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6458895c
2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_d545c0a0
2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_ef2ae44f
2022-07-11 04:26:25,618 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_39357dcf
2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_421ab841
2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_3d9daaed
2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_a3dbe792
2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_9f537032
2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_5b9ccd8c
2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_e67e53bd
2022-07-11 04:26:25,619 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_6e25b0b3
2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_91eaad5
2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_5bdd1db5
2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_e36e4eb8
2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_d51f0ffc
2022-07-11 04:26:25,623 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_ae698650
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_a22b67b4
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_42e796e4
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_84083314
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_fc57081c
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3532324e
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_bf4cdf9a
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_3c5a442c
2022-07-11 04:26:25,624 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_6e84e70b
2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6150198d
2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_a8a8c734
2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_58bc9eff
2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_ad0cdf5e
2022-07-11 04:26:25,625 INFO [utils.component.ComponentContext] (main:null) (logid:) Configuring com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_87f6b7d5
2022-07-11 04:26:25,625 DEBUG [cloud.usage.UsageManagerImpl] (main:null) (logid:) Checking to see if usage.vmops.pid exists.
2022-07-11 04:26:25,625 INFO [cloud.usage.UsageManagerImpl] (main:null) (logid:) Implementation Version is 4.16.1.0
— Also we have seen in the usage_job - there are multiple jobs which are showing as ‘zero’ in success, So my question is - Is last job has to be set to success true to all the jobs has to be success true ?
mysql> select id,host,pid,job_type,scheduled,start_millis,end_millis,exec_time,start_date,end_date,success from usage_job order by id desc limit 10;
+------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
| id | host | pid | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success |
+------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
| 2004 | acs-mgmt-02/198.18.8.6 | 30073 | 0 | 0 | 1630022400000 | 1630108799999 | 194785 | 2021-08-27 00:00:00 | 2021-08-27 23:59:59 | 1 |
| 2003 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1630022400000 | 1630022399999 | 6 | 2021-08-27 00:00:00 | 2021-08-26 23:59:59. | 0 |
| 2002 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1630022400000 | 1630022399999 | 5 | 2021-08-27 00:00:00 | 2021-08-26 23:59:59. | 0 |
| 2001 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629936000000 | 1630022399999 | 189715 | 2021-08-26 00:00:00 | 2021-08-26 23:59:59. | 1 |
| 2000 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | 2021-08-25 23:59:59. | 0 |
| 1999 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629936000000 | 1629935999999 | 2 | 2021-08-26 00:00:00 | 2021-08-25 23:59:59. | 0 |
| 1998 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629936000000 | 1629935999999 | 1 | 2021-08-26 00:00:00 | 2021-08-25 23:59:59. | 0 |
| 1997 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629849600000 | 1629935999999 | 200641 | 2021-08-25 00:00:00 | 2021-08-25 23:59:59. | 1 |
| 1996 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629849600000 | 1629849599999 | 2 | 2021-08-25 00:00:00 | 2021-08-24 23:59:59. | 0 |
| 1995 | acs-mgmt-01/198.18.8.5 | 7026 | 0 | 0 | 1629849600000 | 1629849599999 | 1 | 2021-08-25 00:00:00 | 2021-08-24 23:59:59. | 0 |
+------+------------------------+-------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+
— I have also observed while checking the usage_service - It’s saying something related to mysql jdbc driver. However same driver is working everywhere -
# rpm -qa | grep “connector"
mysql-connector-python-1.1.6-1.el7.noarch
mysql-connector-java-5.1.25-3.el7.noarch
— usage service status
[root@acs-mgmt-01 ~]# systemctl status cloudstack-usage
● cloudstack-usage.service - CloudStack Usage Server
Loaded: loaded (/usr/lib/systemd/system/cloudstack-usage.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2022-07-11 04:26:20 -03; 4s ago
Docs: http://www.cloudstack.org/
Main PID: 1079 (java)
Tasks: 24
CGroup: /system.slice/cloudstack-usage.service
└─1079 /usr/bin/java -Dpid=1079 -Xms256m -Xmx2048m -cp /usr/share/cloudstack-usage/*:/usr/share/cloudstack-usage/lib/*:/usr/share/cloudstack-mysql-ha/lib/*:/etc/cl...
Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Jul 11 04:26:22 acs-mgmt-01 sh[1079]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: An illegal reflective access operation has occurred
Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$1 (file:/usr/share/cloudstack-usage/lib/cglib-nodep-3.3....tionDomain)
Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$1
Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Jul 11 04:26:23 acs-mgmt-01 sh[1079]: WARNING: All illegal access operations will be denied in a future release
Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,927 INFO Transaction:1027 - Is Data Base High Availiability enabled? Ans : true
Jul 11 04:26:23 acs-mgmt-01 sh[1079]: 07:26:23,981 INFO Transaction:1070 - The replicas configured for Cloud Data base is/are : 198.18.8.8
Jul 11 04:26:24 acs-mgmt-01 sh[1079]: Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is ...nnecessary.
— I am also concerned about the time zone, is there any specific time zones which needs to be placed in global setting - I have tried putting UTC and other near Time Zone as well but nothing worked.
mysql> select name,value from configuration where name like '%usage%' and component = 'management-server';
+-----------------------------------+----------------+
| name | value |
+-----------------------------------+----------------+
| enable.usage.server | true |
| publish.usage.events | true |
| usage.aggregation.timezone | America/Cuiaba |
| usage.execution.timezone | America/Cuiaba |
| usage.sanity.check.interval | 1 |
| usage.stats.job.aggregation.range | 60 |
| usage.stats.job.exec.time | 00:10 |
+-----------------------------------+----------------+
Vivek Kumar
Sr. Manager - Cloud & DevOps
TechOps | Indiqus Technologies
vivek.kumar@indiqus.com <ma...@indiqus.com>
www.indiqus.com <https://www.indiqus.com/>
> On 11-Jul-2022, at 12:15 PM, Daan Hoogland <da...@gmail.com> wrote:
>
> vivek,
> *take care with this and make a backup of the cloud_usage.usage_job first*
> check in the usage_job table to see if there is a record for a not yet
> completed job.
> get the pid from that job and check if that is the same as the currently
> running usage server.
> If not, stop the usage server, remove the record and restart the usage
> server.
>
> if the pids match you'll need to investigate further.
>
> On Sun, Jul 10, 2022 at 8:50 AM Vivek Kumar <vi...@indiqus.com.invalid>
> wrote:
>
>> Hello Folks,
>>
>> Can someone please help me on this ?
>>
>>
>> Vivek Kumar
>> Sr. Manager - Cloud & DevOps
>> TechOps | Indiqus Technologies
>>
>> vivek.kumar@indiqus.com <ma...@indiqus.com>
>> www.indiqus.com <https://www.indiqus.com/>
>>
>>
>>
>>
>>> On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vi...@indiqus.com> wrote:
>>>
>>> Hello Guys,
>>>
>>> I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We
>> have recently observed that cloud_usage is not being generated since the
>> moment we had upgrade. Here is the details and logs from usage.log. Can
>> someone help me out what else do we need to change ot check. We have
>> restarted both management and usage service multiple times.
>>>
>>> 1- Global settings -
>>>
>>> mysql> select * from configuration where name like '%usage%' and
>> component = 'management-server';
>>>
>> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
>>> | category | instance | component | name
>> | value | description
>> | default_value |
>> updated | scope | is_dynamic |
>>>
>> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
>>> | Usage | DEFAUL. | management-server | enable.usage.server
>> | true | Flag for enabling usage
>> | true |
>> NULL | NULL | 0 |
>>> | Advanced | DEFAULT | management-server | publish.usage.events
>> | true | enable or disable publishing of usage events on the event
>> bus | true |
>> NULL | NULL | 0 |
>>> | Usage | DEFAULT | management-server | usage.aggregation.timezone
>> | UTC-3 | The timezone to use for usage stats aggregation
>> | GMT |
>> NULL | NULL | 0 |
>>> | Usage | DEFAULT | management-server | usage.execution.timezone
>> | UTC-3 | The timezone to use for usage job execution time
>> | NULL |
>> NULL | NULL | 0 |
>>> | Usage | DEFAULT | management-server | usage.sanity.check.interval
>> | NULL | Interval (in days) to check sanity of usage data
>> | NULL |
>> NULL | NULL | 0 |
>>> | Usage | DEFAULT | management-server |
>> usage.stats.job.aggregation.range | 60 | The range of time for
>> aggregating the user statistics specified in minutes (e.g. 1440 for daily,
>> 60 for hourly. | 1440 | NULL | NULL | 0 |
>>> | Usage | DEFAULT | management-server | usage.stats.job.exec.time
>> | 00:15 | The time at which the usage statistics aggregation job will
>> run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 |
>> NULL | NULL | 0 |
>>>
>> +----------+----------+-------------------+-----------------------------------+-------+—————————————————————————————————————————————————————————
>>>
>>> 2- Rpm installed -
>>>
>>>
>>>
>>> [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack"
>>> cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64
>>> cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64
>>> cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64
>>> cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64
>>> cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64
>>>
>>>
>>> 3- I am getting below logs in all usage.log -
>>>
>>>
>>> 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
>>> 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
>> (logid:) Checking to see if usage.vmops.pid exists.
>>> 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl] (main:null)
>> (logid:) Implementation Version is 4.15.1.0
>>> 2022-03-29 13:21:45,059 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
>> (main:null) (logid:) Encryption Type: file
>>> 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null)
>> (logid:) Is Data Base High Availiability enabled? Ans : true
>>> 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null)
>> (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8
>>> 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null)
>> (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
>>> 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null)
>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
>>> 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null)
>> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
>>> 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Setup Spring Application context
>>> 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef
>>> 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9
>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855
>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263
>>> 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306
>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699
>>> 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910
>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734
>>> 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab
>>> 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a
>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55
>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6
>>> 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a
>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9
>>> 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59
>>> 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb
>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877
>>> 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c
>>> 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc
>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16
>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147
>>> 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d
>>> 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5
>>> 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f
>>> 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f
>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42
>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86
>>> 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da
>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e
>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e
>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e
>>> 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6
>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8
>>> 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24
>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6
>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495
>>> 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717
>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be
>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89
>>> 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8
>>> 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext]
>> (main:null) (logid:) Configuring
>> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
>>> 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
>> (logid:) Checking to see if usage.vmops.pid exists.
>>> 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl] (main:null)
>> (logid:) Implementation Version is 4.15.1.0
>>>
>>>
>>>
>>> Regards,
>>> Vivek Kumar
>>>
>>>
>>>
>>
>>
>> --
>> This message is intended only for the use of the individual or entity to
>> which it is addressed and may contain confidential and/or privileged
>> information. If you are not the intended recipient, please delete the
>> original message and any copy of it from your computer system. You are
>> hereby notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited unless proper authorization has been
>> obtained for such action. If you have received this communication in
>> error,
>> please notify the sender immediately. Although IndiQus attempts to sweep
>> e-mail and attachments for viruses, it does not guarantee that both are
>> virus-free and accepts no liability for any damage sustained as a result
>> of
>> viruses.
>>
>
>
> --
> Daan
--
This message is intended only for the use of the individual or entity to
which it is addressed and may contain confidential and/or privileged
information. If you are not the intended recipient, please delete the
original message and any copy of it from your computer system. You are
hereby notified that any dissemination, distribution or copying of this
communication is strictly prohibited unless proper authorization has been
obtained for such action. If you have received this communication in error,
please notify the sender immediately. Although IndiQus attempts to sweep
e-mail and attachments for viruses, it does not guarantee that both are
virus-free and accepts no liability for any damage sustained as a result of
viruses.
Re: Usage is not working | ACS 4.15
Posted by Daan Hoogland <da...@gmail.com>.
vivek,
*take care with this and make a backup of the cloud_usage.usage_job first*
check in the usage_job table to see if there is a record for a not yet
completed job.
get the pid from that job and check if that is the same as the currently
running usage server.
If not, stop the usage server, remove the record and restart the usage
server.
if the pids match you'll need to investigate further.
On Sun, Jul 10, 2022 at 8:50 AM Vivek Kumar <vi...@indiqus.com.invalid>
wrote:
> Hello Folks,
>
> Can someone please help me on this ?
>
>
> Vivek Kumar
> Sr. Manager - Cloud & DevOps
> TechOps | Indiqus Technologies
>
> vivek.kumar@indiqus.com <ma...@indiqus.com>
> www.indiqus.com <https://www.indiqus.com/>
>
>
>
>
> > On 30-Mar-2022, at 1:01 AM, Vivek Kumar <vi...@indiqus.com> wrote:
> >
> > Hello Guys,
> >
> > I have a ACS 4.15 deployment which was upgraded from 4.11 in past. We
> have recently observed that cloud_usage is not being generated since the
> moment we had upgrade. Here is the details and logs from usage.log. Can
> someone help me out what else do we need to change ot check. We have
> restarted both management and usage service multiple times.
> >
> > 1- Global settings -
> >
> > mysql> select * from configuration where name like '%usage%' and
> component = 'management-server';
> >
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> > | category | instance | component | name
> | value | description
> | default_value |
> updated | scope | is_dynamic |
> >
> +----------+----------+-------------------+-----------------------------------+-------+-------------------------------------------------------------------------------------------------------------------+---------------+---------+-------+------------+
> > | Usage | DEFAUL. | management-server | enable.usage.server
> | true | Flag for enabling usage
> | true |
> NULL | NULL | 0 |
> > | Advanced | DEFAULT | management-server | publish.usage.events
> | true | enable or disable publishing of usage events on the event
> bus | true |
> NULL | NULL | 0 |
> > | Usage | DEFAULT | management-server | usage.aggregation.timezone
> | UTC-3 | The timezone to use for usage stats aggregation
> | GMT |
> NULL | NULL | 0 |
> > | Usage | DEFAULT | management-server | usage.execution.timezone
> | UTC-3 | The timezone to use for usage job execution time
> | NULL |
> NULL | NULL | 0 |
> > | Usage | DEFAULT | management-server | usage.sanity.check.interval
> | NULL | Interval (in days) to check sanity of usage data
> | NULL |
> NULL | NULL | 0 |
> > | Usage | DEFAULT | management-server |
> usage.stats.job.aggregation.range | 60 | The range of time for
> aggregating the user statistics specified in minutes (e.g. 1440 for daily,
> 60 for hourly. | 1440 | NULL | NULL | 0 |
> > | Usage | DEFAULT | management-server | usage.stats.job.exec.time
> | 00:15 | The time at which the usage statistics aggregation job will
> run as an HH24:MM time, e.g. 00:30 to run at 12:30am. | 00:15 |
> NULL | NULL | 0 |
> >
> +----------+----------+-------------------+-----------------------------------+-------+—————————————————————————————————————————————————————————
> >
> > 2- Rpm installed -
> >
> >
> >
> > [root@acs-mgmt-01 ~]# rpm -qa| grep "cloudstack"
> > cloudstack-management-4.15.1.0-shapeblue0.el7.x86_64
> > cloudstack-mysql-ha-4.11.2.0-shapeblue.el7.x86_64
> > cloudstack-agent-4.15.1.0-shapeblue0.el7.x86_64
> > cloudstack-usage-4.15.1.0-shapeblue0.el7.x86_64
> > cloudstack-common-4.15.1.0-shapeblue0.el7.x86_64
> >
> >
> > 3- I am getting below logs in all usage.log -
> >
> >
> > 2022-03-29 13:21:01,102 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> > 2022-03-29 13:21:01,102 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
> (logid:) Checking to see if usage.vmops.pid exists.
> > 2022-03-29 13:21:01,103 INFO [cloud.usage.UsageManagerImpl] (main:null)
> (logid:) Implementation Version is 4.15.1.0
> > 2022-03-29 13:21:45,059 DEBUG [utils.crypt.EncryptionSecretKeyChecker]
> (main:null) (logid:) Encryption Type: file
> > 2022-03-29 13:21:45,239 INFO [db.Transaction.Transaction] (main:null)
> (logid:) Is Data Base High Availiability enabled? Ans : true
> > 2022-03-29 13:21:45,916 INFO [db.Transaction.Transaction] (main:null)
> (logid:) The slaves configured for Cloud Data base is/are : 198.18.8.8
> > 2022-03-29 13:21:46,344 DEBUG [utils.db.DriverLoader] (main:null)
> (logid:) Successfully loaded DB driver com.mysql.cj.jdbc.Driver
> > 2022-03-29 13:21:46,719 DEBUG [utils.db.DriverLoader] (main:null)
> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> > 2022-03-29 13:21:46,810 DEBUG [utils.db.DriverLoader] (main:null)
> (logid:) DB driver com.mysql.cj.jdbc.Driver was already loaded.
> > 2022-03-29 13:21:55,270 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Setup Spring Application context
> > 2022-03-29 13:21:56,947 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_bfbc13ef
> > 2022-03-29 13:21:56,948 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_4d85f5d9
> > 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c9f19855
> > 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_b10c0263
> > 2022-03-29 13:21:56,949 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_408a2306
> > 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_89749699
> > 2022-03-29 13:21:56,950 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_a36f3910
> > 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_ae855734
> > 2022-03-29 13:21:56,951 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_cb1733ab
> > 2022-03-29 13:21:56,954 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_8d96e26a
> > 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_a05d1c55
> > 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_6f0406e6
> > 2022-03-29 13:21:56,955 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_dff13e2a
> > 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_f9d661d9
> > 2022-03-29 13:21:56,958 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_43e0fb59
> > 2022-03-29 13:21:56,959 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_4cc635cb
> > 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_48492877
> > 2022-03-29 13:21:56,960 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_ae87651c
> > 2022-03-29 13:21:56,961 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_a9feedbc
> > 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_66484b16
> > 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_f129d147
> > 2022-03-29 13:21:56,962 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_78d12e3d
> > 2022-03-29 13:21:56,963 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.ExternalPublicIpStatisticsDaoImpl_EnhancerByCloudStack_6e93cdb5
> > 2022-03-29 13:21:56,964 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_13ca285f
> > 2022-03-29 13:21:56,969 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.configuration.dao.ResourceLimitDaoImpl_EnhancerByCloudStack_66889b3f
> > 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.alert.dao.AlertDaoImpl_EnhancerByCloudStack_ee19cc42
> > 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_dfca8d86
> > 2022-03-29 13:21:56,970 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.VmDiskStatisticsDaoImpl_EnhancerByCloudStack_b91503da
> > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsagePortForwardingRuleDaoImpl_EnhancerByCloudStack_acd6e53e
> > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_4d93146e
> > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_8eb3b09e
> > 2022-03-29 13:21:56,971 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_70285a6
> > 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_3fddafd8
> > 2022-03-29 13:21:56,972 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageVMSnapshotOnPrimaryDaoImpl_EnhancerByCloudStack_c9f85d24
> > 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageBackupDaoImpl_EnhancerByCloudStack_4705c1b6
> > 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageSecurityGroupDaoImpl_EnhancerByCloudStack_79306495
> > 2022-03-29 13:21:56,973 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageIPAddressDaoImpl_EnhancerByCloudStack_6bfb9717
> > 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaAccountDaoImpl_EnhancerByCloudStack_b35444be
> > 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_63681c89
> > 2022-03-29 13:21:56,974 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_b7b85ce8
> > 2022-03-29 13:21:56,975 INFO [utils.component.ComponentContext]
> (main:null) (logid:) Configuring
> com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_92a2355f
> > 2022-03-29 13:21:56,975 DEBUG [cloud.usage.UsageManagerImpl] (main:null)
> (logid:) Checking to see if usage.vmops.pid exists.
> > 2022-03-29 13:21:56,975 INFO [cloud.usage.UsageManagerImpl] (main:null)
> (logid:) Implementation Version is 4.15.1.0
> >
> >
> >
> > Regards,
> > Vivek Kumar
> >
> >
> >
>
>
> --
> This message is intended only for the use of the individual or entity to
> which it is addressed and may contain confidential and/or privileged
> information. If you are not the intended recipient, please delete the
> original message and any copy of it from your computer system. You are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited unless proper authorization has been
> obtained for such action. If you have received this communication in
> error,
> please notify the sender immediately. Although IndiQus attempts to sweep
> e-mail and attachments for viruses, it does not guarantee that both are
> virus-free and accepts no liability for any damage sustained as a result
> of
> viruses.
>
--
Daan