You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Sean Lair <sl...@ippathways.com.INVALID> on 2025/02/07 23:48:09 UTC
Re: Async Jobs
Wondering if any other insights into async_job entries with NULL job_cmd and never have a job_complete_msid. Here is some detailed info:
MariaDB [cloud]> select id, job_cmd,job_status,last_updated,created,job_complete_msid from async_job order by job_complete_msid,last_updated,created;
+--------+-----------------------------------------------------------------------+------------+---------------------+---------------------+-------------------+
| id | job_cmd | job_status | last_updated | created | job_complete_msid |
+--------+-----------------------------------------------------------------------+------------+---------------------+---------------------+-------------------+
| 626231 | NULL | 0 | NULL | 2025-01-24 04:58:57 | NULL |
| 626236 | NULL | 0 | NULL | 2025-01-24 04:58:57 | NULL |
| 627366 | NULL | 0 | NULL | 2025-01-30 20:19:51 | NULL |
| 629751 | NULL | 0 | NULL | 2025-02-06 04:58:08 | NULL |
MariaDB [cloud]> select uuid from async_job where id=629751;
+--------------------------------------+
| uuid |
+--------------------------------------+
| 7981ecf8-4f65-419c-8f1d-0f8289b9065a |
Output from queryAsyncJobResult:
https://xxxx/client/api?command=queryAsyncJobResult&jobid=7981ecf8-4f65-419c-8f1d-0f8289b9065a
<queryasyncjobresultresponse cloud-stack-version="4.15.2.0">
<accountid>275f3f54-e249-11e5-90e8-005056a324ff</accountid>
<userid>275f4b7b-e249-11e5-90e8-005056a324ff</userid>
<jobstatus>0</jobstatus>
<jobprocstatus>0</jobprocstatus>
<jobresultcode>0</jobresultcode>
<created>2025-02-06T04:58:08+0000</created>
<jobid>7981ecf8-4f65-419c-8f1d-0f8289b9065a</jobid>
</queryasyncjobresultresponse>
MariaDB [cloud]> select * from user where user.uuid ='275f4b7b-e249-11e5-90e8-005056a324ff'\G
*************************** 1. row ***************************
id: 1
uuid: 275f4b7b-e249-11e5-90e8-005056a324ff
username: system
password: 0.9057436724246887
account_id: 1
firstname: system
lastname: cloud
email: NULL
state: enabled
api_key: NULL
secret_key: NULL
created: 2016-03-04 20:39:12
removed: NULL
timezone: NULL
registration_token: NULL
is_registered: 0
incorrect_login_attempts: 0
default: 1
source: UNKNOWN
external_entity: NULL
MariaDB [cloud]> select * from account where uuid='275f3f54-e249-11e5-90e8-005056a324ff'\G
*************************** 1. row ***************************
id: 1
account_name: system
uuid: 275f3f54-e249-11e5-90e8-005056a324ff
type: 1
role_id: 1
domain_id: 1
state: enabled
removed: NULL
cleanup_needed: 0
network_domain: NULL
default_zone_id: NULL
default: 1
Some excerpts from the mgmt. server log:
2025-02-06 04:58:08,831 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-2f094c04) (logid:9255bef7) Execute sync-queue item: SyncQueueItemVO {id:33711, queueId: 281451, contentType: AsyncJob, contentId: 629756, lastProcessMsid: 345050927939, lastprocessNumber: 6, lastProcessTime: Thu Feb 06 04:58:08 UTC 2025, created: Thu Feb 06 04:58:08 UTC 2025}
2025-02-06 04:58:08,832 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-2f094c04) (logid:9255bef7) Schedule queued job-629756
2025-02-06 04:58:08,839 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:ca42fda8) Add job-629756 into job monitoring
2025-02-06 04:58:08,845 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Executing AsyncJobVO {id:629756, userId: 1, accountId: 1, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo: rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAQRXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345050908593, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Feb 06 04:58:08 UTC 2025, removed: null}
2025-02-06 04:58:08,848 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Run VM work job: com.cloud.vm.VmWorkStop for VM 4165, job origin: 629751
2025-02-06 04:58:08,849 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Execute VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":4165,"handlerName":"VirtualMachineManagerImpl"}
2025-02-06 04:58:08,852 DEBUG [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Stopped called on VM[User|i-2806-4165-VM] but the state is Destroyed
2025-02-06 04:58:08,852 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Done executing VM work job: com.cloud.vm.VmWorkStop{"cleanup":false,"userId":1,"accountId":1,"vmId":4165,"handlerName":"VirtualMachineManagerImpl"}
2025-02-06 04:58:08,852 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Complete async job-629756, jobStatus: SUCCEEDED, resultCode: 0, result: null
2025-02-06 04:58:08,853 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Publish async job-629756 complete on message bus
2025-02-06 04:58:08,853 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Wake up jobs related to job-629756
2025-02-06 04:58:08,853 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Update db status for job-629756
2025-02-06 04:58:08,854 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756 ctx-ce62fe80) (logid:7981ecf8) Wake up jobs joined with job-629756 and disjoin all subjobs created from job- 629756
2025-02-06 04:58:08,860 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 4165, job origin: 629751
2025-02-06 04:58:08,860 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Done executing com.cloud.vm.VmWorkStop for job-629756
2025-02-06 04:58:08,861 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-31:ctx-e1be74d5 job-629751/job-629756) (logid:7981ecf8) Remove job-629756 from job monitoring
-----Original Message-----
From: Daan Hoogland <da...@gmail.com>
Sent: Monday, August 12, 2024 4:39 AM
To: users@cloudstack.apache.org
Subject: [CAUTION: SUSPECT SENDER] Re: Async Jobs
Warning! This message was sent from outside your organization and we were unable to verify the sender.
Steve,
Did you get to the bottom of this?
On Tue, May 28, 2024 at 4:56 PM S.Fuller <st...@gmail.com> wrote:
>
> I have a question about asynchronous jobs. My understanding is that
> any async jobs will be recorded in the async_jobs table in the
> database. They should have a value in the job_complete_msid table when
> they are done, and then be removed 24 hours later.
>
> Every once in a while, we will have a job that shows up in the table,
> with a NULL value for job_complete_msid implying that the job is still running.
> However, when I see this, there's a null command for the job_cmd and
> the last_updated value as well.
>
> Should I be concerned, and do you have any suggestions on how to track
> down what might be causing these? I might not see one of these for 4
> or 5 months, so it's not a large volume.
>
> --
> Steve Fuller
> stevefuller@gmail.com
--
Daan