You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ugo Vasi <ug...@procne.it.INVALID> on 2019/02/06 09:59:53 UTC
Mysql got timeout and problems with UI
Hi all,
in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
network) I have some problems in cloudstack usage.
During the usual operations, I find a slowdown in the responses of the
UI and occasionally errors.
This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of a
specific network (net-name).
The UI has stopped in the assignment phase even if in the logs (which I
report at the bottom of the email) I see that the job is executed in a
few seconds and updating the list of associated IPs in the network list
I see that the association is correct.
I noticed that a series of these messages appear in the mysql log:
2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db: 'dbname' user: 'username' host: 'localhost' (Got timeout reading communication packets)
2019-02-06T00:51:59.352203Z 6269 [Note] Aborted connection 6269 to db: 'dbname' user: 'username' host: 'localhost' (Got timeout reading communication packets)
2019-02-06T00:51:59.421092Z 6289 [Note] Aborted connection 6289 to db: 'dbname' user: 'username' host: 'localhost' (Got timeout reading communication packets)
2019-02-06T08:00:35.717673Z 6101 [Note] Aborted connection 6101 to db: 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading communication packets
Note that the mysql server is used only by cloustack and is configured
according to the instructions in the cloudstack manual.
I've already had a number of similar problems before (the version of ACS
was 4.11.1) and the situation has been getting worse and worse until the
interface and the whole cloudstack system has stopped working (see
"URGENT: Unable to schedule async job for command
com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to
apply firewall rules on router").
Has anyone had similar experiences or can help me understand where the
problem is?
Job log (extracted from management-server.log):
2019-02-06 10:21:07,282 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876
into job monitoring 2019-02-06 10:21:07,294 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2,
accountId: 2, instanceType: IpAddress, instanceId: 31, cmd:
org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin,
cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface
com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface
com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null} 2019-02-06
10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit
async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2,
instanceType: IpAddress, instanceId: 31, cmd:
org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin,
cmdInfo:
{"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface
com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface
com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null} 2019-02-06
10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl]
(API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) (logid:ce8dc8fb)
Associating ip Ip[xx.xx.xx.xx-1] to network Ntwk[206|Guest|8] 2019-02-06
10:21:07,419 DEBUG [c.c.r.ResourceLimitManagerImpl]
(API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) (logid:ce8dc8fb)
Updating resource Type = public_ip count for Account = 2 Operation =
increasing Amount = 1 2019-02-06 10:21:07,426 DEBUG
[c.c.n.IpAddressManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a job-876
ctx-462aefc2) (logid:ce8dc8fb) Successfully associated ip address
xx.xx.xx.xx to network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876,
jobStatus: SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","allocated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88d7d429107","zonename":"somezone","issourcenat":false,"account":"admin","domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","forvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4","vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"associatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatednetworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-78d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true}
2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) (logid:ce8dc8fb)
Publish async job-876 complete on message bus 2019-02-06 10:21:07,441
DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
job-876 ctx-462aefc2) (logid:ce8dc8fb) Wake up jobs related to job-876
2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) (logid:ce8dc8fb)
Update db status for job-876 2019-02-06 10:21:07,442 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
job-876 ctx-462aefc2) (logid:ce8dc8fb) Wake up jobs joined with job-876
and disjoin all subjobs created from job- 876 2019-02-06 10:21:07,446
DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
job-876) (logid:ce8dc8fb) Done executing
org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin
for job-876 2019-02-06 10:21:07,447 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:ce8dc8fb) Remove
job-876 from job monitoring
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
RE: Mysql got timeout and problems with UI
Posted by Andrija Panic <an...@shapeblue.com>.
Ugo,
Please do proper log analysis, otherwise we can only guess.
If you restart network with clean up - that means a BRAND NEW VR was created for your networks..
That sounds like something was wrong with old one.
And that needs investigation of log (mgmt and agent) and/or inside VR, before you destroy it.
I don't get the DNS part of your reply...
Kind regards,
Andrija
andrija.panic@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-----Original Message-----
From: Ugo Vasi <ug...@procne.it>
Sent: 07 February 2019 11:47
To: users@cloudstack.apache.org; Andrija Panic <an...@shapeblue.com>
Subject: Re: Mysql got timeout and problems with UI
Hi Adrija,
I restarted the network with the appropriate button (enabling the clean
option) and then the firewall responds correctly, this makes me think that it is not a problem of communication between browsers and mgmt.
It would seem to me a DNS problem that delays the answers, how can I check the calls that the software agent makes inside the router?
Is there a query to count the number of rules of that firewall or for ACL? I've never used more than 15 rules for single IP.
Il 07/02/19 09:19, Andrija Panic ha scritto:
> Hi Ugo,
>
> I have never observed such behavior in any release so far (general sluggishness of GUI) - I hope you don't mind me asking you to make sure that your network connectivity is good enough and so on, between browser/API client and the mgmt server itself (and of course between mgmt and mysql server) - reason being that in some of my ex companies I got one (high level) guy complaining on lost packet when doing ping to the DC, but it was his WiFi which has problems (once moved to wire, everything magically worked).
>
> To be fair, I did see dead gui when a customer has been abusing the ACL rules, i.e. had like 200 rules on single ACL, and browser could not handle showing it and it was completely unmanageable.
>
> As for other issue, this seems to be something you had before (if I remember ML correctly) - please check the logs inside VR and any lines in mgmt. log ABOVE that final error string (usually it will tell you that i.e. ssh connection to VR could not be done).
>
> Also worth checking Agent logs on specific host where VR is running (make sure to increase log level to debug first)
> For MySQL issues, again check network, and make sure no LB with short TCP timeout is set in between.
>
> Kind regards,
> Andrija
>
> andrija.panic@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Ugo Vasi <ug...@procne.it.INVALID>
> Sent: 06 February 2019 14:53
> To: users@cloudstack.apache.org
> Subject: Re: Mysql got timeout and problems with UI
>
> Hi all,
> adding a new firewall rule I receive this error (note, after 1189 seconds):
>
> 2019-02-06 14:42:08,441 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending for 1189 seconds
> 2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Seq 7-4193977152988777787: Received: { Ans: , MgmtId: 220777304233416,
> via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } }
> 2019-02-06 14:42:20,358 WARN [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Failed to apply firewall rules due to : Resource [DataCenter:1] is
> unreachable: Unable to apply firewall rules on router
> 2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete async job-886, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create firewall rule"}
>
>
> The rule now is in "Deleting" state.
>
>
> Il 06/02/19 10:59, Ugo Vasi ha scritto:
>> Hi all,
>> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
>> network) I have some problems in cloudstack usage.
>>
>> During the usual operations, I find a slowdown in the responses of the
>> UI and occasionally errors.
>>
>> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of
>> a specific network (net-name).
>>
>> The UI has stopped in the assignment phase even if in the logs (which
>> I report at the bottom of the email) I see that the job is executed in
>> a few seconds and updating the list of associated IPs in the network
>> list I see that the association is correct.
>>
>> I noticed that a series of these messages appear in the mysql log:
>>
>> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db:
>> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets) 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted
>> connection 6269 to db:
>> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets) 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted
>> connection 6289 to db:
>> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets) 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted
>> connection 6101 to db:
>> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets
>>
>>
>> Note that the mysql server is used only by cloustack and is configured
>> according to the instructions in the cloudstack manual.
>>
>> I've already had a number of similar problems before (the version of
>> ACS was 4.11.1) and the situation has been getting worse and worse
>> until the interface and the whole cloudstack system has stopped
>> working (see "URGENT: Unable to schedule async job for command
>> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to
>> apply firewall rules on router").
>>
>>
>> Has anyone had similar experiences or can help me understand where the
>> problem is?
>>
>>
>>
>> Job log (extracted from management-server.log):
>>
>> 2019-02-06 10:21:07,282 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876
>> into job monitoring 2019-02-06 10:21:07,294 DEBUG
>> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2,
>> accountId: 2, instanceType: IpAddress, instanceId: 31, cmd:
>> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
>> min,
>> cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0
>> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31",
>> "ctxDetails":"{\"interface
>> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\"
>> ,\"interface
>> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}"
>> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd
>> EventType":"NET.IPASSIGN","_":"1549444854378"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
>> 10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit
>> async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2,
>> instanceType: IpAddress, instanceId: 31, cmd:
>> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
>> min,
>> cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0
>> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31",
>> "ctxDetails":"{\"interface
>> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\"
>> ,\"interface
>> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}"
>> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd
>> EventType":"NET.IPASSIGN","_":"1549444854378"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
>> 10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Associating ip Ip[xx.xx.xx.xx-1] to network
>> Ntwk[206|Guest|8] 2019-02-06 10:21:07,419 DEBUG
>> [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876 ctx-462aefc2) (logid:ce8dc8fb) Updating resource Type =
>> public_ip count for Account = 2 Operation = increasing Amount = 1
>> 2019-02-06 10:21:07,426 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Successfully associated ip address xx.xx.xx.xx to
>> network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG
>> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876,
>> jobStatus: SUCCEEDED, resultCode: 0, result:
>> org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":"
>> a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","alloc
>> ated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88
>> d7d429107","zonename":"somezone","issourcenat":false,"account":"admin"
>> ,"domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","fo
>> rvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4",
>> "vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"ass
>> ociatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatedne
>> tworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d
>> 6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-7
>> 8d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true}
>> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Publish async job-876 complete on message bus
>> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Wake up jobs related to job-876 2019-02-06
>> 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Update db status for job-876 2019-02-06 10:21:07,442
>> DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Wake up jobs joined with job-876 and disjoin all
>> subjobs created from job- 876 2019-02-06 10:21:07,446 DEBUG
>> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876) (logid:ce8dc8fb) Done executing
>> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
>> min for job-876 2019-02-06 10:21:07,447 INFO
>> [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876) (logid:ce8dc8fb) Remove
>> job-876 from job monitoring
>>
>>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: Mysql got timeout and problems with UI
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Hi Adrija,
I restarted the network with the appropriate button (enabling the clean
option) and then the firewall responds correctly, this makes me think
that it is not a problem of communication between browsers and mgmt.
It would seem to me a DNS problem that delays the answers, how can I
check the calls that the software agent makes inside the router?
Is there a query to count the number of rules of that firewall or for
ACL? I've never used more than 15 rules for single IP.
Il 07/02/19 09:19, Andrija Panic ha scritto:
> Hi Ugo,
>
> I have never observed such behavior in any release so far (general sluggishness of GUI) - I hope you don't mind me asking you to make sure that your network connectivity is good enough and so on, between browser/API client and the mgmt server itself (and of course between mgmt and mysql server) - reason being that in some of my ex companies I got one (high level) guy complaining on lost packet when doing ping to the DC, but it was his WiFi which has problems (once moved to wire, everything magically worked).
>
> To be fair, I did see dead gui when a customer has been abusing the ACL rules, i.e. had like 200 rules on single ACL, and browser could not handle showing it and it was completely unmanageable.
>
> As for other issue, this seems to be something you had before (if I remember ML correctly) - please check the logs inside VR and any lines in mgmt. log ABOVE that final error string (usually it will tell you that i.e. ssh connection to VR could not be done).
>
> Also worth checking Agent logs on specific host where VR is running (make sure to increase log level to debug first)
> For MySQL issues, again check network, and make sure no LB with short TCP timeout is set in between.
>
> Kind regards,
> Andrija
>
> andrija.panic@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
>
> -----Original Message-----
> From: Ugo Vasi <ug...@procne.it.INVALID>
> Sent: 06 February 2019 14:53
> To: users@cloudstack.apache.org
> Subject: Re: Mysql got timeout and problems with UI
>
> Hi all,
> adding a new firewall rule I receive this error (note, after 1189 seconds):
>
> 2019-02-06 14:42:08,441 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending for 1189 seconds
> 2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Seq 7-4193977152988777787: Received: { Ans: , MgmtId: 220777304233416,
> via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } }
> 2019-02-06 14:42:20,358 WARN [c.c.n.f.FirewallManagerImpl]
> (API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Failed to apply firewall rules due to : Resource [DataCenter:1] is
> unreachable: Unable to apply firewall rules on router
> 2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete async job-886, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create firewall rule"}
>
>
> The rule now is in "Deleting" state.
>
>
> Il 06/02/19 10:59, Ugo Vasi ha scritto:
>> Hi all,
>> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
>> network) I have some problems in cloudstack usage.
>>
>> During the usual operations, I find a slowdown in the responses of the
>> UI and occasionally errors.
>>
>> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of
>> a specific network (net-name).
>>
>> The UI has stopped in the assignment phase even if in the logs (which
>> I report at the bottom of the email) I see that the job is executed in
>> a few seconds and updating the list of associated IPs in the network
>> list I see that the association is correct.
>>
>> I noticed that a series of these messages appear in the mysql log:
>>
>> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db:
>> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets) 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted
>> connection 6269 to db:
>> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets) 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted
>> connection 6289 to db:
>> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets) 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted
>> connection 6101 to db:
>> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading
>> communication packets
>>
>>
>> Note that the mysql server is used only by cloustack and is configured
>> according to the instructions in the cloudstack manual.
>>
>> I've already had a number of similar problems before (the version of
>> ACS was 4.11.1) and the situation has been getting worse and worse
>> until the interface and the whole cloudstack system has stopped
>> working (see "URGENT: Unable to schedule async job for command
>> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to
>> apply firewall rules on router").
>>
>>
>> Has anyone had similar experiences or can help me understand where the
>> problem is?
>>
>>
>>
>> Job log (extracted from management-server.log):
>>
>> 2019-02-06 10:21:07,282 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876
>> into job monitoring 2019-02-06 10:21:07,294 DEBUG
>> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2,
>> accountId: 2, instanceType: IpAddress, instanceId: 31, cmd:
>> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
>> min,
>> cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0
>> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31",
>> "ctxDetails":"{\"interface
>> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\"
>> ,\"interface
>> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}"
>> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd
>> EventType":"NET.IPASSIGN","_":"1549444854378"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
>> 10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit
>> async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2,
>> instanceType: IpAddress, instanceId: 31, cmd:
>> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
>> min,
>> cmdInfo:
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0
>> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31",
>> "ctxDetails":"{\"interface
>> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\"
>> ,\"interface
>> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}"
>> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd
>> EventType":"NET.IPASSIGN","_":"1549444854378"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
>> 10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Associating ip Ip[xx.xx.xx.xx-1] to network
>> Ntwk[206|Guest|8] 2019-02-06 10:21:07,419 DEBUG
>> [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876 ctx-462aefc2) (logid:ce8dc8fb) Updating resource Type =
>> public_ip count for Account = 2 Operation = increasing Amount = 1
>> 2019-02-06 10:21:07,426 DEBUG [c.c.n.IpAddressManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Successfully associated ip address xx.xx.xx.xx to
>> network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG
>> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876,
>> jobStatus: SUCCEEDED, resultCode: 0, result:
>> org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":"
>> a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","alloc
>> ated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88
>> d7d429107","zonename":"somezone","issourcenat":false,"account":"admin"
>> ,"domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","fo
>> rvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4",
>> "vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"ass
>> ociatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatedne
>> tworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d
>> 6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-7
>> 8d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true}
>> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Publish async job-876 complete on message bus
>> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Wake up jobs related to job-876 2019-02-06
>> 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Update db status for job-876 2019-02-06 10:21:07,442
>> DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
>> (logid:ce8dc8fb) Wake up jobs joined with job-876 and disjoin all
>> subjobs created from job- 876 2019-02-06 10:21:07,446 DEBUG
>> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876) (logid:ce8dc8fb) Done executing
>> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
>> min for job-876 2019-02-06 10:21:07,447 INFO
>> [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-6:ctx-5d20ed8a
>> job-876) (logid:ce8dc8fb) Remove
>> job-876 from job monitoring
>>
>>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
RE: Mysql got timeout and problems with UI
Posted by Andrija Panic <an...@shapeblue.com>.
Hi Ugo,
I have never observed such behavior in any release so far (general sluggishness of GUI) - I hope you don't mind me asking you to make sure that your network connectivity is good enough and so on, between browser/API client and the mgmt server itself (and of course between mgmt and mysql server) - reason being that in some of my ex companies I got one (high level) guy complaining on lost packet when doing ping to the DC, but it was his WiFi which has problems (once moved to wire, everything magically worked).
To be fair, I did see dead gui when a customer has been abusing the ACL rules, i.e. had like 200 rules on single ACL, and browser could not handle showing it and it was completely unmanageable.
As for other issue, this seems to be something you had before (if I remember ML correctly) - please check the logs inside VR and any lines in mgmt. log ABOVE that final error string (usually it will tell you that i.e. ssh connection to VR could not be done).
Also worth checking Agent logs on specific host where VR is running (make sure to increase log level to debug first)
For MySQL issues, again check network, and make sure no LB with short TCP timeout is set in between.
Kind regards,
Andrija
andrija.panic@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-----Original Message-----
From: Ugo Vasi <ug...@procne.it.INVALID>
Sent: 06 February 2019 14:53
To: users@cloudstack.apache.org
Subject: Re: Mysql got timeout and problems with UI
Hi all,
adding a new firewall rule I receive this error (note, after 1189 seconds):
2019-02-06 14:42:08,441 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending for 1189 seconds
2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request]
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Seq 7-4193977152988777787: Received: { Ans: , MgmtId: 220777304233416,
via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } }
2019-02-06 14:42:20,358 WARN [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) Failed to apply firewall rules due to : Resource [DataCenter:1] is
unreachable: Unable to apply firewall rules on router
2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete async job-886, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create firewall rule"}
The rule now is in "Deleting" state.
Il 06/02/19 10:59, Ugo Vasi ha scritto:
> Hi all,
> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
> network) I have some problems in cloudstack usage.
>
> During the usual operations, I find a slowdown in the responses of the
> UI and occasionally errors.
>
> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of
> a specific network (net-name).
>
> The UI has stopped in the assignment phase even if in the logs (which
> I report at the bottom of the email) I see that the job is executed in
> a few seconds and updating the list of associated IPs in the network
> list I see that the association is correct.
>
> I noticed that a series of these messages appear in the mysql log:
>
> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets) 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted
> connection 6269 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets) 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted
> connection 6289 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets) 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted
> connection 6101 to db:
> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading
> communication packets
>
>
> Note that the mysql server is used only by cloustack and is configured
> according to the instructions in the cloudstack manual.
>
> I've already had a number of similar problems before (the version of
> ACS was 4.11.1) and the situation has been getting worse and worse
> until the interface and the whole cloudstack system has stopped
> working (see "URGENT: Unable to schedule async job for command
> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to
> apply firewall rules on router").
>
>
> Has anyone had similar experiences or can help me understand where the
> problem is?
>
>
>
> Job log (extracted from management-server.log):
>
> 2019-02-06 10:21:07,282 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876
> into job monitoring 2019-02-06 10:21:07,294 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2,
> accountId: 2, instanceType: IpAddress, instanceId: 31, cmd:
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
> min,
> cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0
> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31",
> "ctxDetails":"{\"interface
> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\"
> ,\"interface
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}"
> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd
> EventType":"NET.IPASSIGN","_":"1549444854378"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
> 10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit
> async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2,
> instanceType: IpAddress, instanceId: 31, cmd:
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
> min,
> cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0
> cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31",
> "ctxDetails":"{\"interface
> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\"
> ,\"interface
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}"
> ,"ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmd
> EventType":"NET.IPASSIGN","_":"1549444854378"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
> 10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Associating ip Ip[xx.xx.xx.xx-1] to network
> Ntwk[206|Guest|8] 2019-02-06 10:21:07,419 DEBUG
> [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Updating resource Type =
> public_ip count for Account = 2 Operation = increasing Amount = 1
> 2019-02-06 10:21:07,426 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Successfully associated ip address xx.xx.xx.xx to
> network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876,
> jobStatus: SUCCEEDED, resultCode: 0, result:
> org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":"
> a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","alloc
> ated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88
> d7d429107","zonename":"somezone","issourcenat":false,"account":"admin"
> ,"domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","fo
> rvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4",
> "vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"ass
> ociatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatedne
> tworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d
> 6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-7
> 8d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true}
> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Publish async job-876 complete on message bus
> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Wake up jobs related to job-876 2019-02-06
> 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Update db status for job-876 2019-02-06 10:21:07,442
> DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Wake up jobs joined with job-876 and disjoin all
> subjobs created from job- 876 2019-02-06 10:21:07,446 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Done executing
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAd
> min for job-876 2019-02-06 10:21:07,447 INFO
> [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Remove
> job-876 from job monitoring
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone od alla Società sopraindicati. La diffusione, distribuzione e/o copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003 "Codice in materia di protezione dei dati personali". Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di informare immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail info@procne.it <ma...@procne.it>.
Re: Mysql got timeout and problems with UI
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Hi all,
adding a new firewall rule I receive this error (note, after 1189 seconds):
2019-02-06 14:42:08,441 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending
for 1189 seconds
2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request]
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266)
Seq 7-4193977152988777787: Received: { Ans: , MgmtId: 220777304233416,
via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } }
2019-02-06 14:42:20,358 WARN [c.c.n.f.FirewallManagerImpl]
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266)
Failed to apply firewall rules due to : Resource [DataCenter:1] is
unreachable: Unable to apply firewall rules on router
2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete
async job-886, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create firewall rule"}
The rule now is in "Deleting" state.
Il 06/02/19 10:59, Ugo Vasi ha scritto:
> Hi all,
> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
> network) I have some problems in cloudstack usage.
>
> During the usual operations, I find a slowdown in the responses of the
> UI and occasionally errors.
>
> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of
> a specific network (net-name).
>
> The UI has stopped in the assignment phase even if in the logs (which
> I report at the bottom of the email) I see that the job is executed in
> a few seconds and updating the list of associated IPs in the network
> list I see that the association is correct.
>
> I noticed that a series of these messages appear in the mysql log:
>
> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets)
> 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted connection 6269 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets)
> 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted connection 6289 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets)
> 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted connection 6101 to db:
> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading
> communication packets
>
>
> Note that the mysql server is used only by cloustack and is configured
> according to the instructions in the cloudstack manual.
>
> I've already had a number of similar problems before (the version of
> ACS was 4.11.1) and the situation has been getting worse and worse
> until the interface and the whole cloudstack system has stopped
> working (see "URGENT: Unable to schedule async job for command
> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to
> apply firewall rules on router").
>
>
> Has anyone had similar experiences or can help me understand where the
> problem is?
>
>
>
> Job log (extracted from management-server.log):
>
> 2019-02-06 10:21:07,282 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876
> into job monitoring 2019-02-06 10:21:07,294 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2,
> accountId: 2, instanceType: IpAddress, instanceId: 31, cmd:
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin,
> cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface
> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
> 10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit
> async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2,
> instanceType: IpAddress, instanceId: 31, cmd:
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin,
> cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface
> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
> 10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Associating ip Ip[xx.xx.xx.xx-1] to network
> Ntwk[206|Guest|8] 2019-02-06 10:21:07,419 DEBUG
> [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Updating resource Type =
> public_ip count for Account = 2 Operation = increasing Amount = 1
> 2019-02-06 10:21:07,426 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Successfully associated ip address xx.xx.xx.xx to
> network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876,
> jobStatus: SUCCEEDED, resultCode: 0, result:
> org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","allocated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88d7d429107","zonename":"somezone","issourcenat":false,"account":"admin","domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","forvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4","vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"associatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatednetworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-78d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true}
> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Publish async job-876 complete on message bus
> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Wake up jobs related to job-876 2019-02-06
> 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Update db status for job-876 2019-02-06 10:21:07,442
> DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Wake up jobs joined with job-876 and disjoin all
> subjobs created from job- 876 2019-02-06 10:21:07,446 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Done executing
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin
> for job-876 2019-02-06 10:21:07,447 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:ce8dc8fb) Remove
> job-876 from job monitoring
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: Mysql got timeout and problems with UI
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Another strange details that are repeated periodically (every 30
minutes) are these message on syslog:
Feb 6 10:50:01 WARN [network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cff1f828) (logid:9a5ee6a7) Invalid last alert
timestamp received while collecting alerts from router: r-4-VM
Feb 6 10:50:01 WARN [network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:ctx-cff1f828) (logid:9a5ee6a7) Invalid last alert
timestamp received while collecting alerts from router: r-6-VM
Il 06/02/19 10:59, Ugo Vasi ha scritto:
> Hi all,
> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced
> network) I have some problems in cloudstack usage.
>
> During the usual operations, I find a slowdown in the responses of the
> UI and occasionally errors.
>
> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of
> a specific network (net-name).
>
> The UI has stopped in the assignment phase even if in the logs (which
> I report at the bottom of the email) I see that the job is executed in
> a few seconds and updating the list of associated IPs in the network
> list I see that the association is correct.
>
> I noticed that a series of these messages appear in the mysql log:
>
> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets)
> 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted connection 6269 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets)
> 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted connection 6289 to db:
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading
> communication packets)
> 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted connection 6101 to db:
> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading
> communication packets
>
>
> Note that the mysql server is used only by cloustack and is configured
> according to the instructions in the cloudstack manual.
>
> I've already had a number of similar problems before (the version of
> ACS was 4.11.1) and the situation has been getting worse and worse
> until the interface and the whole cloudstack system has stopped
> working (see "URGENT: Unable to schedule async job for command
> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to
> apply firewall rules on router").
>
>
> Has anyone had similar experiences or can help me understand where the
> problem is?
>
>
>
> Job log (extracted from management-server.log):
>
> 2019-02-06 10:21:07,282 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876
> into job monitoring 2019-02-06 10:21:07,294 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2,
> accountId: 2, instanceType: IpAddress, instanceId: 31, cmd:
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin,
> cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface
> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
> 10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit
> async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2,
> instanceType: IpAddress, instanceId: 31, cmd:
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin,
> cmdInfo:
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface
> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06
> 10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Associating ip Ip[xx.xx.xx.xx-1] to network
> Ntwk[206|Guest|8] 2019-02-06 10:21:07,419 DEBUG
> [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Updating resource Type =
> public_ip count for Account = 2 Operation = increasing Amount = 1
> 2019-02-06 10:21:07,426 DEBUG [c.c.n.IpAddressManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Successfully associated ip address xx.xx.xx.xx to
> network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876,
> jobStatus: SUCCEEDED, resultCode: 0, result:
> org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","allocated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88d7d429107","zonename":"somezone","issourcenat":false,"account":"admin","domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","forvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4","vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"associatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatednetworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-78d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true}
> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Publish async job-876 complete on message bus
> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Wake up jobs related to job-876 2019-02-06
> 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Update db status for job-876 2019-02-06 10:21:07,442
> DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2)
> (logid:ce8dc8fb) Wake up jobs joined with job-876 and disjoin all
> subjobs created from job- 876 2019-02-06 10:21:07,446 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a
> job-876) (logid:ce8dc8fb) Done executing
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin
> for job-876 2019-02-06 10:21:07,447 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:ce8dc8fb) Remove
> job-876 from job monitoring
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.