You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ugo Vasi <ug...@procne.it.INVALID> on 2018/12/06 08:49:05 UTC
Re: urgent: Unable to apply firewall rules on router
Hi all,
I still have this problem, the same router that blocked me the first
time has also blocked a second time and now another router that has been
operating for a few months has slowed down a lot (it did not give me
errors).
this is the job log of inserting a new firewall rule in the router:
2018-12-05 09:01:27,579 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-100:ctx-49a89073 job-1664) (logid:fd9d8cd1) Add
job-1664 into job monitoring
2018-12-05 09:01:27,589 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp1096283470-926:ctx-f607bc6e ctx-75946dbc) (logid:16270ff5) submit
async job-1664, details: AsyncJobVO {id:1664, userId: 2, accountId: 2,
instanceType: FirewallRule, instanceId: 359, cmd:
org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd,
cmdInfo:
{"startport":"80","ipaddressid":"5e93afc0-43a0-483b-8086-580162b98c2d","httpmethod":"GET","ctxAccountId":"2","uuid":"4ae0123b-8596-4ed8-9466-43e725679146","cmdEventType":"FIREWALL.OPEN","cidrlist":"0.0.0.0/0","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEventId":"6003","id":"359","ctxDetails":"{\"interface
com.cloud.network.rules.FirewallRule\":\"4ae0123b-8596-4ed8-9466-43e725679146\",\"interface
com.cloud.network.IpAddress\":\"5e93afc0-43a0-483b-8086-580162b98c2d\"}","_":"1543996887133"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2018-12-05 09:01:27,590 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-100:ctx-49a89073 job-1664) (logid:7109e74e) Executing
AsyncJobVO {id:1664, userId: 2, accountId: 2, instanceType:
FirewallRule, instanceId: 359, cmd:
org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd,
cmdInfo:
{"startport":"80","ipaddressid":"5e93afc0-43a0-483b-8086-580162b98c2d","httpmethod":"GET","ctxAccountId":"2","uuid":"4ae0123b-8596-4ed8-9466-43e725679146","cmdEventType":"FIREWALL.OPEN","cidrlist":"0.0.0.0/0","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEventId":"6003","id":"359","ctxDetails":"{\"interface
com.cloud.network.rules.FirewallRule\":\"4ae0123b-8596-4ed8-9466-43e725679146\",\"interface
com.cloud.network.IpAddress\":\"5e93afc0-43a0-483b-8086-580162b98c2d\"}","_":"1543996887133"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 220777304233416, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}
2018-12-05 09:01:27,650 DEBUG [o.a.c.n.t.BasicNetworkTopology]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) APPLYING FIREWALL RULES
2018-12-05 09:01:27,650 DEBUG [o.a.c.n.t.BasicNetworkTopology]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Applying firewall rules in network Ntwk[205|Guest|8]
2018-12-05 09:01:27,668 DEBUG [c.c.a.t.Request]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Seq 4-8314770812033476439: Sending { Cmd , MgmtId:
220777304233416, via: 4(cshp132), Ver: v1, Flags: 100001,
[{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":356,"srcIp":"yy.yy.yy.43","protocol":"tcp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":true,"sourceCidrList":["xx.xx.xx.xx/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":357,"srcIp":"yy.yy.yy.43","protocol":"tcp","srcPortRange":[443,443],"revoked":false,"alreadyAdded":true,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":358,"srcIp":"yy.yy.yy.43","protocol":"udp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":true,"sourceCidrList":["xx.xx.xx.xx/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":359,"srcIp":"yy.yy.yy.43","protocol":"tcp","srcPortRange":[80,80],"revoked":false,"alreadyAdded":false,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"router.name":"r-29-VM","router.guest.ip":"10.11.12.1","router.ip":"169.254.3.228","zone.network.type":"Advanced","firewall.egress.default":"false"},"wait":0}}]
}
2018-12-05 09:03:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-9f534fa1) (logid:ef1d6d8f) Task (job-1664) has been pending
for 106 seconds
2018-12-05 09:04:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-4dd690a2) (logid:86485fc4) Task (job-1664) has been pending
for 166 seconds
2018-12-05 09:05:14,182 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-1ac7a2bb) (logid:71fbc15d) Task (job-1664) has been pending
for 226 seconds
2018-12-05 09:06:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-14ff35d8) (logid:786dbd01) Task (job-1664) has been pending
for 286 seconds
2018-12-05 09:07:14,182 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-01f380a2) (logid:448af255) Task (job-1664) has been pending
for 346 seconds
2018-12-05 09:08:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-164e5d2d) (logid:f1d49e29) Task (job-1664) has been pending
for 406 seconds
2018-12-05 09:09:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-381ef9a0) (logid:eb42c557) Task (job-1664) has been pending
for 466 seconds
2018-12-05 09:10:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-0a5a6645) (logid:39ca12b1) Task (job-1664) has been pending
for 526 seconds
2018-12-05 09:11:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
(Timer-1:ctx-d0b5e6ca) (logid:421221d0) Task (job-1664) has been pending
for 586 seconds
2018-12-05 09:11:28,021 DEBUG [c.c.a.t.Request]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Seq 4-8314770812033476439: Received: { Ans: , MgmtId:
220777304233416, via: 4(cshp132), Ver: v1, Flags: 0, { GroupAnswer } }
2018-12-05 09:11:28,147 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Complete async job-1664, jobStatus: SUCCEEDED,
resultCode: 0, result:
org.apache.cloudstack.api.response.FirewallResponse/firewallrule/{"id":"4ae0123b-8596-4ed8-9466-43e725679146","protocol":"tcp","startport":80,"endport":80,"ipaddressid":"5e93afc0-43a0-483b-8086-580162b98c2d","networkid":"2f1e01d4-e7ce-4e2a-86f9-d9f32eaa66ca","ipaddress":"yy.yy.yy.43","state":"Active","cidrlist":"0.0.0.0/0","tags":[],"fordisplay":true}
2018-12-05 09:11:28,149 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Publish async job-1664 complete on message bus
2018-12-05 09:11:28,149 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Wake up jobs related to job-1664
2018-12-05 09:11:28,149 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Update db status for job-1664
2018-12-05 09:11:28,150 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
(logid:7109e74e) Wake up jobs joined with job-1664 and disjoin all
subjobs created from job- 1664
2018-12-05 09:11:28,154 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-100:ctx-49a89073 job-1664) (logid:7109e74e) Done
executing
org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
for job-1664
2018-12-05 09:11:28,155 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-100:ctx-49a89073 job-1664) (logid:7109e74e) Remove
job-1664 from job monitoring
n the router logs I see that the router management scripts (CsHelper.py,
CsRoute.py, CsAddress.py, etc) take about 5 seconds to do their job.
Il 10/11/18 11:47, Andrija Panic ha scritto:
> IF VR is part of VPC, then just restart VPC wtih "Clean Up" options - never
> restart VR (unless you really prefer to for some reason).
> If you are using Isolated Network setup (single VR and single network
> behind VR), then restart the Network, with "Clean UP" options.
> In both cases, a BRAND NEW router will be created (after previous one was
> destroyed)
>
> Interestingly, this is also good to know, when i.e. using custom VR
> systemVM template - i.e. there is bug (i.e. dnsmasq issue in past), you
> upload new systemVM template, define it as the default routing template in
> Global Settings... then you really need to restart VPC/Network with Clean
> UP in order to create BRAND NEW VR from whatever is the NEW template.
> Stopping and restarting/rebooting VR (which does recreate OS disk from
> scratch) - does it from the same linked template....(there are some
> workarrounds for this also...)
>
>
> Hope that help
> Andrija
>
> On Sat, 10 Nov 2018 at 10:12, Ugo Vasi<ug...@procne.it> wrote:
>
>> Hi Rafael,
>> in the file pippo.log I see messages similar to the following but also
>> appear in other virtual routers that work regularly:
>>
>> 2018-09-29 17:26:32,554 CsHelper.py execute:193 Command 'iptables -t
>> mangle -D PREROUTING -s xx.xx.xx.xx/32 -m state --state NEW -j CONNMARK'
>> returned non-zero exit status 2
>> 2018-09-29 17:26:32,554 CsNetfilter.py get_unseen:131 Delete rule -D
>> PREROUTING -s xx.xx.xx.xx/32 -m state --state NEW -j CONNMARK from table
>> mangle
>>
>> I tried to restart the router but the behavior was not changed.
>> Regenerating it instead seems to work well, the rules are added in a few
>> seconds.
>>
>> I noticed that by destroying a router with the appropriate button, the
>> manager does not automatically recreate it. I have to stop and restart a
>> machine that depends on that router to get it re-created.
>> Is there another procedure to automate the destruction and recreation of
>> the routers?
>>
>>
>> Thanks
>>
>>
>>
>> Il 09/11/18 12:29, Rafael Weingärtner ha scritto:
>>> Did you check the logs in the affected router?
>>>
>>> On Fri, Nov 9, 2018 at 9:28 AM Ugo Vasi<ug...@procne.it.invalid>
>> wrote:
>>>> Hi Glenn,
>>>> I tried to restart the manager but nothing changed. Note that this
>>>> behavior only occurs on this router, the others work regularly.
>>>> As soon as possible restart the router and see what happens.
>>>>
>>>> Thanks
>>>>
>>>> Il 08/11/18 19:36, Glenn Wagner ha scritto:
>>>>> Hi Ugo,
>>>>>
>>>>> Have you tried to just restart the management service to clear any
>>>> running tasks?
>>>>> And then try add the rules again.
>>>>>
>>>>> Regards
>>>>> Glenn Wagner
>>>>>
>>>>>
>>>>> glenn.wagner@shapeblue.com
>>>>> www.shapeblue.com
>>>>> Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset West, Cape
>>>> Town 7129South Africa
>>>>> @shapeblue
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Ugo Vasi<ug...@procne.it.INVALID>
>>>>> Sent: Thursday, 08 November 2018 5:33 PM
>>>>> To:users@cloudstack.apache.org; Andrija Panic <
>> andrija.panic@gmail.com>
>>>>> Subject: Re: urgent: Unable to apply firewall rules on router
>>>>>
>>>>> Hi Andrija,
>>>>> from the checks you have suggested I do not show up long running jobs.
>>>>>
>>>>> There are no error messages in the agent logs. By migrating the router,
>>>> the behavior has not changed.
>>>>> Doing further tests I found that the added rules become effective
>>>> immediately but the interface takes about 25 minutes to show it as
>> active.
>>>> A couple of times gave error:
>>>>> 2018-11-08 16:22:28,588 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-17:ctx-36b7f3eb job-942) (logid:a107efdf) Complete
>> async
>>>> job-942, jobStatus: FAILED, resultCode: 530, result:
>>>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>>> to create firewall rule"}
>>>>>
>>>>> When I delete a rule, it remains active until the status is updated and
>>>> then disappears (about 20 minutes after).
>>>>> Il 07/11/18 18:38, Andrija Panic ha scritto:
>>>>>> Hi Ugo,
>>>>>>
>>>>>> I have seen similar issues with i.e. starting a VM when there are
>>>>>> other long running jobs - check if there are any ongoing long jobs
>>>>>> already, that might be blocking the executioon of this job - i.e. long
>>>>>> running snapshots, or other thing.
>>>>>> I would also examine agent.log on the host where this VR is located -
>>>>>> there might be some traces there...
>>>>>>
>>>>>> Try this SQL to list aysnc jobs:
>>>>>>
>>>>>> select aj.id,
>>>>>> case when aj.job_status=1 then 'completed' when
>>>>>> aj.job_status=2 then 'progress' when aj.job_status=3 then 'error' end
>>>> as status,
>>>>>> aj.created, aj.last_updated, aj.related,
>>>>>> account.account_name, user.username, host.name as host, vm.name as
>>>> instance, vmj.step, aj.job_cmd
>>>>>> from async_job aj
>>>>>> inner join vm_work_job vmj on aj.id = vmj.id
>>>>>> left join vm_instance vm on vmj.vm_instance_id=vm.id
>>>>>> left join user on aj.user_id=user.id
>>>>>> left join account on aj.account_id=account.id
>>>>>> left join host on vm.host_id=host.id
>>>>>>
>>>>>> Alternatively, try to live-migrate VR to another host, and try to add
>>>>>> rule again.
>>>>>>
>>>>>> Cheers
>>>>>> Andrija
>>>>>>
>>>>>>
>>>>>> On Wed, 7 Nov 2018 at 17:59, Ugo Vasi<ug...@procne.it.invalid>
>>>> wrote:
>>>>>>> Hi all,
>>>>>>> I'm having a problem when I try to insert a firewall rule of an
>>>>>>> address connected to a new VM of a Guest Isolated Network.
>>>>>>>
>>>>>>> After a while the job is removed as FAILED. I try to repeat the
>>>>>>> operation but the problem remains. How can I unblock the situation?
>>>>>>>
>>>>>>> here it is the log of job-927:
>>>>>>>
>>>>>>> 2018-11-07 17:16:45,256 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0787853c) Add
>>>>>>> job-927 into job monitoring
>>>>>>> 2018-11-07 17:16:45,279 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Executing
>>>>>>> AsyncJobVO {id:927, userId: 2, accountId: 2, instanceType:
>>>>>>> FirewallRule,
>>>>>>> instanceId: 289, cmd:
>>>>>>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>>>>>>> ,
>>>>>>> cmdInfo:
>>>>>>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
>>>>>>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
>>>>>>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
>>>>>>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
>>>>>>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
>>>>>>>
>>>>>>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
>>>>>>> 806cc457\",\"interface
>>>>>>>
>>>>>>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
>>>>>>> "}","_":"1541607404902"},
>>>>>>>
>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>>>>> lastUpdated: null, lastPolled: null, created: null}
>>>>>>> 2018-11-07 17:16:45,280 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (qtp1096283470-466:ctx-27e3330a ctx-7e984b1b) (logid:5ebca5bb) submit
>>>>>>> async job-927, details: AsyncJobVO {id:927, userId: 2, accountId: 2,
>>>>>>> instanceType: FirewallRule, instanceId: 289, cmd:
>>>>>>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>>>>>>> ,
>>>>>>> cmdInfo:
>>>>>>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
>>>>>>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
>>>>>>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
>>>>>>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
>>>>>>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
>>>>>>>
>>>>>>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
>>>>>>> 806cc457\",\"interface
>>>>>>>
>>>>>>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
>>>>>>> "}","_":"1541607404902"},
>>>>>>>
>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>>>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>>>>> lastUpdated: null, lastPolled: null, created: null}
>>>>>>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) APPLYING FIREWALL RULES
>>>>>>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) Applying firewall rules in network Ntwk[206|Guest|8]
>>>>>>> 2018-11-07 17:16:45,345 DEBUG [c.c.a.t.Request]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115866969: Sending { Cmd , MgmtId:
>>>>>>> 220777304233416,
>>>>>>> via: 1(cshp121), Ver: v1, Flags: 100001,
>>>>>>>
>>>>>>>
>> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
>>>>>>> router.name
>> ":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
>>>>>>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
>>>>>>> :"false"},"wait":0}}]
>>>>>>>
>>>>>>> }
>>>>>>> 2018-11-07 17:18:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-1960b382) (logid:bcb6ab77) Task (job-927) has been
>>>>>>> pending for 107 seconds
>>>>>>> 2018-11-07 17:19:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-c7b405f5) (logid:2eda05d8) Task (job-927) has been
>>>>>>> pending for 167 seconds
>>>>>>> 2018-11-07 17:20:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-9661b60b) (logid:432b6bd2) Task (job-927) has been
>>>>>>> pending for 227 seconds
>>>>>>> 2018-11-07 17:21:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-18fa2315) (logid:fa867749) Task (job-927) has been
>>>>>>> pending for 287 seconds
>>>>>>> 2018-11-07 17:22:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-ba0654c9) (logid:572f3a44) Task (job-927) has been
>>>>>>> pending for 347 seconds
>>>>>>> 2018-11-07 17:23:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-2acb9ef9) (logid:83a6be92) Task (job-927) has been
>>>>>>> pending for 407 seconds
>>>>>>> 2018-11-07 17:24:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-8658487d) (logid:8ad384ee) Task (job-927) has been
>>>>>>> pending for 467 seconds
>>>>>>> 2018-11-07 17:25:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-9b2a9bc2) (logid:6d4f5007) Task (job-927) has been
>>>>>>> pending for 527 seconds
>>>>>>> 2018-11-07 17:26:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-3522c7f8) (logid:c5609631) Task (job-927) has been
>>>>>>> pending for 587 seconds
>>>>>>> 2018-11-07 17:27:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-762be74d) (logid:2942dfbd) Task (job-927) has been
>>>>>>> pending for 647 seconds
>>>>>>> 2018-11-07 17:28:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-2ce78e8b) (logid:ae408435) Task (job-927) has been
>>>>>>> pending for 707 seconds
>>>>>>> 2018-11-07 17:29:31,232 DEBUG [c.c.a.t.Request]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115866969: Received: { Ans: ,
>>>>>>> MgmtId: 220777304233416,
>>>>>>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
>>>>>>> 2018-11-07 17:29:31,235 WARN [c.c.n.f.FirewallManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) Failed to apply firewall rules due to : Resource
>>>>>>> [DataCenter:1] is
>>>>>>> unreachable: Unable to apply firewall rules on router
>>>>>>> 2018-11-07 17:29:31,300 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) APPLYING FIREWALL RULES
>>>>>>> 2018-11-07 17:29:31,301 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) Applying firewall rules in network Ntwk[206|Guest|8]
>>>>>>> 2018-11-07 17:29:31,314 DEBUG [c.c.a.t.Request]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115867196: Sending { Cmd , MgmtId:
>>>>>>> 220777304233416,
>>>>>>> via: 1(cshp121), Ver: v1, Flags: 100001,
>>>>>>>
>>>>>>>
>> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":true,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
>>>>>>> router.name
>> ":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
>>>>>>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
>>>>>>> :"false"},"wait":0}}]
>>>>>>>
>>>>>>> }
>>>>>>> 2018-11-07 17:29:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-23b76d0d) (logid:57a65a25) Task (job-927) has been
>>>>>>> pending for 767 seconds
>>>>>>> 2018-11-07 17:30:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-f049b29a) (logid:7fbb726e) Task (job-927) has been
>>>>>>> pending for 827 seconds
>>>>>>> 2018-11-07 17:31:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-717decf8) (logid:88f19102) Task (job-927) has been
>>>>>>> pending for 887 seconds
>>>>>>> 2018-11-07 17:32:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-4768ae42) (logid:55f233fa) Task (job-927) has been
>>>>>>> pending for 947 seconds
>>>>>>> 2018-11-07 17:33:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-816fef7b) (logid:5d9db903) Task (job-927) has been
>>>>>>> pending for 1007 seconds
>>>>>>> 2018-11-07 17:34:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-b8559261) (logid:4dcb351e) Task (job-927) has been
>>>>>>> pending for 1067 seconds
>>>>>>> 2018-11-07 17:35:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-94e242a4) (logid:6388b17a) Task (job-927) has been
>>>>>>> pending for 1127 seconds
>>>>>>> 2018-11-07 17:36:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-79404740) (logid:0dcdd7aa) Task (job-927) has been
>>>>>>> pending for 1187 seconds
>>>>>>> 2018-11-07 17:37:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-5f60335c) (logid:2039a058) Task (job-927) has been
>>>>>>> pending for 1247 seconds
>>>>>>> 2018-11-07 17:38:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Timer-1:ctx-ca5488fa) (logid:0c78bc1a) Task (job-927) has been
>>>>>>> pending for 1307 seconds
>>>>>>> 2018-11-07 17:39:31,688 DEBUG [c.c.a.t.Request]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115867196: Received: { Ans: ,
>>>>>>> MgmtId: 220777304233416,
>>>>>>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
>>>>>>> 2018-11-07 17:39:31,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Complete
>>>>>>> async job-927, jobStatus: FAILED, resultCode: 530, result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList"
>>>>>>> :[],"errorcode":530,"errortext":"Failed
>>>>>>>
>>>>>>> to create firewall rule"}
>>>>>>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Publish
>>>>>>> async
>>>>>>> job-927 complete on message bus
>>>>>>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up
>>>>>>> jobs related to job-927
>>>>>>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Update db
>>>>>>> status for job-927
>>>>>>> 2018-11-07 17:39:31,739 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up
>>>>>>> jobs joined with job-927 and disjoin all subjobs created from job-
>>>>>>> 927
>>>>>>> 2018-11-07 17:39:31,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Done
>>>>>>> executing
>>>>>>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>>>>>>> for job-927
>>>>>>> 2018-11-07 17:39:31,744 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Remove
>>>>>>> job-927 from job monitoring
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Configuration:
>>>>>>> ACS version 4.11.1.0
>>>>>>> Hypervisor KVM
>>>>>>> S.O. Ubuntu 16.04
>>>>>>> --
>>>>>>>
>>>>>>> *Ugo Vasi* / System Administrator
>>>>>>> ugo.vasi@procne.it <ma...@procne.it>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Procne S.r.l.*
>>>>>>> +39 0432 486 523
>>>>>>> via Cotonificio, 45
>>>>>>> 33010 Tavagnacco (UD)
>>>>>>> www.procne.it <http://www.procne.it/>
>>>>>>>
>>>>>>>
>>>>>>> Le informazioni contenute nella presente comunicazione ed i relativi
>>>>>>> allegati possono essere riservate e sono, comunque, destinate
>>>>>>> esclusivamente alle persone od alla Società sopraindicati. La
>>>>>>> diffusione, distribuzione e/o copiatura del documento trasmesso da
>>>>>>> parte di qualsiasi soggetto diverso dal destinatario è proibita sia
>>>>>>> ai sensi dell'art. 616 c.p., che ai sensi del Decreto Legislativo n.
>>>>>>> 196/2003 "Codice in materia di protezione dei dati personali". Se
>>>>>>> avete ricevuto questo messaggio per errore, vi preghiamo di
>>>>>>> distruggerlo e di informare immediatamente Procne S.r.l. scrivendo
>>>>>>> all' indirizzo e-mailinfo@procne.it <ma...@procne.it>.
>>>>>>>
>>>>>>>
>>>> --
>>>>
>>>> *Ugo Vasi* / System Administrator
>>>> ugo.vasi@procne.it <ma...@procne.it>
>>>>
>>>>
>>>>
>>>>
>>>> *Procne S.r.l.*
>>>> +39 0432 486 523
>>>> via Cotonificio, 45
>>>> 33010 Tavagnacco (UD)
>>>> www.procne.it <http://www.procne.it/>
>>>>
>>>>
>>>> Le informazioni contenute nella presente comunicazione ed i relativi
>>>> allegati possono essere riservate e sono, comunque, destinate
>>>> esclusivamente alle persone od alla Società sopraindicati. La
>>>> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
>>>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>>>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>>>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>>>> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
>>>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>>>> info@procne.it <ma...@procne.it>.
>>>>
>>>>
>> --
>>
>> *Ugo Vasi* / System Administrator
>> ugo.vasi@procne.it <ma...@procne.it>
>>
>>
>>
>>
>> *Procne S.r.l.*
>> +39 0432 486 523
>> via Cotonificio, 45
>> 33010 Tavagnacco (UD)
>> www.procne.it <http://www.procne.it/>
>>
>>
>> Le informazioni contenute nella presente comunicazione ed i relativi
>> allegati possono essere riservate e sono, comunque, destinate
>> esclusivamente alle persone od alla Società sopraindicati. La
>> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>> info@procne.it <ma...@procne.it>.
>>
>>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.
Re: urgent: Unable to apply firewall rules on router
Posted by Ugo Vasi <ug...@procne.it.INVALID>.
Hi all,
I tried to restart the network where the router is "slowed down". I used
the "clean up" option and the "Make redundant" option. Cloudstack
created a second master router as I expected but then gave me the
following error (I quote an extract from the file management-server.log):
2018-12-11 08:35:16,317 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203 ctx-136ac03e)
(logid:d325b88a) Attempting to destroy router 29
2018-12-11 08:35:16,334 DEBUG [c.c.u.d.T.Transaction]
(API-Job-Executor-17:ctx-abf9fa80 job-1203 ctx-136ac03e)
(logid:d325b88a) Rolling back the transaction: Time = 3 Name =
API-Job-Executor-17; called by
-TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-Transaction.execute:43-Transaction.execute:47-AsyncJobManagerImpl.submitAsyncJob:231-VirtualMachineManagerImpl.stopVmThroughJobQueue:4498-VirtualMachineManagerImpl.advanceStop:1600-VirtualMachineManagerImpl.advanceExpunge:513-VirtualMachineManagerImpl.advanceExpunge:502-VirtualMachineManagerImpl.expunge:491-NetworkHelperImpl.destroyRouter:253
2018-12-11 08:35:16,338 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203 ctx-136ac03e)
(logid:d325b88a) Unable to schedule async job for command
com.cloud.vm.VmWorkStop, unexpected exception.
javax.persistence.EntityExistsException: Entity already exists:
at
com.cloud.utils.db.GenericDaoBase.persist(GenericDaoBase.java:1434)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:235)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$1.doInTransaction(AsyncJobManagerImpl.java:231)
at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:50)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:231)
...
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry '1206' for key 'PRIMARY'
at
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
... 74 more
2018-12-11 08:35:16,352 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
com.cloud.utils.exception.CloudRuntimeException: Unable to schedule
async job for command com.cloud.vm.VmWorkStop, unexpected exception.
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.submitAsyncJob(AsyncJobManagerImpl.java:247)
at
com.cloud.vm.VirtualMachineManagerImpl.stopVmThroughJobQueue(VirtualMachineManagerImpl.java:4498)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1600)
...
2018-12-11 08:35:16,355 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Complete
async job-1203, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Unable
to schedule async job for command com.cloud.vm.VmWorkStop, unexpected
exception."}
2018-12-11 08:35:16,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Publish
async job-1203 complete on message bus
2018-12-11 08:35:16,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Wake up
jobs related to job-1203
2018-12-11 08:35:16,357 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Update db
status for job-1203
2018-12-11 08:35:16,359 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Wake up
jobs joined with job-1203 and disjoin all subjobs created from job- 1203
2018-12-11 08:35:16,363 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Done
executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for
job-1203
2018-12-11 08:35:16,363 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-17:ctx-abf9fa80 job-1203) (logid:d325b88a) Remove
job-1203 from job monitoring
Note that the network continues to work but the error returns to appear
even making further attempts.
Do you have any suggestions on what to do?
Il 06/12/18 09:49, Ugo Vasi ha scritto:
> Hi all,
> I still have this problem, the same router that blocked me the first
> time has also blocked a second time and now another router that has
> been operating for a few months has slowed down a lot (it did not give
> me errors).
>
>
> this is the job log of inserting a new firewall rule in the router:
>
> 2018-12-05 09:01:27,579 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-100:ctx-49a89073 job-1664) (logid:fd9d8cd1) Add
> job-1664 into job monitoring
> 2018-12-05 09:01:27,589 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp1096283470-926:ctx-f607bc6e ctx-75946dbc) (logid:16270ff5) submit
> async job-1664, details: AsyncJobVO {id:1664, userId: 2, accountId: 2,
> instanceType: FirewallRule, instanceId: 359, cmd:
> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd,
> cmdInfo:
> {"startport":"80","ipaddressid":"5e93afc0-43a0-483b-8086-580162b98c2d","httpmethod":"GET","ctxAccountId":"2","uuid":"4ae0123b-8596-4ed8-9466-43e725679146","cmdEventType":"FIREWALL.OPEN","cidrlist":"0.0.0.0/0","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEventId":"6003","id":"359","ctxDetails":"{\"interface
> com.cloud.network.rules.FirewallRule\":\"4ae0123b-8596-4ed8-9466-43e725679146\",\"interface
> com.cloud.network.IpAddress\":\"5e93afc0-43a0-483b-8086-580162b98c2d\"}","_":"1543996887133"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2018-12-05 09:01:27,590 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-100:ctx-49a89073 job-1664) (logid:7109e74e)
> Executing AsyncJobVO {id:1664, userId: 2, accountId: 2, instanceType:
> FirewallRule, instanceId: 359, cmd:
> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd,
> cmdInfo:
> {"startport":"80","ipaddressid":"5e93afc0-43a0-483b-8086-580162b98c2d","httpmethod":"GET","ctxAccountId":"2","uuid":"4ae0123b-8596-4ed8-9466-43e725679146","cmdEventType":"FIREWALL.OPEN","cidrlist":"0.0.0.0/0","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEventId":"6003","id":"359","ctxDetails":"{\"interface
> com.cloud.network.rules.FirewallRule\":\"4ae0123b-8596-4ed8-9466-43e725679146\",\"interface
> com.cloud.network.IpAddress\":\"5e93afc0-43a0-483b-8086-580162b98c2d\"}","_":"1543996887133"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 220777304233416, completeMsid: null,
> lastUpdated: null, lastPolled: null, created: null}
> 2018-12-05 09:01:27,650 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) APPLYING FIREWALL RULES
> 2018-12-05 09:01:27,650 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Applying firewall rules in network Ntwk[205|Guest|8]
> 2018-12-05 09:01:27,668 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Seq 4-8314770812033476439: Sending { Cmd , MgmtId:
> 220777304233416, via: 4(cshp132), Ver: v1, Flags: 100001,
> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":356,"srcIp":"yy.yy.yy.43","protocol":"tcp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":true,"sourceCidrList":["xx.xx.xx.xx/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":357,"srcIp":"yy.yy.yy.43","protocol":"tcp","srcPortRange":[443,443],"revoked":false,"alreadyAdded":true,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":358,"srcIp":"yy.yy.yy.43","protocol":"udp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":true,"sourceCidrList":["xx.xx.xx.xx/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false},{"id":359,"srcIp":"yy.yy.yy.43","protocol":"tcp","srcPortRange":[80,80],"revoked":false,"alreadyAdded":false,"sourceCidrList":["0.0.0.0/0"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"router.name":"r-29-VM","router.guest.ip":"10.11.12.1","router.ip":"169.254.3.228","zone.network.type":"Advanced","firewall.egress.default":"false"},"wait":0}}]
> }
> 2018-12-05 09:03:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-9f534fa1) (logid:ef1d6d8f) Task (job-1664) has been
> pending for 106 seconds
> 2018-12-05 09:04:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-4dd690a2) (logid:86485fc4) Task (job-1664) has been
> pending for 166 seconds
> 2018-12-05 09:05:14,182 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-1ac7a2bb) (logid:71fbc15d) Task (job-1664) has been
> pending for 226 seconds
> 2018-12-05 09:06:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-14ff35d8) (logid:786dbd01) Task (job-1664) has been
> pending for 286 seconds
> 2018-12-05 09:07:14,182 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-01f380a2) (logid:448af255) Task (job-1664) has been
> pending for 346 seconds
> 2018-12-05 09:08:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-164e5d2d) (logid:f1d49e29) Task (job-1664) has been
> pending for 406 seconds
> 2018-12-05 09:09:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-381ef9a0) (logid:eb42c557) Task (job-1664) has been
> pending for 466 seconds
> 2018-12-05 09:10:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-0a5a6645) (logid:39ca12b1) Task (job-1664) has been
> pending for 526 seconds
> 2018-12-05 09:11:14,181 WARN [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-d0b5e6ca) (logid:421221d0) Task (job-1664) has been
> pending for 586 seconds
> 2018-12-05 09:11:28,021 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Seq 4-8314770812033476439: Received: { Ans: ,
> MgmtId: 220777304233416, via: 4(cshp132), Ver: v1, Flags: 0, {
> GroupAnswer } }
> 2018-12-05 09:11:28,147 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Complete async job-1664, jobStatus: SUCCEEDED,
> resultCode: 0, result:
> org.apache.cloudstack.api.response.FirewallResponse/firewallrule/{"id":"4ae0123b-8596-4ed8-9466-43e725679146","protocol":"tcp","startport":80,"endport":80,"ipaddressid":"5e93afc0-43a0-483b-8086-580162b98c2d","networkid":"2f1e01d4-e7ce-4e2a-86f9-d9f32eaa66ca","ipaddress":"yy.yy.yy.43","state":"Active","cidrlist":"0.0.0.0/0","tags":[],"fordisplay":true}
> 2018-12-05 09:11:28,149 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Publish async job-1664 complete on message bus
> 2018-12-05 09:11:28,149 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Wake up jobs related to job-1664
> 2018-12-05 09:11:28,149 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Update db status for job-1664
> 2018-12-05 09:11:28,150 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-100:ctx-49a89073 job-1664 ctx-8bf9116f)
> (logid:7109e74e) Wake up jobs joined with job-1664 and disjoin all
> subjobs created from job- 1664
> 2018-12-05 09:11:28,154 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-100:ctx-49a89073 job-1664) (logid:7109e74e) Done
> executing
> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
> for job-1664
> 2018-12-05 09:11:28,155 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-100:ctx-49a89073 job-1664) (logid:7109e74e) Remove
> job-1664 from job monitoring
>
> n the router logs I see that the router management scripts
> (CsHelper.py, CsRoute.py, CsAddress.py, etc) take about 5 seconds to
> do their job.
>
>
> Il 10/11/18 11:47, Andrija Panic ha scritto:
>> IF VR is part of VPC, then just restart VPC wtih "Clean Up" options -
>> never
>> restart VR (unless you really prefer to for some reason).
>> If you are using Isolated Network setup (single VR and single network
>> behind VR), then restart the Network, with "Clean UP" options.
>> In both cases, a BRAND NEW router will be created (after previous one
>> was
>> destroyed)
>>
>> Interestingly, this is also good to know, when i.e. using custom VR
>> systemVM template - i.e. there is bug (i.e. dnsmasq issue in past), you
>> upload new systemVM template, define it as the default routing
>> template in
>> Global Settings... then you really need to restart VPC/Network with
>> Clean
>> UP in order to create BRAND NEW VR from whatever is the NEW template.
>> Stopping and restarting/rebooting VR (which does recreate OS disk from
>> scratch) - does it from the same linked template....(there are some
>> workarrounds for this also...)
>>
>>
>> Hope that help
>> Andrija
>>
>> On Sat, 10 Nov 2018 at 10:12, Ugo Vasi<ug...@procne.it> wrote:
>>
>>> Hi Rafael,
>>> in the file pippo.log I see messages similar to the following but also
>>> appear in other virtual routers that work regularly:
>>>
>>> 2018-09-29 17:26:32,554 CsHelper.py execute:193 Command 'iptables -t
>>> mangle -D PREROUTING -s xx.xx.xx.xx/32 -m state --state NEW -j
>>> CONNMARK'
>>> returned non-zero exit status 2
>>> 2018-09-29 17:26:32,554 CsNetfilter.py get_unseen:131 Delete rule -D
>>> PREROUTING -s xx.xx.xx.xx/32 -m state --state NEW -j CONNMARK from
>>> table
>>> mangle
>>>
>>> I tried to restart the router but the behavior was not changed.
>>> Regenerating it instead seems to work well, the rules are added in a
>>> few
>>> seconds.
>>>
>>> I noticed that by destroying a router with the appropriate button, the
>>> manager does not automatically recreate it. I have to stop and
>>> restart a
>>> machine that depends on that router to get it re-created.
>>> Is there another procedure to automate the destruction and
>>> recreation of
>>> the routers?
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Il 09/11/18 12:29, Rafael Weingärtner ha scritto:
>>>> Did you check the logs in the affected router?
>>>>
>>>> On Fri, Nov 9, 2018 at 9:28 AM Ugo Vasi<ug...@procne.it.invalid>
>>> wrote:
>>>>> Hi Glenn,
>>>>> I tried to restart the manager but nothing changed. Note that this
>>>>> behavior only occurs on this router, the others work regularly.
>>>>> As soon as possible restart the router and see what happens.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Il 08/11/18 19:36, Glenn Wagner ha scritto:
>>>>>> Hi Ugo,
>>>>>>
>>>>>> Have you tried to just restart the management service to clear any
>>>>> running tasks?
>>>>>> And then try add the rules again.
>>>>>>
>>>>>> Regards
>>>>>> Glenn Wagner
>>>>>>
>>>>>>
>>>>>> glenn.wagner@shapeblue.com
>>>>>> www.shapeblue.com
>>>>>> Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset
>>>>>> West, Cape
>>>>> Town 7129South Africa
>>>>>> @shapeblue
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Ugo Vasi<ug...@procne.it.INVALID>
>>>>>> Sent: Thursday, 08 November 2018 5:33 PM
>>>>>> To:users@cloudstack.apache.org; Andrija Panic <
>>> andrija.panic@gmail.com>
>>>>>> Subject: Re: urgent: Unable to apply firewall rules on router
>>>>>>
>>>>>> Hi Andrija,
>>>>>> from the checks you have suggested I do not show up long running
>>>>>> jobs.
>>>>>>
>>>>>> There are no error messages in the agent logs. By migrating the
>>>>>> router,
>>>>> the behavior has not changed.
>>>>>> Doing further tests I found that the added rules become effective
>>>>> immediately but the interface takes about 25 minutes to show it as
>>> active.
>>>>> A couple of times gave error:
>>>>>> 2018-11-08 16:22:28,588 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-17:ctx-36b7f3eb job-942) (logid:a107efdf) Complete
>>> async
>>>>> job-942, jobStatus: FAILED, resultCode: 530, result:
>>>>>
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>
>>>>>> to create firewall rule"}
>>>>>>
>>>>>> When I delete a rule, it remains active until the status is
>>>>>> updated and
>>>>> then disappears (about 20 minutes after).
>>>>>> Il 07/11/18 18:38, Andrija Panic ha scritto:
>>>>>>> Hi Ugo,
>>>>>>>
>>>>>>> I have seen similar issues with i.e. starting a VM when there are
>>>>>>> other long running jobs - check if there are any ongoing long jobs
>>>>>>> already, that might be blocking the executioon of this job -
>>>>>>> i.e. long
>>>>>>> running snapshots, or other thing.
>>>>>>> I would also examine agent.log on the host where this VR is
>>>>>>> located -
>>>>>>> there might be some traces there...
>>>>>>>
>>>>>>> Try this SQL to list aysnc jobs:
>>>>>>>
>>>>>>> select aj.id,
>>>>>>> case when aj.job_status=1 then 'completed' when
>>>>>>> aj.job_status=2 then 'progress' when aj.job_status=3 then
>>>>>>> 'error' end
>>>>> as status,
>>>>>>> aj.created, aj.last_updated, aj.related,
>>>>>>> account.account_name, user.username, host.name as host, vm.name as
>>>>> instance, vmj.step, aj.job_cmd
>>>>>>> from async_job aj
>>>>>>> inner join vm_work_job vmj on aj.id = vmj.id
>>>>>>> left join vm_instance vm on vmj.vm_instance_id=vm.id
>>>>>>> left join user on aj.user_id=user.id
>>>>>>> left join account on aj.account_id=account.id
>>>>>>> left join host on vm.host_id=host.id
>>>>>>>
>>>>>>> Alternatively, try to live-migrate VR to another host, and try
>>>>>>> to add
>>>>>>> rule again.
>>>>>>>
>>>>>>> Cheers
>>>>>>> Andrija
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 7 Nov 2018 at 17:59, Ugo Vasi<ug...@procne.it.invalid>
>>>>> wrote:
>>>>>>>> Hi all,
>>>>>>>> I'm having a problem when I try to insert a firewall rule of an
>>>>>>>> address connected to a new VM of a Guest Isolated Network.
>>>>>>>>
>>>>>>>> After a while the job is removed as FAILED. I try to repeat the
>>>>>>>> operation but the problem remains. How can I unblock the
>>>>>>>> situation?
>>>>>>>>
>>>>>>>> here it is the log of job-927:
>>>>>>>>
>>>>>>>> 2018-11-07 17:16:45,256 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0787853c) Add
>>>>>>>> job-927 into job monitoring
>>>>>>>> 2018-11-07 17:16:45,279 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7)
>>>>>>>> Executing
>>>>>>>> AsyncJobVO {id:927, userId: 2, accountId: 2, instanceType:
>>>>>>>> FirewallRule,
>>>>>>>> instanceId: 289, cmd:
>>>>>>>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>>>>>>>>
>>>>>>>> ,
>>>>>>>> cmdInfo:
>>>>>>>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
>>>>>>>>
>>>>>>>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
>>>>>>>>
>>>>>>>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
>>>>>>>>
>>>>>>>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
>>>>>>>>
>>>>>>>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
>>>>>>>>
>>>>>>>>
>>>>>>>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
>>>>>>>>
>>>>>>>> 806cc457\",\"interface
>>>>>>>>
>>>>>>>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
>>>>>>>>
>>>>>>>> "}","_":"1541607404902"},
>>>>>>>>
>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>>>>>>>> resultCode: 0,
>>>>>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>>>>>> lastUpdated: null, lastPolled: null, created: null}
>>>>>>>> 2018-11-07 17:16:45,280 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (qtp1096283470-466:ctx-27e3330a ctx-7e984b1b) (logid:5ebca5bb)
>>>>>>>> submit
>>>>>>>> async job-927, details: AsyncJobVO {id:927, userId: 2,
>>>>>>>> accountId: 2,
>>>>>>>> instanceType: FirewallRule, instanceId: 289, cmd:
>>>>>>>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>>>>>>>>
>>>>>>>> ,
>>>>>>>> cmdInfo:
>>>>>>>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
>>>>>>>>
>>>>>>>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
>>>>>>>>
>>>>>>>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
>>>>>>>>
>>>>>>>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
>>>>>>>>
>>>>>>>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
>>>>>>>>
>>>>>>>>
>>>>>>>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
>>>>>>>>
>>>>>>>> 806cc457\",\"interface
>>>>>>>>
>>>>>>>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
>>>>>>>>
>>>>>>>> "}","_":"1541607404902"},
>>>>>>>>
>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>>>>>>>> resultCode: 0,
>>>>>>>> result: null, initMsid: 220777304233416, completeMsid: null,
>>>>>>>> lastUpdated: null, lastPolled: null, created: null}
>>>>>>>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) APPLYING FIREWALL RULES
>>>>>>>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) Applying firewall rules in network
>>>>>>>> Ntwk[206|Guest|8]
>>>>>>>> 2018-11-07 17:16:45,345 DEBUG [c.c.a.t.Request]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115866969: Sending { Cmd ,
>>>>>>>> MgmtId:
>>>>>>>> 220777304233416,
>>>>>>>> via: 1(cshp121), Ver: v1, Flags: 100001,
>>>>>>>>
>>>>>>>>
>>> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
>>>
>>>>>>>> router.name
>>> ":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
>>>>>>>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
>>>>>>>>
>>>>>>>> :"false"},"wait":0}}]
>>>>>>>>
>>>>>>>> }
>>>>>>>> 2018-11-07 17:18:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-1960b382) (logid:bcb6ab77) Task (job-927) has been
>>>>>>>> pending for 107 seconds
>>>>>>>> 2018-11-07 17:19:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-c7b405f5) (logid:2eda05d8) Task (job-927) has been
>>>>>>>> pending for 167 seconds
>>>>>>>> 2018-11-07 17:20:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-9661b60b) (logid:432b6bd2) Task (job-927) has been
>>>>>>>> pending for 227 seconds
>>>>>>>> 2018-11-07 17:21:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-18fa2315) (logid:fa867749) Task (job-927) has been
>>>>>>>> pending for 287 seconds
>>>>>>>> 2018-11-07 17:22:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-ba0654c9) (logid:572f3a44) Task (job-927) has been
>>>>>>>> pending for 347 seconds
>>>>>>>> 2018-11-07 17:23:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-2acb9ef9) (logid:83a6be92) Task (job-927) has been
>>>>>>>> pending for 407 seconds
>>>>>>>> 2018-11-07 17:24:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-8658487d) (logid:8ad384ee) Task (job-927) has been
>>>>>>>> pending for 467 seconds
>>>>>>>> 2018-11-07 17:25:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-9b2a9bc2) (logid:6d4f5007) Task (job-927) has been
>>>>>>>> pending for 527 seconds
>>>>>>>> 2018-11-07 17:26:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-3522c7f8) (logid:c5609631) Task (job-927) has been
>>>>>>>> pending for 587 seconds
>>>>>>>> 2018-11-07 17:27:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-762be74d) (logid:2942dfbd) Task (job-927) has been
>>>>>>>> pending for 647 seconds
>>>>>>>> 2018-11-07 17:28:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-2ce78e8b) (logid:ae408435) Task (job-927) has been
>>>>>>>> pending for 707 seconds
>>>>>>>> 2018-11-07 17:29:31,232 DEBUG [c.c.a.t.Request]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115866969: Received: { Ans: ,
>>>>>>>> MgmtId: 220777304233416,
>>>>>>>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
>>>>>>>> 2018-11-07 17:29:31,235 WARN [c.c.n.f.FirewallManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) Failed to apply firewall rules due to : Resource
>>>>>>>> [DataCenter:1] is
>>>>>>>> unreachable: Unable to apply firewall rules on router
>>>>>>>> 2018-11-07 17:29:31,300 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) APPLYING FIREWALL RULES
>>>>>>>> 2018-11-07 17:29:31,301 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) Applying firewall rules in network
>>>>>>>> Ntwk[206|Guest|8]
>>>>>>>> 2018-11-07 17:29:31,314 DEBUG [c.c.a.t.Request]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115867196: Sending { Cmd ,
>>>>>>>> MgmtId:
>>>>>>>> 220777304233416,
>>>>>>>> via: 1(cshp121), Ver: v1, Flags: 100001,
>>>>>>>>
>>>>>>>>
>>> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":true,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
>>>
>>>>>>>> router.name
>>> ":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
>>>>>>>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
>>>>>>>>
>>>>>>>> :"false"},"wait":0}}]
>>>>>>>>
>>>>>>>> }
>>>>>>>> 2018-11-07 17:29:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-23b76d0d) (logid:57a65a25) Task (job-927) has been
>>>>>>>> pending for 767 seconds
>>>>>>>> 2018-11-07 17:30:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-f049b29a) (logid:7fbb726e) Task (job-927) has been
>>>>>>>> pending for 827 seconds
>>>>>>>> 2018-11-07 17:31:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-717decf8) (logid:88f19102) Task (job-927) has been
>>>>>>>> pending for 887 seconds
>>>>>>>> 2018-11-07 17:32:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-4768ae42) (logid:55f233fa) Task (job-927) has been
>>>>>>>> pending for 947 seconds
>>>>>>>> 2018-11-07 17:33:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-816fef7b) (logid:5d9db903) Task (job-927) has been
>>>>>>>> pending for 1007 seconds
>>>>>>>> 2018-11-07 17:34:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-b8559261) (logid:4dcb351e) Task (job-927) has been
>>>>>>>> pending for 1067 seconds
>>>>>>>> 2018-11-07 17:35:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-94e242a4) (logid:6388b17a) Task (job-927) has been
>>>>>>>> pending for 1127 seconds
>>>>>>>> 2018-11-07 17:36:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-79404740) (logid:0dcdd7aa) Task (job-927) has been
>>>>>>>> pending for 1187 seconds
>>>>>>>> 2018-11-07 17:37:32,512 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-5f60335c) (logid:2039a058) Task (job-927) has been
>>>>>>>> pending for 1247 seconds
>>>>>>>> 2018-11-07 17:38:32,511 WARN [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (Timer-1:ctx-ca5488fa) (logid:0c78bc1a) Task (job-927) has been
>>>>>>>> pending for 1307 seconds
>>>>>>>> 2018-11-07 17:39:31,688 DEBUG [c.c.a.t.Request]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
>>>>>>>> (logid:0e6c51f7) Seq 1-5860309015115867196: Received: { Ans: ,
>>>>>>>> MgmtId: 220777304233416,
>>>>>>>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
>>>>>>>> 2018-11-07 17:39:31,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7)
>>>>>>>> Complete
>>>>>>>> async job-927, jobStatus: FAILED, resultCode: 530, result:
>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList"
>>>>>>>>
>>>>>>>> :[],"errorcode":530,"errortext":"Failed
>>>>>>>>
>>>>>>>> to create firewall rule"}
>>>>>>>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Publish
>>>>>>>> async
>>>>>>>> job-927 complete on message bus
>>>>>>>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up
>>>>>>>> jobs related to job-927
>>>>>>>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7)
>>>>>>>> Update db
>>>>>>>> status for job-927
>>>>>>>> 2018-11-07 17:39:31,739 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up
>>>>>>>> jobs joined with job-927 and disjoin all subjobs created from job-
>>>>>>>> 927
>>>>>>>> 2018-11-07 17:39:31,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Done
>>>>>>>> executing
>>>>>>>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>>>>>>>>
>>>>>>>> for job-927
>>>>>>>> 2018-11-07 17:39:31,744 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Remove
>>>>>>>> job-927 from job monitoring
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Configuration:
>>>>>>>> ACS version 4.11.1.0
>>>>>>>> Hypervisor KVM
>>>>>>>> S.O. Ubuntu 16.04
>>>>>>>> --
>>>>>>>>
>>>>>>>> *Ugo Vasi* / System Administrator
>>>>>>>> ugo.vasi@procne.it <ma...@procne.it>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Procne S.r.l.*
>>>>>>>> +39 0432 486 523
>>>>>>>> via Cotonificio, 45
>>>>>>>> 33010 Tavagnacco (UD)
>>>>>>>> www.procne.it <http://www.procne.it/>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le informazioni contenute nella presente comunicazione ed i
>>>>>>>> relativi
>>>>>>>> allegati possono essere riservate e sono, comunque, destinate
>>>>>>>> esclusivamente alle persone od alla Società sopraindicati. La
>>>>>>>> diffusione, distribuzione e/o copiatura del documento trasmesso da
>>>>>>>> parte di qualsiasi soggetto diverso dal destinatario è proibita
>>>>>>>> sia
>>>>>>>> ai sensi dell'art. 616 c.p., che ai sensi del Decreto
>>>>>>>> Legislativo n.
>>>>>>>> 196/2003 "Codice in materia di protezione dei dati personali". Se
>>>>>>>> avete ricevuto questo messaggio per errore, vi preghiamo di
>>>>>>>> distruggerlo e di informare immediatamente Procne S.r.l. scrivendo
>>>>>>>> all' indirizzo e-mailinfo@procne.it <ma...@procne.it>.
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>>
>>>>> *Ugo Vasi* / System Administrator
>>>>> ugo.vasi@procne.it <ma...@procne.it>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Procne S.r.l.*
>>>>> +39 0432 486 523
>>>>> via Cotonificio, 45
>>>>> 33010 Tavagnacco (UD)
>>>>> www.procne.it <http://www.procne.it/>
>>>>>
>>>>>
>>>>> Le informazioni contenute nella presente comunicazione ed i relativi
>>>>> allegati possono essere riservate e sono, comunque, destinate
>>>>> esclusivamente alle persone od alla Società sopraindicati. La
>>>>> diffusione, distribuzione e/o copiatura del documento trasmesso da
>>>>> parte
>>>>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai
>>>>> sensi
>>>>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>>>>> "Codice in materia di protezione dei dati personali". Se avete
>>>>> ricevuto
>>>>> questo messaggio per errore, vi preghiamo di distruggerlo e di
>>>>> informare
>>>>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>>>>> info@procne.it <ma...@procne.it>.
>>>>>
>>>>>
>>> --
>>>
>>> *Ugo Vasi* / System Administrator
>>> ugo.vasi@procne.it <ma...@procne.it>
>>>
>>>
>>>
>>>
>>> *Procne S.r.l.*
>>> +39 0432 486 523
>>> via Cotonificio, 45
>>> 33010 Tavagnacco (UD)
>>> www.procne.it <http://www.procne.it/>
>>>
>>>
>>> Le informazioni contenute nella presente comunicazione ed i relativi
>>> allegati possono essere riservate e sono, comunque, destinate
>>> esclusivamente alle persone od alla Società sopraindicati. La
>>> diffusione, distribuzione e/o copiatura del documento trasmesso da
>>> parte
>>> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
>>> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
>>> "Codice in materia di protezione dei dati personali". Se avete ricevuto
>>> questo messaggio per errore, vi preghiamo di distruggerlo e di
>>> informare
>>> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
>>> info@procne.it <ma...@procne.it>.
>>>
>>>
>
>
--
*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <ma...@procne.it>
*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>
Le informazioni contenute nella presente comunicazione ed i relativi
allegati possono essere riservate e sono, comunque, destinate
esclusivamente alle persone od alla Società sopraindicati. La
diffusione, distribuzione e/o copiatura del documento trasmesso da parte
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
"Codice in materia di protezione dei dati personali". Se avete ricevuto
questo messaggio per errore, vi preghiamo di distruggerlo e di informare
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
info@procne.it <ma...@procne.it>.