You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Sonali Jadhav <so...@servercentralen.se> on 2015/03/03 12:39:01 UTC
RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan
Re: unable to remove NIC from instance
Posted by Daan Hoogland <da...@gmail.com>.
Sonali, sure. It will have to be improved. Please be very explicit in
the ticket, better change it will be taken by someone.
On Fri, Mar 20, 2015 at 9:01 AM, Sonali Jadhav
<so...@servercentralen.se> wrote:
> @Somesh: added ticket https://issues.apache.org/jira/browse/CLOUDSTACK-8328
> @Daan: should I add ticket for overzealous exception handling ?
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Tuesday, March 10, 2015 8:00 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> It could be a bug in that case. I guess you could file it on the tracker for further investigation.
>
> Regards,
> Somesh
>
> -----Original Message-----
> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
> Sent: Tuesday, March 10, 2015 8:21 AM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Hi somesh,
>
> I just cross checked that, service_offering where id=14, is nothing but "Custom" compute offering I created. That’s why ram_size value is set to NULL. So basically I cant/don’t want to change value of ram_size service offering, Is this bug?
>
> /Sonali
>
> -----Original Message-----
> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
> Sent: Tuesday, March 10, 2015 5:38 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Hi Somesh,
>
> I have very weird values in Service offering table,
>
> mysql> select * from service_offering;
> +----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
> | id | cpu | speed | ram_size | nw_rate | mc_rate | ha_enabled | limit_cpu_use | host_tag | default_use | vm_type | sort_key | is_volatile | deployment_planner |
> +----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
> | 1 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
> | 2 | 1 | 1000 | 1024 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
> | 7 | 1 | 500 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | domainrouter | 0 | 0 | NULL |
> | 8 | 1 | 500 | 1024 | 0 | 0 | 0 | 0 | NULL | 1 | consoleproxy | 0 | 0 | NULL |
> | 9 | 1 | 256 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | internalloadbalancervm | 0 | 0 | NULL |
> | 10 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 1 | secondarystoragevm | 0 | 0 | NULL |
> | 11 | 1 | 128 | 128 | 0 | 0 | 1 | 0 | NULL | 1 | elasticloadbalancervm | 0 | 0 | NULL |
> | 12 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
> | 13 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
> | 14 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
> | 16 | 2 | 2000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
> | 17 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
> | 18 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
> | 20 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
> +----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
> 14 rows in set (0.00 sec)
>
> mysql>
>
> any clue? There are 4 rows, which are having NULL values set, I have no clue from where did that came.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Thursday, March 5, 2015 8:49 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> That's causing the NPE. When you fix that, ram_size should have some value, you should be able to remove the NIC.
>
> Regards,
> Somesh
>
> -----Original Message-----
> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
> Sent: Thursday, March 05, 2015 12:20 AM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Hi Somesh,
>
> Here is output,
>
> mysql> select ram_size from service_offering where id=14;
> +----------+
> | ram_size |
> +----------+
> | NULL |
> +----------+
> 1 row in set (0.00 sec)
>
> Thanks,
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Thursday, March 5, 2015 2:19 AM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
>
> Could you send me the output of the following SQL:
> select ram_size from service_offering where id=14;
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
> Sent: Tuesday, March 03, 2015 6:39 AM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Hi Somesh,
>
> Have you looked at the logs and query results I sent, what do you think?
>
> /Sonali
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Thursday, February 26, 2015 1:42 PM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
>
> If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
>
>
> On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Somesh n Daan,
>>
>> Thanks for reply.
>>
>> Here are logs from point I logged to CloudStack till the error
>> occurred http://pastebin.com/j1rfEq5s
>>
>> And here is sql query output,
>>
>> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
>> *************************** 1. row ***************************
>> id: 24
>> name: linuxvm
>> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
>> instance_name: i-9-24-VM
>> state: Stopped
>> vm_template_id: 202
>> guest_os_id: 201
>> private_mac_address: 02:00:3e:c4:00:01
>> private_ip_address: 10.1.1.14
>> pod_id: 1
>> data_center_id: 1
>> host_id: NULL
>> last_host_id: 1
>> proxy_id: 1
>> proxy_assign_time: 2015-02-20 11:25:18
>> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
>> ha_enabled: 0
>> limit_cpu_use: 0
>> update_count: 25
>> update_time: 2015-02-23 06:26:46
>> created: 2015-02-20 11:24:26
>> removed: NULL
>> type: User
>> vm_type: User
>> account_id: 9
>> domain_id: 5
>> service_offering_id: 14
>> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
>> hypervisor_type: XenServer
>> disk_offering_id: 6
>> owner: 9
>> host_name: linuxvm
>> display_name: linuxvm
>> desired_state: NULL
>> dynamically_scalable: 0
>> display_vm: 1
>> power_state: NULL
>> power_state_update_time: NULL
>> power_state_update_count: 0
>> power_host: NULL
>> 1 row in set (0.00 sec)
>>
>> mysql>
>>
>> mysql> select * from nics where instance_id=24 \G
>> *************************** 1. row ***************************
>> id: 60
>> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
>> instance_id: 24
>> mac_address: 06:by:bg:00:01:33
>> ip4_address: 1.1.1.1
>> netmask: 255.255.0.0
>> gateway: 1.1.1.1
>> ip_type: Ip4
>> broadcast_uri: vlan://201
>> network_id: 207
>> mode: Dhcp
>> state: Allocated
>> strategy: Create
>> reserver_name: DirectNetworkGuru
>> reservation_id: NULL
>> device_id: 1
>> update_time: 2015-02-23 07:25:40
>> isolation_uri: vlan://201
>> ip6_address: NULL
>> default_nic: 0
>> vm_type: User
>> created: 2015-02-20 11:24:26
>> removed: NULL
>> ip6_gateway: NULL
>> ip6_cidr: NULL
>> secondary_ip: 0
>> display_nic: 1
>> *************************** 2. row ***************************
>> id: 61
>> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
>> instance_id: 24
>> mac_address: 02:00:3f:a4:00:01
>> ip4_address: 10.1.1.14
>> netmask: 255.255.255.0
>> gateway: 10.1.1.1
>> ip_type: Ip4
>> broadcast_uri: NULL
>> network_id: 211
>> mode: Dhcp
>> state: Allocated
>> strategy: Start
>> reserver_name: ExternalGuestNetworkGuru
>> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
>> device_id: 0
>> update_time: 2015-02-23 07:25:41
>> isolation_uri: NULL
>> ip6_address: NULL
>> default_nic: 1
>> vm_type: User
>> created: 2015-02-20 13:21:07
>> removed: NULL
>> ip6_gateway: NULL
>> ip6_cidr: NULL
>> secondary_ip: 0
>> display_nic: 1
>> 2 rows in set (0.00 sec)
>>
>> mysql>
>>
>> Let me know if you find something from this.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
>> Sent: Wednesday, February 25, 2015 11:07 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> Sonali,
>>
>> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>>
>> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>>
>> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>>
>> What you can do is,
>> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
>> a. select * from vm_instance where id=<VM_ID>;
>> b. select * from nics where instance_id=<VM_ID>;
>>
>> If I have this information I can try and identify why the original operation is failing.
>>
>> Somesh
>> CloudPlatform Escalations
>> Citrix Systems, Inc.
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Wednesday, February 25, 2015 7:47 AM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> I have no experience with this. It could be fixed in the db probably
>> but that is always a bad advice
>> 1 - coming from a developer
>> 2 - for application in a production system
>>
>> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>>
>>
>> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>> Sent: Tuesday, February 24, 2015 1:27 PM
>>> To: users@cloudstack.apache.org
>>> Subject: RE: unable to remove NIC from instance
>>>
>>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:22 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>>> What type of hypervisors are you running?
>>>
>>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> Super thanks for reply. I am running version 4.4.2
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> It seems to me you are a victim of overzealous exception handling.
>>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>>
>>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> Hi Daan,
>>>>>
>>>>> I have checked logs form starting of api call, and I have not found
>>>>> anything special, But apart from error, I found one warning,
>>>>>
>>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>>
>>>>> Don’t know meaning of this.
>>>>>
>>>>> Here are my logs,
>>>>>
>>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>>> 10,
>>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>>> _
>>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>>> r
>>>>> -
>>>>> 6
>>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>>> v
>>>>> i
>>>>> r
>>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>>> i
>>>>> a
>>>>> n
>>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>>> r
>>>>> i
>>>>> d
>>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>>> ;
>>>>> a
>>>>> c
>>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>>> a
>>>>> p
>>>>> i
>>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>>> s
>>>>> u
>>>>> l
>>>>> t":true,"wait":0}}] }
>>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>>> request
>>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>>> 0.11499999999999999
>>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>>> 249373720971888: Resp: Routing to peer
>>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>>> request
>>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>>> 249373720971888: Resp: Routing to peer
>>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>>> request
>>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>>> 0.13343749999999996
>>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>>> 249373720971888: Resp: Routing to peer
>>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>>> E
>>>>> l
>>>>> w
>>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>>> f
>>>>> 0
>>>>> 7
>>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>>> None, instanceId: null, cmd:
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>>> n
>>>>> ,
>>>>> cmdInfo:
>>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>>> c
>>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>>> d
>>>>> 9
>>>>> 7
>>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>>> 7
>>>>> 5
>>>>> 3
>>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>>> c
>>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>>> 9
>>>>> 7
>>>>> 9
>>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>>> IN_PROGRESS,
>>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>>> null, created: null}
>>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>>> monitoring
>>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>>> null, cmd:
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>>> n
>>>>> ,
>>>>> cmdInfo:
>>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>>> c
>>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>>> d
>>>>> 9
>>>>> 7
>>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>>> 7
>>>>> 5
>>>>> 3
>>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>>> c
>>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>>> 9
>>>>> 7
>>>>> 9
>>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>>> IN_PROGRESS,
>>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>>> null, created: null}
>>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>>> 192.168.45.32
>>>>> -- GET
>>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>>> E
>>>>> l
>>>>> w
>>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>>> f
>>>>> 0
>>>>> 7
>>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>>> proxy
>>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>>> job-527 execution on object VmWorkJobQueue.24
>>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>>> storage VM
>>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>>> request
>>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>>> GetHostStatsAnswer } }
>>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>>> request
>>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>>> {
>>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>>> GetHostStatsAnswer } }
>>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>>> request
>>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>>> GetHostStatsAnswer } }
>>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>>> b
>>>>> 8
>>>>> a
>>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>>> 3
>>>>> 4
>>>>> 0
>>>>> 6327
>>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>>> request
>>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>>> {
>>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>>> GetHostStatsAnswer } }
>>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>>> 192.168.45.32
>>>>> -- GET
>>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>>> b
>>>>> 8
>>>>> a
>>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>>> 3
>>>>> 4
>>>>> 0
>>>>> 6327
>>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>>> n
>>>>> for job-526
>>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>>> monitoring
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonali,
>>>>>
>>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>>
>>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>>
>>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>>> GET
>>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>>> 9
>>>>>> 4
>>>>>> c
>>>>>> f
>>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>>> 7
>>>>>> 0
>>>>>> 7
>>>>>> 3
>>>>>> 8796
>>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>>> 115.114.134.114 -- GET
>>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>>> 9
>>>>>> 4
>>>>>> c
>>>>>> f
>>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>>> 7
>>>>>> 0
>>>>>> 7
>>>>>> 3
>>>>>> 8796
>>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>>> i
>>>>>> n
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>>> job-499,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>>> i
>>>>>> n
>>>>>> for job-499
>>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>>> job monitoring
>>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>>> GET
>>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>>> 9
>>>>>> 4
>>>>>> c
>>>>>> f
>>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>>> 7
>>>>>> 0
>>>>>> 7
>>>>>> 4
>>>>>> 1797
>>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>>> 115.114.134.114 -- GET
>>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>>> 9
>>>>>> 4
>>>>>> c
>>>>>> f
>>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>>> 7
>>>>>> 0
>>>>>> 7
>>>>>> 4
>>>>>> 1797
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: RE: unable to remove NIC from instance
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>>
>>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>>
>>>>>> thanks,
>>>>>>
>>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>>> Anyone help?
>>>>>>> /Sonali
>>>>>>>
>>>>>>>
>>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>>
>>>>>>> Anyone?
>>>>>>>
>>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>>
>>>>>>>
>>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>>
>>>>>>> Here are logs,
>>>>>>>
>>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>>> {
>>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>>> 0
>>>>>>> }
>>>>>>> }
>>>>>>> ]
>>>>>>> }
>>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>>> job-483
>>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>>> T
>>>>>>> r
>>>>>>> a
>>>>>>> n
>>>>>>> s
>>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>>> f
>>>>>>> l
>>>>>>> e
>>>>>>> c
>>>>>>> t
>>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>>> 9
>>>>>>> 1
>>>>>>> -
>>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>>> 2
>>>>>>> 0
>>>>>>> 4
>>>>>>> -
>>>>>>> $
>>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>>> e
>>>>>>> a
>>>>>>> n
>>>>>>> u
>>>>>>> p
>>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>>> a
>>>>>>> g
>>>>>>> e
>>>>>>> d
>>>>>>> C
>>>>>>> ontext$1.call:56
>>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>>> to
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>>> ... 26 more
>>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>>> a
>>>>>>> b
>>>>>>> 3
>>>>>>> f
>>>>>>> 1
>>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>>> 8
>>>>>>> 6
>>>>>>> 9
>>>>>>> 6
>>>>>>> 3
>>>>>>> 84
>>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>>> 192.168.34.1
>>>>>>> -- GET
>>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>>> a
>>>>>>> b
>>>>>>> 3
>>>>>>> f
>>>>>>> 1
>>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>>> 8
>>>>>>> 6
>>>>>>> 9
>>>>>>> 6
>>>>>>> 3
>>>>>>> 84
>>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>>> while executing
>>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>>> job-482,
>>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>>> null"}
>>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>>> job-482
>>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>>> suitable for reconnect
>>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>>> job monitoring
>>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>>> hosts suitable for reconnect
>>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>>> clusters already owned by this management server
>>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>>> hosts for clusters already owned by this management server
>>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>>> clusters not owned by any management server
>>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>>> hosts for clusters not owned by any management server
>>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>>> a
>>>>>>> b
>>>>>>> 3
>>>>>>> f
>>>>>>> 1
>>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>>> 8
>>>>>>> 6
>>>>>>> 9
>>>>>>> 9
>>>>>>> 3
>>>>>>> 85
>>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>>> 192.168.34.1
>>>>>>> -- GET
>>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>>> a
>>>>>>> b
>>>>>>> 3
>>>>>>> f
>>>>>>> 1
>>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>>> 8
>>>>>>> 6
>>>>>>> 9
>>>>>>> 9
>>>>>>> 3
>>>>>>> 85
>>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>>> status on
>>>>>>> 4(SeSolXS02)
>>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>>> status on
>>>>>>> 1(SeSolXS01)
>>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>>> status on
>>>>>>> 5(SeSolXS03)
>>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>>> on
>>>>>>> 6(SeSolXS04)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> /Sonali
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>>> To: users@cloudstack.apache.org
>>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>>
>>>>>>> Hi Sonali,
>>>>>>>
>>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>>
>>>>>>> Thank You.
>>>>>>> Gopalakrishnan.S
>>>>>>> Cloud Engineer - Fogpanel
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> From: Sonali Jadhav
>>>>>>> To: users@cloudstack.apache.org
>>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>>> Subject: unable to remove NIC from instance
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> /sonali
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Daan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>
>
>
> --
> Daan
>
>
>
>
>
>
>
>
>
>
--
Daan
RE: unable to remove NIC from instance
Posted by Sonali Jadhav <so...@servercentralen.se>.
@Somesh: added ticket https://issues.apache.org/jira/browse/CLOUDSTACK-8328
@Daan: should I add ticket for overzealous exception handling ?
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Tuesday, March 10, 2015 8:00 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
It could be a bug in that case. I guess you could file it on the tracker for further investigation.
Regards,
Somesh
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 10, 2015 8:21 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi somesh,
I just cross checked that, service_offering where id=14, is nothing but "Custom" compute offering I created. That’s why ram_size value is set to NULL. So basically I cant/don’t want to change value of ram_size service offering, Is this bug?
/Sonali
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 10, 2015 5:38 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
I have very weird values in Service offering table,
mysql> select * from service_offering;
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| id | cpu | speed | ram_size | nw_rate | mc_rate | ha_enabled | limit_cpu_use | host_tag | default_use | vm_type | sort_key | is_volatile | deployment_planner |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| 1 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 2 | 1 | 1000 | 1024 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 7 | 1 | 500 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | domainrouter | 0 | 0 | NULL |
| 8 | 1 | 500 | 1024 | 0 | 0 | 0 | 0 | NULL | 1 | consoleproxy | 0 | 0 | NULL |
| 9 | 1 | 256 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | internalloadbalancervm | 0 | 0 | NULL |
| 10 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 1 | secondarystoragevm | 0 | 0 | NULL |
| 11 | 1 | 128 | 128 | 0 | 0 | 1 | 0 | NULL | 1 | elasticloadbalancervm | 0 | 0 | NULL |
| 12 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 13 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 14 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 16 | 2 | 2000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 17 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 18 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 20 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
14 rows in set (0.00 sec)
mysql>
any clue? There are 4 rows, which are having NULL values set, I have no clue from where did that came.
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 8:49 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
That's causing the NPE. When you fix that, ram_size should have some value, you should be able to remove the NIC.
Regards,
Somesh
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Thursday, March 05, 2015 12:20 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Here is output,
mysql> select ram_size from service_offering where id=14;
+----------+
| ram_size |
+----------+
| NULL |
+----------+
1 row in set (0.00 sec)
Thanks,
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 2:19 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
Could you send me the output of the following SQL:
select ram_size from service_offering where id=14;
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 03, 2015 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan
RE: unable to remove NIC from instance
Posted by Somesh Naidu <So...@citrix.com>.
It could be a bug in that case. I guess you could file it on the tracker for further investigation.
Regards,
Somesh
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 10, 2015 8:21 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi somesh,
I just cross checked that, service_offering where id=14, is nothing but "Custom" compute offering I created. That’s why ram_size value is set to NULL. So basically I cant/don’t want to change value of ram_size service offering, Is this bug?
/Sonali
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 10, 2015 5:38 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
I have very weird values in Service offering table,
mysql> select * from service_offering;
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| id | cpu | speed | ram_size | nw_rate | mc_rate | ha_enabled | limit_cpu_use | host_tag | default_use | vm_type | sort_key | is_volatile | deployment_planner |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| 1 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 2 | 1 | 1000 | 1024 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 7 | 1 | 500 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | domainrouter | 0 | 0 | NULL |
| 8 | 1 | 500 | 1024 | 0 | 0 | 0 | 0 | NULL | 1 | consoleproxy | 0 | 0 | NULL |
| 9 | 1 | 256 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | internalloadbalancervm | 0 | 0 | NULL |
| 10 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 1 | secondarystoragevm | 0 | 0 | NULL |
| 11 | 1 | 128 | 128 | 0 | 0 | 1 | 0 | NULL | 1 | elasticloadbalancervm | 0 | 0 | NULL |
| 12 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 13 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 14 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 16 | 2 | 2000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 17 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 18 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 20 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
14 rows in set (0.00 sec)
mysql>
any clue? There are 4 rows, which are having NULL values set, I have no clue from where did that came.
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 8:49 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
That's causing the NPE. When you fix that, ram_size should have some value, you should be able to remove the NIC.
Regards,
Somesh
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Thursday, March 05, 2015 12:20 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Here is output,
mysql> select ram_size from service_offering where id=14;
+----------+
| ram_size |
+----------+
| NULL |
+----------+
1 row in set (0.00 sec)
Thanks,
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 2:19 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
Could you send me the output of the following SQL:
select ram_size from service_offering where id=14;
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 03, 2015 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan
RE: unable to remove NIC from instance
Posted by Sonali Jadhav <so...@servercentralen.se>.
Hi somesh,
I just cross checked that, service_offering where id=14, is nothing but "Custom" compute offering I created. That’s why ram_size value is set to NULL. So basically I cant/don’t want to change value of ram_size service offering, Is this bug?
/Sonali
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 10, 2015 5:38 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
I have very weird values in Service offering table,
mysql> select * from service_offering;
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| id | cpu | speed | ram_size | nw_rate | mc_rate | ha_enabled | limit_cpu_use | host_tag | default_use | vm_type | sort_key | is_volatile | deployment_planner |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| 1 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 2 | 1 | 1000 | 1024 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 7 | 1 | 500 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | domainrouter | 0 | 0 | NULL |
| 8 | 1 | 500 | 1024 | 0 | 0 | 0 | 0 | NULL | 1 | consoleproxy | 0 | 0 | NULL |
| 9 | 1 | 256 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | internalloadbalancervm | 0 | 0 | NULL |
| 10 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 1 | secondarystoragevm | 0 | 0 | NULL |
| 11 | 1 | 128 | 128 | 0 | 0 | 1 | 0 | NULL | 1 | elasticloadbalancervm | 0 | 0 | NULL |
| 12 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 13 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 14 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 16 | 2 | 2000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 17 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 18 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 20 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
14 rows in set (0.00 sec)
mysql>
any clue? There are 4 rows, which are having NULL values set, I have no clue from where did that came.
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 8:49 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
That's causing the NPE. When you fix that, ram_size should have some value, you should be able to remove the NIC.
Regards,
Somesh
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Thursday, March 05, 2015 12:20 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Here is output,
mysql> select ram_size from service_offering where id=14;
+----------+
| ram_size |
+----------+
| NULL |
+----------+
1 row in set (0.00 sec)
Thanks,
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 2:19 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
Could you send me the output of the following SQL:
select ram_size from service_offering where id=14;
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 03, 2015 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan
RE: unable to remove NIC from instance
Posted by Sonali Jadhav <so...@servercentralen.se>.
Hi Somesh,
I have very weird values in Service offering table,
mysql> select * from service_offering;
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| id | cpu | speed | ram_size | nw_rate | mc_rate | ha_enabled | limit_cpu_use | host_tag | default_use | vm_type | sort_key | is_volatile | deployment_planner |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
| 1 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 2 | 1 | 1000 | 1024 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 7 | 1 | 500 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | domainrouter | 0 | 0 | NULL |
| 8 | 1 | 500 | 1024 | 0 | 0 | 0 | 0 | NULL | 1 | consoleproxy | 0 | 0 | NULL |
| 9 | 1 | 256 | 128 | NULL | NULL | 1 | 0 | NULL | 1 | internalloadbalancervm | 0 | 0 | NULL |
| 10 | 1 | 500 | 512 | NULL | NULL | 0 | 0 | NULL | 1 | secondarystoragevm | 0 | 0 | NULL |
| 11 | 1 | 128 | 128 | 0 | 0 | 1 | 0 | NULL | 1 | elasticloadbalancervm | 0 | 0 | NULL |
| 12 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 13 | 4 | 1000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | secondarystoragevm | 0 | 0 | NULL |
| 14 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 16 | 2 | 2000 | 4096 | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 17 | NULL | NULL | NULL | NULL | NULL | 0 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 18 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
| 20 | NULL | NULL | NULL | NULL | NULL | 1 | 0 | NULL | 0 | NULL | 0 | 0 | NULL |
+----+------+-------+----------+---------+---------+------------+---------------+----------+-------------+------------------------+----------+-------------+--------------------+
14 rows in set (0.00 sec)
mysql>
any clue? There are 4 rows, which are having NULL values set, I have no clue from where did that came.
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 8:49 PM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
That's causing the NPE. When you fix that, ram_size should have some value, you should be able to remove the NIC.
Regards,
Somesh
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Thursday, March 05, 2015 12:20 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Here is output,
mysql> select ram_size from service_offering where id=14;
+----------+
| ram_size |
+----------+
| NULL |
+----------+
1 row in set (0.00 sec)
Thanks,
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 2:19 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
Could you send me the output of the following SQL:
select ram_size from service_offering where id=14;
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 03, 2015 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan
RE: unable to remove NIC from instance
Posted by Somesh Naidu <So...@citrix.com>.
That's causing the NPE. When you fix that, ram_size should have some value, you should be able to remove the NIC.
Regards,
Somesh
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Thursday, March 05, 2015 12:20 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Here is output,
mysql> select ram_size from service_offering where id=14;
+----------+
| ram_size |
+----------+
| NULL |
+----------+
1 row in set (0.00 sec)
Thanks,
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 2:19 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
Could you send me the output of the following SQL:
select ram_size from service_offering where id=14;
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 03, 2015 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan
RE: unable to remove NIC from instance
Posted by Sonali Jadhav <so...@servercentralen.se>.
Hi Somesh,
Here is output,
mysql> select ram_size from service_offering where id=14;
+----------+
| ram_size |
+----------+
| NULL |
+----------+
1 row in set (0.00 sec)
Thanks,
/Sonali
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Thursday, March 5, 2015 2:19 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
Could you send me the output of the following SQL:
select ram_size from service_offering where id=14;
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 03, 2015 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan
RE: unable to remove NIC from instance
Posted by Somesh Naidu <So...@citrix.com>.
Yes, I did. Looks strange, an NPE when calculating minimum memory required for this VM.
Could you send me the output of the following SQL:
select ram_size from service_offering where id=14;
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Sonali Jadhav [mailto:sonali@servercentralen.se]
Sent: Tuesday, March 03, 2015 6:39 AM
To: users@cloudstack.apache.org
Subject: RE: unable to remove NIC from instance
Hi Somesh,
Have you looked at the logs and query results I sent, what do you think?
/Sonali
-----Original Message-----
From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
Sent: Thursday, February 26, 2015 1:42 PM
To: users@cloudstack.apache.org
Subject: Re: unable to remove NIC from instance
As far as I can tell it tries to delete a secondary ip from a nic instead of a second (or first) nic. This is going to cost me some diving into. I hope Somesh has a simple answer for you. The initial error we talked about is in your log between lines 97 and 133. I will have a look when I find the time but I can't promise.
If you are familiar with git you could look at the history of the 4.4 branch since the release to see if a similar problem has been fixed.
On Thu, Feb 26, 2015 at 8:35 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
> Hi Somesh n Daan,
>
> Thanks for reply.
>
> Here are logs from point I logged to CloudStack till the error
> occurred http://pastebin.com/j1rfEq5s
>
> And here is sql query output,
>
> mysql> select * from vm_instance where instance_name="i-9-24-VM" \G
> *************************** 1. row ***************************
> id: 24
> name: linuxvm
> uuid: b5c535b4-10c4-44a9-b117-6df07537434c
> instance_name: i-9-24-VM
> state: Stopped
> vm_template_id: 202
> guest_os_id: 201
> private_mac_address: 02:00:3e:c4:00:01
> private_ip_address: 10.1.1.14
> pod_id: 1
> data_center_id: 1
> host_id: NULL
> last_host_id: 1
> proxy_id: 1
> proxy_assign_time: 2015-02-20 11:25:18
> vnc_password: 7fJhsb8PVskfWRFgiQw02qDERH4PYxUS7xHy+2OU/HI=
> ha_enabled: 0
> limit_cpu_use: 0
> update_count: 25
> update_time: 2015-02-23 06:26:46
> created: 2015-02-20 11:24:26
> removed: NULL
> type: User
> vm_type: User
> account_id: 9
> domain_id: 5
> service_offering_id: 14
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> hypervisor_type: XenServer
> disk_offering_id: 6
> owner: 9
> host_name: linuxvm
> display_name: linuxvm
> desired_state: NULL
> dynamically_scalable: 0
> display_vm: 1
> power_state: NULL
> power_state_update_time: NULL
> power_state_update_count: 0
> power_host: NULL
> 1 row in set (0.00 sec)
>
> mysql>
>
> mysql> select * from nics where instance_id=24 \G
> *************************** 1. row ***************************
> id: 60
> uuid: 99378e9f-ba64-4c76-ac16-d9ac977d979f
> instance_id: 24
> mac_address: 06:by:bg:00:01:33
> ip4_address: 1.1.1.1
> netmask: 255.255.0.0
> gateway: 1.1.1.1
> ip_type: Ip4
> broadcast_uri: vlan://201
> network_id: 207
> mode: Dhcp
> state: Allocated
> strategy: Create
> reserver_name: DirectNetworkGuru
> reservation_id: NULL
> device_id: 1
> update_time: 2015-02-23 07:25:40
> isolation_uri: vlan://201
> ip6_address: NULL
> default_nic: 0
> vm_type: User
> created: 2015-02-20 11:24:26
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> *************************** 2. row ***************************
> id: 61
> uuid: 42178ae9-1c4c-43be-8a05-5d6259ee9ee4
> instance_id: 24
> mac_address: 02:00:3f:a4:00:01
> ip4_address: 10.1.1.14
> netmask: 255.255.255.0
> gateway: 10.1.1.1
> ip_type: Ip4
> broadcast_uri: NULL
> network_id: 211
> mode: Dhcp
> state: Allocated
> strategy: Start
> reserver_name: ExternalGuestNetworkGuru
> reservation_id: c59c1ead-fd98-4822-b7bb-71124fbf0018
> device_id: 0
> update_time: 2015-02-23 07:25:41
> isolation_uri: NULL
> ip6_address: NULL
> default_nic: 1
> vm_type: User
> created: 2015-02-20 13:21:07
> removed: NULL
> ip6_gateway: NULL
> ip6_cidr: NULL
> secondary_ip: 0
> display_nic: 1
> 2 rows in set (0.00 sec)
>
> mysql>
>
> Let me know if you find something from this.
>
> /Sonali
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Wednesday, February 25, 2015 11:07 PM
> To: users@cloudstack.apache.org
> Subject: RE: unable to remove NIC from instance
>
> Sonali,
>
> While this could be fixed in the DB, as Daan mentioned, it is not recommended. The primary table that needs to be update is the nics table but in addition, relevant tables like user_ip_address, etc need to be updated. There may also be impact on usage. Moreover, updating the DB won’t take care of removing the nic on the hypervisor/VM.
>
> A better way to deal with this is to resolve the original error that is preventing CCP from removing the nic.
>
> I have looked at the information (log snippet) you have shared. As Daan pointed out, these logs contain consequential errors where queryAsyncJobResult API is failing to get the result of a job (happens to be the RemoveNicFromVMCmdByAdmin API call that you are interested in) that relate to some trouble with the job framework. But this is not why the original remove nic command failed.
>
> What you can do is,
> 1. Note the time of the last line of log in mgmt. server (just do a tail on mgmt. log) 2. From CS UI, remove the nic from the VM 3. Send us mgmt. logs starting from time noted in step #1 4. In addition provide details of the VM,
> a. select * from vm_instance where id=<VM_ID>;
> b. select * from nics where instance_id=<VM_ID>;
>
> If I have this information I can try and identify why the original operation is failing.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, February 25, 2015 7:47 AM
> To: users@cloudstack.apache.org
> Subject: Re: unable to remove NIC from instance
>
> I have no experience with this. It could be fixed in the db probably
> but that is always a bad advice
> 1 - coming from a developer
> 2 - for application in a production system
>
> If you are very familiar with the datamodel you might try to, I would only try to do it in a test environment (or at least first) Also I do not have the db change that would help you by head.
>
>
> On Wed, Feb 25, 2015 at 12:25 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>> Hi Daan,
>>
>> Is there any manual way to fix it temporary ? I have production customer who is facing this issue.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>> Sent: Tuesday, February 24, 2015 1:27 PM
>> To: users@cloudstack.apache.org
>> Subject: RE: unable to remove NIC from instance
>>
>> I am running xenserver 6.2, I have 1 cluster with 4 Hosts.
>>
>> /Sonali
>>
>> -----Original Message-----
>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>> Sent: Tuesday, February 24, 2015 1:22 PM
>> To: users@cloudstack.apache.org
>> Subject: Re: unable to remove NIC from instance
>>
>> Ok, thanks, we are running a custom 4.4.2 version we had to make due to problems with the stock 4.4.2. I will look into making another release candidate for 4.4.3 and have a look at the code to see if I can find a cause to your problem. The last 4.4.3 release candidate I made was largely ignored and therefore never released. It also has a problem with creating new volumes on vmware which is not solved yet.
>> What type of hypervisors are you running?
>>
>> On Tue, Feb 24, 2015 at 8:45 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>> Hi Daan,
>>>
>>> Super thanks for reply. I am running version 4.4.2
>>>
>>> /Sonali
>>>
>>> -----Original Message-----
>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>> Sent: Tuesday, February 24, 2015 1:05 PM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: unable to remove NIC from instance
>>>
>>> It seems to me you are a victim of overzealous exception handling.
>>> I've seen that happen in the async job handling. The job dispatching or construction has an exception that is not handled but caught and ignored, it end and the async job is not removed from the queue. I'd like to look at the code near the stack trace but for that I need the exact version of Cloudstack you are running.
>>>
>>> On Tue, Feb 24, 2015 at 5:44 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>> Hi Daan,
>>>>
>>>> I have checked logs form starting of api call, and I have not found
>>>> anything special, But apart from error, I found one warning,
>>>>
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>>
>>>> Don’t know meaning of this.
>>>>
>>>> Here are my logs,
>>>>
>>>> 2015-02-23 13:11:45,457 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067: Response Received:
>>>> 2015-02-23 13:11:45,458 DEBUG [c.c.a.t.Request]
>>>> (DirectAgentCronJob-296:ctx-11de89f6) Seq 5-8090716730571096067:
>>>> Processing: { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags:
>>>> 10,
>>>> [{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"
>>>> _
>>>> vmMetaDatum":{"i-2-11-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:
>>>> true","s-15-VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","
>>>> r
>>>> -
>>>> 6
>>>> -VM":"viridian:true;acpi:1;apic:true;pae:true;nx:true","i-2-5-VM":"
>>>> v
>>>> i
>>>> r
>>>> idian:true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-13-VM":"
>>>> viridian:true;acpi:1;apic:true;pae:true;nx:true","i-5-12-VM":"virid
>>>> i
>>>> a
>>>> n
>>>> :true;acpi:1;apic:true;pae:true;nx:true;timeoffset:0","r-14-VM":"vi
>>>> r
>>>> i
>>>> d
>>>> ian:true;acpi:1;apic:true;pae:true;nx:true","v-1-VM":"viridian:true
>>>> ;
>>>> a
>>>> c
>>>> pi:1;apic:true;pae:true;nx:true","i-9-23-VM":"viridian:true;acpi:1;
>>>> a
>>>> p
>>>> i
>>>> c:true;pae:true;nx:true;timeoffset:-28800"},"_isExecuted":false,"re
>>>> s
>>>> u
>>>> l
>>>> t":true,"wait":0}}] }
>>>> 2015-02-23 13:11:45,572 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Executing
>>>> request
>>>> 2015-02-23 13:11:45,628 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-48:ctx-9e4c61a9) Vm cpu utilization
>>>> 0.11499999999999999
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: Response Received:
>>>> 2015-02-23 13:11:45,650 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-48:ctx-9e4c61a9) Seq 1-7548877400403099459: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,682 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Executing
>>>> request
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.1152941176470588
>>>> 2015-02-23 13:11:45,707 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Vm cpu utilization 0.2599999999999999
>>>> 2015-02-23 13:11:45,766 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: Response Received:
>>>> 2015-02-23 13:11:45,767 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-95:ctx-5f7ee8a6) Seq 5-249105354388946804: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,791 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Executing
>>>> request
>>>> 2015-02-23 13:11:45,835 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-192:ctx-61e81d25) Vm cpu utilization
>>>> 0.13343749999999996
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: Response Received:
>>>> 2015-02-23 13:11:45,858 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>> (DirectAgent-192:ctx-61e81d25) Seq 6-2490209118959188220: MgmtId
>>>> 249373720971888: Resp: Routing to peer
>>>> 2015-02-23 13:11:45,921 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122) ===START=== 192.168.45.32 -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,095 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) submit async job-526,
>>>> details: AsyncJobVO {id:526, userId: 2, accountId: 2, instanceType:
>>>> None, instanceId: null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,096 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Add job-526 into job
>>>> monitoring
>>>> 2015-02-23 13:11:46,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Executing AsyncJobVO
>>>> {id:526, userId: 2, accountId: 2, instanceType: None, instanceId:
>>>> null, cmd:
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> ,
>>>> cmdInfo:
>>>> {"response":"json","sessionkey":"7O6X4iElw4iQSPf+X1ypVjWpwMo\u003d","
>>>> c
>>>> txDetails":"{\"com.cloud.vm.Nic\":\"99378e9f-ba64-4c76-ac16-d9ac977
>>>> d
>>>> 9
>>>> 7
>>>> 9f\",\"com.cloud.vm.VirtualMachine\":\"b5c535b4-10c4-44a9-b117-6df0
>>>> 7
>>>> 5
>>>> 3
>>>> 7434c\"}","virtualmachineid":"b5c535b4-10c4-44a9-b117-6df07537434c","
>>>> c
>>>> mdEventType":"NIC.DELETE","nicid":"99378e9f-ba64-4c76-ac16-d9ac977d
>>>> 9
>>>> 7
>>>> 9
>>>> f","ctxUserId":"2","httpmethod":"GET","_":"1424693402955","ctxAccou
>>>> n t I d":"2","ctxStartEventId":"1160"}, cmdVersion: 0, status:
>>>> IN_PROGRESS,
>>>> processStatus: 0, resultCode: 0, result: null, initMsid:
>>>> 59778234354585, completeMsid: null, lastUpdated: null, lastPolled:
>>>> null, created: null}
>>>> 2015-02-23 13:11:46,097 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-4:ctx-5a612122 ctx-b6192261) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=removeNicFromVirtualMachine&response=json&sessionkey=7O6X4i
>>>> E
>>>> l
>>>> w
>>>> 4iQSPf%2BX1ypVjWpwMo%3D&virtualmachineid=b5c535b4-10c4-44a9-b117-6d
>>>> f
>>>> 0
>>>> 7
>>>> 537434c&nicid=99378e9f-ba64-4c76-ac16-d9ac977d979f&_=1424693402955
>>>> 2015-02-23 13:11:46,219 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>>>> (consoleproxy-1:ctx-a5fe3580) Zone 1 is ready to launch console
>>>> proxy
>>>> 2015-02-23 13:11:46,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Sync
>>>> job-527 execution on object VmWorkJobQueue.24
>>>> 2015-02-23 13:11:46,226 WARN [c.c.u.d.Merovingian2]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526 ctx-0585e904) Was unable
>>>> to find lock for the key vm_instance24 and thread id 137380473
>>>> 2015-02-23 13:11:46,534 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
>>>> (secstorage-1:ctx-be45d820) Zone 1 is ready to launch secondary
>>>> storage VM
>>>> 2015-02-23 13:11:49,095 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-4a027738) HostStatsCollector is running...
>>>> 2015-02-23 13:11:49,115 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Executing
>>>> request
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-93:ctx-3810dbca) Seq 1-9127107594819755362: Response Received:
>>>> 2015-02-23 13:11:49,166 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 1-9127107594819755362: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 1, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,182 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Executing
>>>> request
>>>> 2015-02-23 13:11:49,226 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-4:ctx-5d8fe3d8) Seq 4-677791743919277703: Response Received:
>>>> 2015-02-23 13:11:49,227 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 4-677791743919277703: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 4, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,242 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Executing
>>>> request
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-194:ctx-9a8ead36) Seq 5-8090716730571121376: Response Received:
>>>> 2015-02-23 13:11:49,280 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 5-8090716730571121376: Received:
>>>> { Ans: , MgmtId: 59778234354585, via: 5, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,288 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7) ===START=== 192.168.45.32 -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,302 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Executing
>>>> request
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-383:ctx-5f562eca) Seq 6-892275676172801479: Response Received:
>>>> 2015-02-23 13:11:49,354 DEBUG [c.c.a.t.Request]
>>>> (StatsCollector-4:ctx-4a027738) Seq 6-892275676172801479: Received:
>>>> {
>>>> Ans: , MgmtId: 59778234354585, via: 6, Ver: v1, Flags: 10, {
>>>> GetHostStatsAnswer } }
>>>> 2015-02-23 13:11:49,626 DEBUG [c.c.a.ApiServlet]
>>>> (http-6443-exec-3:ctx-c1a661f7 ctx-62507b94) ===END===
>>>> 192.168.45.32
>>>> -- GET
>>>> command=queryAsyncJobResult&jobId=af305f51-35d5-417a-acd7-ecb3ddadb
>>>> b
>>>> 8
>>>> a
>>>> &response=json&sessionkey=7O6X4iElw4iQSPf%2BX1ypVjWpwMo%3D&_=142469
>>>> 3
>>>> 4
>>>> 0
>>>> 6327
>>>> 2015-02-23 13:11:49,672 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Unexpected exception
>>>> while executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> java.lang.RuntimeException: Job failed due to exception null
>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:744)
>>>> 2015-02-23 13:11:49,673 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Complete async job-526,
>>>> jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>> 2015-02-23 13:11:49,730 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Done executing
>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdmi
>>>> n
>>>> for job-526
>>>> 2015-02-23 13:11:49,757 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-118:ctx-adab6031 job-526) Remove job-526 from job
>>>> monitoring
>>>>
>>>>
>>>> /Sonali
>>>>
>>>> -----Original Message-----
>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>> Sent: Monday, February 23, 2015 2:42 PM
>>>> To: users@cloudstack.apache.org
>>>> Subject: Re: unable to remove NIC from instance
>>>>
>>>> Sonali,
>>>>
>>>> this looks like a consequential error. You need to look further back into the log to find the root cause. The start job is the a query for results, please find the start for the actual api command.
>>>>
>>>> On Mon, Feb 23, 2015 at 6:55 AM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>> I am trying to remove "Shared guest network". These are new logs,
>>>>>
>>>>> 2015-02-23 06:53:59,737 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-28b1e134) Found 0 networks to update RvR status.
>>>>> 2015-02-23 06:54:01,726 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,818 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-3:ctx-6eb9497d ctx-007c94be) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 3
>>>>> 8796
>>>>> 2015-02-23 06:54:01,993 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Unexpected exception
>>>>> while executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>> 2015-02-23 06:54:01,994 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Complete async
>>>>> job-499,
>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>> [],"errorcode":530,"errortext":"Job failed due to exception null"}
>>>>> 2015-02-23 06:54:02,051 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Done executing
>>>>> org.apache.cloudstack.api.command.admin.vm.RemoveNicFromVMCmdByAdm
>>>>> i
>>>>> n
>>>>> for job-499
>>>>> 2015-02-23 06:54:02,077 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>> (API-Job-Executor-105:ctx-b2370fa0 job-499) Remove job-499 from
>>>>> job monitoring
>>>>> 2015-02-23 06:54:03,942 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ce679066) AutoScaling Monitor is running...
>>>>> 2015-02-23 06:54:04,727 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3) ===START=== 115.114.134.114 --
>>>>> GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>> 2015-02-23 06:54:04,802 DEBUG [c.c.a.ApiServlet]
>>>>> (http-6443-exec-4:ctx-e51e68c3 ctx-a70a6fff) ===END===
>>>>> 115.114.134.114 -- GET
>>>>> command=queryAsyncJobResult&jobId=1fb40d3e-b0aa-4c68-b7d7-14bccb1b
>>>>> 9
>>>>> 4
>>>>> c
>>>>> f
>>>>> &response=json&sessionkey=bjqAH0BCMrO1DvWk%2Beb9zCi1nuo%3D&_=14246
>>>>> 7
>>>>> 0
>>>>> 7
>>>>> 4
>>>>> 1797
>>>>>
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sonali Jadhav [mailto:sonali@servercentralen.se]
>>>>> Sent: Monday, February 23, 2015 10:25 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: RE: unable to remove NIC from instance
>>>>>
>>>>> Hi,
>>>>>
>>>>> Yes I have tried that, and issue is still same. Same logs.
>>>>>
>>>>> /Sonali
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
>>>>> Sent: Sunday, February 22, 2015 10:57 PM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Re: unable to remove NIC from instance
>>>>>
>>>>> Sonil, Did you try what Gopalakrishnan suggested?
>>>>>
>>>>> Also can you expand on the version of cloudstack you are running?
>>>>>
>>>>> thanks,
>>>>>
>>>>> On Sun, Feb 22, 2015 at 12:24 PM, Sonali Jadhav <so...@servercentralen.se> wrote:
>>>>>> Anyone help?
>>>>>> /Sonali
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Anyone?
>>>>>>
>>>>>> Sent from my Sony Xperia™ smartphone
>>>>>>
>>>>>>
>>>>>> ---- Sonali Jadhav wrote ----
>>>>>>
>>>>>> Here are logs,
>>>>>>
>>>>>> 2015-02-20 14:26:34,122 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-12:null) SeqA 2-260495: Sending Seq 2-260495:
>>>>>> {
>>>>>> Ans: , MgmtId: 59778234354585, via: 2, Ver: v1, Flags: 100010,
>>>>>> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":
>>>>>> 0
>>>>>> }
>>>>>> }
>>>>>> ]
>>>>>> }
>>>>>> 2015-02-20 14:26:34,638 DEBUG [o.a.c.f.j.d.VmWorkJobDaoImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Expunge completed work
>>>>>> job-483
>>>>>> 2015-02-20 14:26:34,640 DEBUG [c.c.u.d.T.Transaction]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) Rolling back the transaction:
>>>>>> Time = 2 Name = Vm-Operations-Cleanup-1; called by
>>>>>> -TransactionLegacy.rollback:900-TransactionLegacy.removeUpTo:843-
>>>>>> T
>>>>>> r
>>>>>> a
>>>>>> n
>>>>>> s
>>>>>> actionLegacy.close:667-TransactionContextInterceptor.invoke:36-Re
>>>>>> f
>>>>>> l
>>>>>> e
>>>>>> c
>>>>>> t
>>>>>> iveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:
>>>>>> 9
>>>>>> 1
>>>>>> -
>>>>>> ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:
>>>>>> 2
>>>>>> 0
>>>>>> 4
>>>>>> -
>>>>>> $
>>>>>> Proxy182.expungeCompletedWorkJobs:-1-VirtualMachineManagerImpl$Cl
>>>>>> e
>>>>>> a
>>>>>> n
>>>>>> u
>>>>>> p
>>>>>> Task.runInContext:2389-ManagedContextRunnable$1.run:49-DefaultMan
>>>>>> a
>>>>>> g
>>>>>> e
>>>>>> d
>>>>>> C
>>>>>> ontext$1.call:56
>>>>>> 2015-02-20 14:26:34,658 ERROR [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Vm-Operations-Cleanup-1:ctx-53171ec5) VM Operations failed due
>>>>>> to
>>>>>> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.PreparedStatement@e7a49c5: DELETE FROM async_job WHERE async_job.id= 483
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1178)
>>>>>> at org.apache.cloudstack.framework.jobs.dao.VmWorkJobDaoImpl.expungeCompletedWorkJobs(VmWorkJobDaoImpl.java:149)
>>>>>> at sun.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>>>>>> at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
>>>>>> at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>>>>>> at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>>> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>>> at com.sun.proxy.$Proxy182.expungeCompletedWorkJobs(Unknown Source)
>>>>>> at com.cloud.vm.VirtualMachineManagerImpl$CleanupTask.runInContext(VirtualMachineManagerImpl.java:2389)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>>>>>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`async_job_join_map`, CONSTRAINT `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES `async_job` (`id`))
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>> at com.mysql.jdbc.Util.getInstance(Util.java:386)
>>>>>> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
>>>>>> at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
>>>>>> at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1169)
>>>>>> ... 26 more
>>>>>> 2015-02-20 14:26:34,770 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:34,797 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:34,858 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-4:ctx-46b96aa1 ctx-7b8c1717) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 6
>>>>>> 3
>>>>>> 84
>>>>>> 2015-02-20 14:26:35,024 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Unexpected exception
>>>>>> while executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd
>>>>>> java.lang.RuntimeException: Job failed due to exception null
>>>>>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
>>>>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:744)
>>>>>> 2015-02-20 14:26:35,025 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Complete async
>>>>>> job-482,
>>>>>> jobStatus: FAILED, resultCode: 530, result:
>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>>>>>> [],"errorcode":530,"errortext":"Job failed due to exception
>>>>>> null"}
>>>>>> 2015-02-20 14:26:35,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Done executing
>>>>>> org.apache.cloudstack.api.command.user.vm.RemoveNicFromVMCmd for
>>>>>> job-482
>>>>>> 2015-02-20 14:26:35,111 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Resetting hosts
>>>>>> suitable for reconnect
>>>>>> 2015-02-20 14:26:35,113 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-98:ctx-e2fe93d7 job-482) Remove job-482 from
>>>>>> job monitoring
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed resetting
>>>>>> hosts suitable for reconnect
>>>>>> 2015-02-20 14:26:35,114 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters already owned by this management server
>>>>>> 2015-02-20 14:26:35,116 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Acquiring hosts for
>>>>>> clusters not owned by any management server
>>>>>> 2015-02-20 14:26:35,117 DEBUG [c.c.h.d.HostDaoImpl]
>>>>>> (ClusteredAgentManager Timer:ctx-c3486e1a) Completed acquiring
>>>>>> hosts for clusters not owned by any management server
>>>>>> 2015-02-20 14:26:37,771 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f) ===START=== 192.168.34.1 -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,798 DEBUG [c.c.u.AccountManagerImpl]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) Access to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] granted to
>>>>>> Acct[35719f1c-832c-474d-8045-983ed17523a1-corp] by DomainChecker
>>>>>> 2015-02-20 14:26:37,857 DEBUG [c.c.a.ApiServlet]
>>>>>> (http-6443-exec-2:ctx-1053809f ctx-5db06e08) ===END===
>>>>>> 192.168.34.1
>>>>>> -- GET
>>>>>> command=queryAsyncJobResult&jobId=ff043283-9481-46d2-bcd9-71fbdd0
>>>>>> a
>>>>>> b
>>>>>> 3
>>>>>> f
>>>>>> 1
>>>>>> &response=json&sessionkey=CkNBaXkKYXP6itmOoSTQVEKXiJo%3D&_=142443
>>>>>> 8
>>>>>> 6
>>>>>> 9
>>>>>> 9
>>>>>> 3
>>>>>> 85
>>>>>> 2015-02-20 14:26:37,975 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-2:null) Ping from 2
>>>>>> 2015-02-20 14:26:38,048 DEBUG [c.c.a.m.AgentManagerImpl]
>>>>>> (AgentManager-Handler-7:null) Ping from 9
>>>>>> 2015-02-20 14:26:40,509 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-261:ctx-f3e99173) Unable to get current
>>>>>> status on
>>>>>> 4(SeSolXS02)
>>>>>> 2015-02-20 14:26:40,687 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-130:ctx-8b85b559) Unable to get current
>>>>>> status on
>>>>>> 1(SeSolXS01)
>>>>>> 2015-02-20 14:26:40,747 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-131:ctx-e6c7b284) Unable to get current
>>>>>> status on
>>>>>> 5(SeSolXS03)
>>>>>> 2015-02-20 14:26:40,981 WARN [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgentCronJob-33:ctx-c6748244) Unable to get current status
>>>>>> on
>>>>>> 6(SeSolXS04)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> /Sonali
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Gopalakrishnan S [mailto:gopal@assistanz.com]
>>>>>> Sent: Friday, February 20, 2015 7:12 PM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Re: unable to remove NIC from instance
>>>>>>
>>>>>> Hi Sonali,
>>>>>>
>>>>>> It may be still routing that shared guest network. Try to restart your virutal router and virtual machine or shutdown your VM before removing that NIC. let me know the logs when you remove that NIC.
>>>>>>
>>>>>> Thank You.
>>>>>> Gopalakrishnan.S
>>>>>> Cloud Engineer - Fogpanel
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: Sonali Jadhav
>>>>>> To: users@cloudstack.apache.org
>>>>>> Sent: Friday, February 20, 2015 7:06 PM
>>>>>> Subject: unable to remove NIC from instance
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have one linux instance to which I had added "Shared guest network" NIC. Then I added "isolated guest network" nic to it. Made "isolated guest network" NIC as default and tried to remove "Shared guest network" NIC, but its failing with null exception. I have attached logs.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> /sonali
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Daan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Daan
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Daan
>>>
>>>
>>
>>
>>
>> --
>> Daan
>>
>>
>>
>>
>
>
>
> --
> Daan
>
>
--
Daan