You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Kyle Flavin <ky...@citrix.com> on 2015/04/09 02:40:31 UTC
CloudStack hands out IP address of a stopped VM
I'm trying to get some help understanding the following behavior.
Yesterday we had an instance of CloudStack giving out the IP address of a stopped VM to a newly created VM. The existing server was found in the MySQL database with the assigned IP (sanitized outputs):
mysql> select name,private_ip_address,state from vm_instance where name like "<myhost>%";
+--------------------+--------------------+---------+
| name | private_ip_address | state |
+--------------------+--------------------+---------+
| <myhost> | 1.1.1.1 | Stopped |
+--------------------+--------------------+---------+
The new server booted up, and was given that same 1.1.1.1 IP as well, which caused a conflict in our external host management system.
It looks to me like the DHCP lease is expiring on the stopped VM, and then CloudStack is just handing it out again. However, it had previously been explained to me that CloudStack would not hand out IP’s of stopped VM’s (and I do see the IP address registered to the VM in the database). Is that true and is this a possible bug, or is that the expected behavior?
-Kyle
RE: CloudStack hands out IP address of a stopped VM
Posted by Kyle Flavin <ky...@citrix.com>.
Ah, I see. I set the new NIC to the default, and now it won't let me remove it; which is what I want. I'm not sure how it got removed in the first place, but it looks like we're back to a consistent state now.
Thanks!
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Tuesday, April 14, 2015 9:25 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
When you delete the VM additional nic (added to vm from UI/API non-default nic) then the nic entry from the nics table get deleted.
default nic is not allowed to delete from the CS. When VM is deleted the nic entry from the DB is marked as removed but it won't be deleted. When you deploy a VM there will be one default nic for the VM.
Please find below results for your reference.
mysql> SELECT id, instance_id,
mysql> mac_address,ip4_address,removed,default_nic FROM cloud.nics
mysql> where instance_id=161;
+-----+-------------+-------------------+-------------+---------+-------------+
| id | instance_id | mac_address | ip4_address | removed | default_nic |
+-----+-------------+-------------------+-------------+---------+-------------+
| 264 | 161 | 02:00:4e:65:00:01 | 10.1.1.238 | NULL | 1 |
+-----+-------------+-------------------+-------------+---------+-------------+
1 row in set (0.00 sec)
mysql> SELECT id, instance_id,
mysql> mac_address,ip4_address,removed,default_nic FROM cloud.nics
mysql> where instance_id=10;
+----+-------------+-------------------+-------------+---------------------+-------------+
| id | instance_id | mac_address | ip4_address | removed | default_nic |
+----+-------------+-------------------+-------------+---------------------+-------------+
| 21 | 10 | 02:00:27:80:00:06 | 10.1.1.250 | 2015-02-24 10:06:10 | 1 |
+----+-------------+-------------------+-------------+---------------------+-------------+
1 row in set (0.00 sec)
Thanks,
Jayapal
On 15-Apr-2015, at 3:27 AM, Kyle Flavin <ky...@citrix.com> wrote:
> Thanks Somesh. I added the NIC, verified there was an entry in the "nics" table, and then modified the IP/MAC in vm_instance to match what was in the nics table.
>
> Then to test, I deleted the NIC through the Web UI. It completely removed the entry from the "nics" table, and left the IP in the vm_instance table. It doesn't sound like that's the correct behavior?
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Tuesday, April 14, 2015 1:37 PM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
>> Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
> This is strange. CS would never delete an entry in the nics table, only set the removed column. I see only two possibilities, 1. The entry was never added for some reason (error condition), 2. It was manually removed.
>
> Anyway, at this point, just make sure that the IP/MAC in nics and vm_instance table matches. Also, the IP you are manually setting isn't assigned to any other nic.
>
> Regards,
> Somesh
>
> -----Original Message-----
> From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
> Sent: Tuesday, April 14, 2015 2:49 PM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
> Hi Somesh,
>
>>> Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
> Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
>
>>> Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
> That was me - I was trying to sanitize the data and missed an entry.
>
>>> Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
> They did not match. I added a NIC to the VM through the web interface, and it created an entry in the "nics" table. But the IP address did not match what was in the vm_instance table.
>
> I'm not sure what the history of this VM is. Perhaps someone tried to do some restoration on it. Is it safe to clear the IP from the vm_instance table, or change it to match what's in the nics table?
>
> -Kyle
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Tuesday, April 14, 2015 10:13 AM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
>> In the WebUI, I saw no NIC associated with the machine (maybe the
>> user deleted it?)
>
> Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
>
> Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
>
> Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
>
> It would be interesting to check if this VM was destroyed at some point and then recovered manually or via recoverVirtualMachine API.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
> Sent: Tuesday, April 14, 2015 12:10 PM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
> In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?). So I went ahead and added one:
>
> mysql> select * from nics where instance_id='2071';
> +------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
> | id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
> +------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
> | 2793 | 84566500-6a7e-4c3a-8035-d4b98f1ecebe | 101 | 06:50:a2:00:0a:ab | 10.1.131.11 | 255.255.254.0 | 10.1.130.1 | Ip4 | vlan://111 | 206 | Dhcp | Allocated | Create | DirectNetworkGuru | NULL | 0 | 2015-04-14 09:01:30 | vlan://111 | NULL | 0 | User | 2015-04-14 16:01:30 | NULL | NULL | NULL | 0 | 1 |
> +------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
>
>
>
> However, the IP address assigned in the nics table doesn't match the one in the vm_instance table:
>
>
> mysql> select id,name,private_ip_address from vm_instance where name='myhost'; +------+--------------------+--------------------+
> | id | name | private_ip_address |
> +------+--------------------+--------------------+
> | 101 | myhost | 10.1.131.0 |
> +------+--------------------+--------------------+
> 1 row in set (0.00 sec)
>
>
> Shouldn't that IP have been updated as well?
>
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, April 13, 2015 9:47 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: CloudStack hands out IP address of a stopped VM
>
> Hi Kyle,
>
> My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
> While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
>
>
> Thanks,
> Jayapal
>
> On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
>
>> When a vm is in expunging state, NIC table entry should be marked as
>> removed in DB. If you are sure about the IP address rendered by CS to
>> the VM , you can try adding the entry in db.
>>
>> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is it safe for me to attempt to re-add the entry through the
>>> database, since the VM is still in use?
>>>
>>> Also, I've noticed that there are expunged VM's with entries still
>>> in the NIC table. Should that be the case? Or should the record
>>> have been removed when the VM was deleted?
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Sunday, April 12, 2015 10:24 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> Cloudstack won't delete the nic entry, When VM is deleted it marked
>>> as removed.
>>> It might be deleted manually from the DB.
>>>
>>> Thanks,
>>> Jayapal
>>>
>>>
>>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>>
>>>> Jayapal,
>>>> Is there a reason why would the nic entry be empty? When is the
>>>> entry
>>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>>
>>>> -----Original Message-----
>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>> Sent: Thursday, April 09, 2015 9:01 PM
>>>> To: <us...@cloudstack.apache.org>
>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>
>>>> Hi Kyle,
>>>>
>>>> When CS picks the ip for new vm from free ip pool excluding nic
>>>> table
>>> ip4_adress.
>>>> In your case the nic entry for vm is empty due to this the ip will
>>>> goes
>>> free pool.
>>>>
>>>>
>>>> Thanks,
>>>> Jayapal
>>>>
>>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>>> wrote:
>>>>
>>>>> CS would not hand out the IPs of a stopped vm since the lease time
>>>>> is infinite. If you are able to reproduce is consistently please
>>>>> open a JIRA ticket.
>>>>>
>>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin
>>>>> <ky...@citrix.com>
>>> wrote:
>>>>>
>>>>>> Hi Jayapal,
>>>>>> The first query against the nics table returned an empty set,
>>>>>> while the second returned the hostname and IP address. Here is
>>>>>> the sanitized output from both queries:
>>>>>>
>>>>>>
>>>>>> mysql> select instance_id,ip4_address from nics where
>>>>>> mysql> instance_id=<id>;
>>>>>> Empty set (0.00 sec)
>>>>>>
>>>>>>
>>>>>> mysql> select id, name, private_ip_address from vm_instance
>>>>>> mysql> where
>>>>>> id=<id>;
>>>>>> +------+--------------------+--------------------+
>>>>>> | id | name | private_ip_address |
>>>>>> +------+--------------------+--------------------+
>>>>>> | myid | myhostname | 1.1.1.1 |
>>>>>> +------+--------------------+--------------------+
>>>>>> 1 row in set (0.00 sec)
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>>> To: <us...@cloudstack.apache.org>
>>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>>
>>>>>> Hi Kyle,
>>>>>>
>>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>>> ip4_address set to 'null'.
>>>>>> After that I am not able to reproduce the issue.
>>>>>> I will keep looking into my setup for this issue.
>>>>>>
>>>>>> Can you please send the below commands output from your setup.
>>>>>>
>>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>>> vm_instance where id=<id>;
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Jayapal
>>>>>>
>>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>>> wrote:
>>>>>>
>>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>>
>>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>>> address of
>>>>>> a stopped VM to a newly created VM. The existing server was
>>>>>> found in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>>
>>>>>>> mysql> select name,private_ip_address,state from vm_instance
>>>>>>> mysql> where name like "<myhost>%";
>>>>>>> +--------------------+--------------------+---------+
>>>>>>> | name | private_ip_address | state |
>>>>>>> +--------------------+--------------------+---------+
>>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>>> +--------------------+--------------------+---------+
>>>>>>>
>>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>>> well,
>>>>>> which caused a conflict in our external host management system.
>>>>>>>
>>>>>>> It looks to me like the DHCP lease is expiring on the stopped
>>>>>>> VM, and
>>>>>> then CloudStack is just handing it out again. However, it had
>>>>>> previously been explained to me that CloudStack would not hand
>>>>>> out IP's of stopped VM's (and I do see the IP address registered
>>>>>> to the VM
>>> in the database).
>>>>>> Is that true and is this a possible bug, or is that the expected
>>> behavior?
>>>>>>>
>>>>>>> -Kyle
>>>>>>
>>>>>>
>>>>
>>>
>>>
>
Re: CloudStack hands out IP address of a stopped VM
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Kyle,
When you delete the VM additional nic (added to vm from UI/API non-default nic) then the nic entry from the nics table get deleted.
default nic is not allowed to delete from the CS. When VM is deleted the nic entry from the DB is marked as removed but it won't be deleted. When you deploy a VM there will be one default nic for the VM.
Please find below results for your reference.
mysql> SELECT id, instance_id, mac_address,ip4_address,removed,default_nic FROM cloud.nics where instance_id=161;
+-----+-------------+-------------------+-------------+---------+-------------+
| id | instance_id | mac_address | ip4_address | removed | default_nic |
+-----+-------------+-------------------+-------------+---------+-------------+
| 264 | 161 | 02:00:4e:65:00:01 | 10.1.1.238 | NULL | 1 |
+-----+-------------+-------------------+-------------+---------+-------------+
1 row in set (0.00 sec)
mysql> SELECT id, instance_id, mac_address,ip4_address,removed,default_nic FROM cloud.nics where instance_id=10;
+----+-------------+-------------------+-------------+---------------------+-------------+
| id | instance_id | mac_address | ip4_address | removed | default_nic |
+----+-------------+-------------------+-------------+---------------------+-------------+
| 21 | 10 | 02:00:27:80:00:06 | 10.1.1.250 | 2015-02-24 10:06:10 | 1 |
+----+-------------+-------------------+-------------+---------------------+-------------+
1 row in set (0.00 sec)
Thanks,
Jayapal
On 15-Apr-2015, at 3:27 AM, Kyle Flavin <ky...@citrix.com> wrote:
> Thanks Somesh. I added the NIC, verified there was an entry in the "nics" table, and then modified the IP/MAC in vm_instance to match what was in the nics table.
>
> Then to test, I deleted the NIC through the Web UI. It completely removed the entry from the "nics" table, and left the IP in the vm_instance table. It doesn't sound like that's the correct behavior?
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Tuesday, April 14, 2015 1:37 PM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
>> Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
> This is strange. CS would never delete an entry in the nics table, only set the removed column. I see only two possibilities, 1. The entry was never added for some reason (error condition), 2. It was manually removed.
>
> Anyway, at this point, just make sure that the IP/MAC in nics and vm_instance table matches. Also, the IP you are manually setting isn't assigned to any other nic.
>
> Regards,
> Somesh
>
> -----Original Message-----
> From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
> Sent: Tuesday, April 14, 2015 2:49 PM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
> Hi Somesh,
>
>>> Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
> Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
>
>>> Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
> That was me - I was trying to sanitize the data and missed an entry.
>
>>> Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
> They did not match. I added a NIC to the VM through the web interface, and it created an entry in the "nics" table. But the IP address did not match what was in the vm_instance table.
>
> I'm not sure what the history of this VM is. Perhaps someone tried to do some restoration on it. Is it safe to clear the IP from the vm_instance table, or change it to match what's in the nics table?
>
> -Kyle
>
> -----Original Message-----
> From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
> Sent: Tuesday, April 14, 2015 10:13 AM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
>> In the WebUI, I saw no NIC associated with the machine (maybe the user
>> deleted it?)
>
> Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
>
> Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
>
> Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
>
> It would be interesting to check if this VM was destroyed at some point and then recovered manually or via recoverVirtualMachine API.
>
> Somesh
> CloudPlatform Escalations
> Citrix Systems, Inc.
>
> -----Original Message-----
> From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
> Sent: Tuesday, April 14, 2015 12:10 PM
> To: users@cloudstack.apache.org
> Subject: RE: CloudStack hands out IP address of a stopped VM
>
> In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?). So I went ahead and added one:
>
> mysql> select * from nics where instance_id='2071';
> +------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
> | id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
> +------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
> | 2793 | 84566500-6a7e-4c3a-8035-d4b98f1ecebe | 101 | 06:50:a2:00:0a:ab | 10.1.131.11 | 255.255.254.0 | 10.1.130.1 | Ip4 | vlan://111 | 206 | Dhcp | Allocated | Create | DirectNetworkGuru | NULL | 0 | 2015-04-14 09:01:30 | vlan://111 | NULL | 0 | User | 2015-04-14 16:01:30 | NULL | NULL | NULL | 0 | 1 |
> +------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
>
>
>
> However, the IP address assigned in the nics table doesn't match the one in the vm_instance table:
>
>
> mysql> select id,name,private_ip_address from vm_instance where name='myhost'; +------+--------------------+--------------------+
> | id | name | private_ip_address |
> +------+--------------------+--------------------+
> | 101 | myhost | 10.1.131.0 |
> +------+--------------------+--------------------+
> 1 row in set (0.00 sec)
>
>
> Shouldn't that IP have been updated as well?
>
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Monday, April 13, 2015 9:47 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: CloudStack hands out IP address of a stopped VM
>
> Hi Kyle,
>
> My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
> While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
>
>
> Thanks,
> Jayapal
>
> On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
>
>> When a vm is in expunging state, NIC table entry should be marked as
>> removed in DB. If you are sure about the IP address rendered by CS to
>> the VM , you can try adding the entry in db.
>>
>> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is it safe for me to attempt to re-add the entry through the
>>> database, since the VM is still in use?
>>>
>>> Also, I've noticed that there are expunged VM's with entries still in
>>> the NIC table. Should that be the case? Or should the record have
>>> been removed when the VM was deleted?
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Sunday, April 12, 2015 10:24 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> Cloudstack won't delete the nic entry, When VM is deleted it marked
>>> as removed.
>>> It might be deleted manually from the DB.
>>>
>>> Thanks,
>>> Jayapal
>>>
>>>
>>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>>
>>>> Jayapal,
>>>> Is there a reason why would the nic entry be empty? When is the
>>>> entry
>>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>>
>>>> -----Original Message-----
>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>> Sent: Thursday, April 09, 2015 9:01 PM
>>>> To: <us...@cloudstack.apache.org>
>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>
>>>> Hi Kyle,
>>>>
>>>> When CS picks the ip for new vm from free ip pool excluding nic
>>>> table
>>> ip4_adress.
>>>> In your case the nic entry for vm is empty due to this the ip will
>>>> goes
>>> free pool.
>>>>
>>>>
>>>> Thanks,
>>>> Jayapal
>>>>
>>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>>> wrote:
>>>>
>>>>> CS would not hand out the IPs of a stopped vm since the lease time
>>>>> is infinite. If you are able to reproduce is consistently please
>>>>> open a JIRA ticket.
>>>>>
>>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin
>>>>> <ky...@citrix.com>
>>> wrote:
>>>>>
>>>>>> Hi Jayapal,
>>>>>> The first query against the nics table returned an empty set,
>>>>>> while the second returned the hostname and IP address. Here is
>>>>>> the sanitized output from both queries:
>>>>>>
>>>>>>
>>>>>> mysql> select instance_id,ip4_address from nics where
>>>>>> mysql> instance_id=<id>;
>>>>>> Empty set (0.00 sec)
>>>>>>
>>>>>>
>>>>>> mysql> select id, name, private_ip_address from vm_instance where
>>>>>> id=<id>;
>>>>>> +------+--------------------+--------------------+
>>>>>> | id | name | private_ip_address |
>>>>>> +------+--------------------+--------------------+
>>>>>> | myid | myhostname | 1.1.1.1 |
>>>>>> +------+--------------------+--------------------+
>>>>>> 1 row in set (0.00 sec)
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>>> To: <us...@cloudstack.apache.org>
>>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>>
>>>>>> Hi Kyle,
>>>>>>
>>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>>> ip4_address set to 'null'.
>>>>>> After that I am not able to reproduce the issue.
>>>>>> I will keep looking into my setup for this issue.
>>>>>>
>>>>>> Can you please send the below commands output from your setup.
>>>>>>
>>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>>> vm_instance where id=<id>;
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Jayapal
>>>>>>
>>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>>> wrote:
>>>>>>
>>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>>
>>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>>> address of
>>>>>> a stopped VM to a newly created VM. The existing server was found
>>>>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>>
>>>>>>> mysql> select name,private_ip_address,state from vm_instance
>>>>>>> mysql> where name like "<myhost>%";
>>>>>>> +--------------------+--------------------+---------+
>>>>>>> | name | private_ip_address | state |
>>>>>>> +--------------------+--------------------+---------+
>>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>>> +--------------------+--------------------+---------+
>>>>>>>
>>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>>> well,
>>>>>> which caused a conflict in our external host management system.
>>>>>>>
>>>>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>>>>> and
>>>>>> then CloudStack is just handing it out again. However, it had
>>>>>> previously been explained to me that CloudStack would not hand out
>>>>>> IP's of stopped VM's (and I do see the IP address registered to
>>>>>> the VM
>>> in the database).
>>>>>> Is that true and is this a possible bug, or is that the expected
>>> behavior?
>>>>>>>
>>>>>>> -Kyle
>>>>>>
>>>>>>
>>>>
>>>
>>>
>
RE: CloudStack hands out IP address of a stopped VM
Posted by Kyle Flavin <ky...@citrix.com>.
Thanks Somesh. I added the NIC, verified there was an entry in the "nics" table, and then modified the IP/MAC in vm_instance to match what was in the nics table.
Then to test, I deleted the NIC through the Web UI. It completely removed the entry from the "nics" table, and left the IP in the vm_instance table. It doesn't sound like that's the correct behavior?
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Tuesday, April 14, 2015 1:37 PM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
> Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
This is strange. CS would never delete an entry in the nics table, only set the removed column. I see only two possibilities, 1. The entry was never added for some reason (error condition), 2. It was manually removed.
Anyway, at this point, just make sure that the IP/MAC in nics and vm_instance table matches. Also, the IP you are manually setting isn't assigned to any other nic.
Regards,
Somesh
-----Original Message-----
From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
Sent: Tuesday, April 14, 2015 2:49 PM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
Hi Somesh,
>> Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
>> Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
That was me - I was trying to sanitize the data and missed an entry.
>> Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
They did not match. I added a NIC to the VM through the web interface, and it created an entry in the "nics" table. But the IP address did not match what was in the vm_instance table.
I'm not sure what the history of this VM is. Perhaps someone tried to do some restoration on it. Is it safe to clear the IP from the vm_instance table, or change it to match what's in the nics table?
-Kyle
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Tuesday, April 14, 2015 10:13 AM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
> In the WebUI, I saw no NIC associated with the machine (maybe the user
> deleted it?)
Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
It would be interesting to check if this VM was destroyed at some point and then recovered manually or via recoverVirtualMachine API.
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
Sent: Tuesday, April 14, 2015 12:10 PM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?). So I went ahead and added one:
mysql> select * from nics where instance_id='2071';
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| 2793 | 84566500-6a7e-4c3a-8035-d4b98f1ecebe | 101 | 06:50:a2:00:0a:ab | 10.1.131.11 | 255.255.254.0 | 10.1.130.1 | Ip4 | vlan://111 | 206 | Dhcp | Allocated | Create | DirectNetworkGuru | NULL | 0 | 2015-04-14 09:01:30 | vlan://111 | NULL | 0 | User | 2015-04-14 16:01:30 | NULL | NULL | NULL | 0 | 1 |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
However, the IP address assigned in the nics table doesn't match the one in the vm_instance table:
mysql> select id,name,private_ip_address from vm_instance where name='myhost'; +------+--------------------+--------------------+
| id | name | private_ip_address |
+------+--------------------+--------------------+
| 101 | myhost | 10.1.131.0 |
+------+--------------------+--------------------+
1 row in set (0.00 sec)
Shouldn't that IP have been updated as well?
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, April 13, 2015 9:47 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
Thanks,
Jayapal
On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
> When a vm is in expunging state, NIC table entry should be marked as
> removed in DB. If you are sure about the IP address rendered by CS to
> the VM , you can try adding the entry in db.
>
> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Jayapal,
>> Is it safe for me to attempt to re-add the entry through the
>> database, since the VM is still in use?
>>
>> Also, I've noticed that there are expunged VM's with entries still in
>> the NIC table. Should that be the case? Or should the record have
>> been removed when the VM was deleted?
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Sunday, April 12, 2015 10:24 PM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> Cloudstack won't delete the nic entry, When VM is deleted it marked
>> as removed.
>> It might be deleted manually from the DB.
>>
>> Thanks,
>> Jayapal
>>
>>
>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is there a reason why would the nic entry be empty? When is the
>>> entry
>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 9:01 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> When CS picks the ip for new vm from free ip pool excluding nic
>>> table
>> ip4_adress.
>>> In your case the nic entry for vm is empty due to this the ip will
>>> goes
>> free pool.
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>> wrote:
>>>
>>>> CS would not hand out the IPs of a stopped vm since the lease time
>>>> is infinite. If you are able to reproduce is consistently please
>>>> open a JIRA ticket.
>>>>
>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin
>>>> <ky...@citrix.com>
>> wrote:
>>>>
>>>>> Hi Jayapal,
>>>>> The first query against the nics table returned an empty set,
>>>>> while the second returned the hostname and IP address. Here is
>>>>> the sanitized output from both queries:
>>>>>
>>>>>
>>>>> mysql> select instance_id,ip4_address from nics where
>>>>> mysql> instance_id=<id>;
>>>>> Empty set (0.00 sec)
>>>>>
>>>>>
>>>>> mysql> select id, name, private_ip_address from vm_instance where
>>>>> id=<id>;
>>>>> +------+--------------------+--------------------+
>>>>> | id | name | private_ip_address |
>>>>> +------+--------------------+--------------------+
>>>>> | myid | myhostname | 1.1.1.1 |
>>>>> +------+--------------------+--------------------+
>>>>> 1 row in set (0.00 sec)
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>> To: <us...@cloudstack.apache.org>
>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>
>>>>> Hi Kyle,
>>>>>
>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>> ip4_address set to 'null'.
>>>>> After that I am not able to reproduce the issue.
>>>>> I will keep looking into my setup for this issue.
>>>>>
>>>>> Can you please send the below commands output from your setup.
>>>>>
>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>> vm_instance where id=<id>;
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Jayapal
>>>>>
>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>> wrote:
>>>>>
>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>
>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>> address of
>>>>> a stopped VM to a newly created VM. The existing server was found
>>>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>
>>>>>> mysql> select name,private_ip_address,state from vm_instance
>>>>>> mysql> where name like "<myhost>%";
>>>>>> +--------------------+--------------------+---------+
>>>>>> | name | private_ip_address | state |
>>>>>> +--------------------+--------------------+---------+
>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>> +--------------------+--------------------+---------+
>>>>>>
>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>> well,
>>>>> which caused a conflict in our external host management system.
>>>>>>
>>>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>>>> and
>>>>> then CloudStack is just handing it out again. However, it had
>>>>> previously been explained to me that CloudStack would not hand out
>>>>> IP's of stopped VM's (and I do see the IP address registered to
>>>>> the VM
>> in the database).
>>>>> Is that true and is this a possible bug, or is that the expected
>> behavior?
>>>>>>
>>>>>> -Kyle
>>>>>
>>>>>
>>>
>>
>>
RE: CloudStack hands out IP address of a stopped VM
Posted by Somesh Naidu <So...@citrix.com>.
> Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
This is strange. CS would never delete an entry in the nics table, only set the removed column. I see only two possibilities, 1. The entry was never added for some reason (error condition), 2. It was manually removed.
Anyway, at this point, just make sure that the IP/MAC in nics and vm_instance table matches. Also, the IP you are manually setting isn't assigned to any other nic.
Regards,
Somesh
-----Original Message-----
From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
Sent: Tuesday, April 14, 2015 2:49 PM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
Hi Somesh,
>> Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
>> Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
That was me - I was trying to sanitize the data and missed an entry.
>> Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
They did not match. I added a NIC to the VM through the web interface, and it created an entry in the "nics" table. But the IP address did not match what was in the vm_instance table.
I'm not sure what the history of this VM is. Perhaps someone tried to do some restoration on it. Is it safe to clear the IP from the vm_instance table, or change it to match what's in the nics table?
-Kyle
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Tuesday, April 14, 2015 10:13 AM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
> In the WebUI, I saw no NIC associated with the machine (maybe the user
> deleted it?)
Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
It would be interesting to check if this VM was destroyed at some point and then recovered manually or via recoverVirtualMachine API.
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
Sent: Tuesday, April 14, 2015 12:10 PM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?). So I went ahead and added one:
mysql> select * from nics where instance_id='2071';
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| 2793 | 84566500-6a7e-4c3a-8035-d4b98f1ecebe | 101 | 06:50:a2:00:0a:ab | 10.1.131.11 | 255.255.254.0 | 10.1.130.1 | Ip4 | vlan://111 | 206 | Dhcp | Allocated | Create | DirectNetworkGuru | NULL | 0 | 2015-04-14 09:01:30 | vlan://111 | NULL | 0 | User | 2015-04-14 16:01:30 | NULL | NULL | NULL | 0 | 1 |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
However, the IP address assigned in the nics table doesn't match the one in the vm_instance table:
mysql> select id,name,private_ip_address from vm_instance where name='myhost'; +------+--------------------+--------------------+
| id | name | private_ip_address |
+------+--------------------+--------------------+
| 101 | myhost | 10.1.131.0 |
+------+--------------------+--------------------+
1 row in set (0.00 sec)
Shouldn't that IP have been updated as well?
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, April 13, 2015 9:47 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
Thanks,
Jayapal
On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
> When a vm is in expunging state, NIC table entry should be marked as
> removed in DB. If you are sure about the IP address rendered by CS to
> the VM , you can try adding the entry in db.
>
> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Jayapal,
>> Is it safe for me to attempt to re-add the entry through the
>> database, since the VM is still in use?
>>
>> Also, I've noticed that there are expunged VM's with entries still in
>> the NIC table. Should that be the case? Or should the record have
>> been removed when the VM was deleted?
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Sunday, April 12, 2015 10:24 PM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> Cloudstack won't delete the nic entry, When VM is deleted it marked
>> as removed.
>> It might be deleted manually from the DB.
>>
>> Thanks,
>> Jayapal
>>
>>
>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is there a reason why would the nic entry be empty? When is the
>>> entry
>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 9:01 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> When CS picks the ip for new vm from free ip pool excluding nic
>>> table
>> ip4_adress.
>>> In your case the nic entry for vm is empty due to this the ip will
>>> goes
>> free pool.
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>> wrote:
>>>
>>>> CS would not hand out the IPs of a stopped vm since the lease time
>>>> is infinite. If you are able to reproduce is consistently please
>>>> open a JIRA ticket.
>>>>
>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin
>>>> <ky...@citrix.com>
>> wrote:
>>>>
>>>>> Hi Jayapal,
>>>>> The first query against the nics table returned an empty set,
>>>>> while the second returned the hostname and IP address. Here is
>>>>> the sanitized output from both queries:
>>>>>
>>>>>
>>>>> mysql> select instance_id,ip4_address from nics where
>>>>> mysql> instance_id=<id>;
>>>>> Empty set (0.00 sec)
>>>>>
>>>>>
>>>>> mysql> select id, name, private_ip_address from vm_instance where
>>>>> id=<id>;
>>>>> +------+--------------------+--------------------+
>>>>> | id | name | private_ip_address |
>>>>> +------+--------------------+--------------------+
>>>>> | myid | myhostname | 1.1.1.1 |
>>>>> +------+--------------------+--------------------+
>>>>> 1 row in set (0.00 sec)
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>> To: <us...@cloudstack.apache.org>
>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>
>>>>> Hi Kyle,
>>>>>
>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>> ip4_address set to 'null'.
>>>>> After that I am not able to reproduce the issue.
>>>>> I will keep looking into my setup for this issue.
>>>>>
>>>>> Can you please send the below commands output from your setup.
>>>>>
>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>> vm_instance where id=<id>;
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Jayapal
>>>>>
>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>> wrote:
>>>>>
>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>
>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>> address of
>>>>> a stopped VM to a newly created VM. The existing server was found
>>>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>
>>>>>> mysql> select name,private_ip_address,state from vm_instance
>>>>>> mysql> where name like "<myhost>%";
>>>>>> +--------------------+--------------------+---------+
>>>>>> | name | private_ip_address | state |
>>>>>> +--------------------+--------------------+---------+
>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>> +--------------------+--------------------+---------+
>>>>>>
>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>> well,
>>>>> which caused a conflict in our external host management system.
>>>>>>
>>>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>>>> and
>>>>> then CloudStack is just handing it out again. However, it had
>>>>> previously been explained to me that CloudStack would not hand out
>>>>> IP's of stopped VM's (and I do see the IP address registered to
>>>>> the VM
>> in the database).
>>>>> Is that true and is this a possible bug, or is that the expected
>> behavior?
>>>>>>
>>>>>> -Kyle
>>>>>
>>>>>
>>>
>>
>>
RE: CloudStack hands out IP address of a stopped VM
Posted by Kyle Flavin <ky...@citrix.com>.
Hi Somesh,
>> Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
Correct. The VM was active, but stopped. There was no entry in the "nics" table for it.
>> Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
That was me - I was trying to sanitize the data and missed an entry.
>> Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
They did not match. I added a NIC to the VM through the web interface, and it created an entry in the "nics" table. But the IP address did not match what was in the vm_instance table.
I'm not sure what the history of this VM is. Perhaps someone tried to do some restoration on it. Is it safe to clear the IP from the vm_instance table, or change it to match what's in the nics table?
-Kyle
-----Original Message-----
From: Somesh Naidu [mailto:Somesh.Naidu@citrix.com]
Sent: Tuesday, April 14, 2015 10:13 AM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
> In the WebUI, I saw no NIC associated with the machine (maybe the user
> deleted it?)
Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
It would be interesting to check if this VM was destroyed at some point and then recovered manually or via recoverVirtualMachine API.
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
Sent: Tuesday, April 14, 2015 12:10 PM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?). So I went ahead and added one:
mysql> select * from nics where instance_id='2071';
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| 2793 | 84566500-6a7e-4c3a-8035-d4b98f1ecebe | 101 | 06:50:a2:00:0a:ab | 10.1.131.11 | 255.255.254.0 | 10.1.130.1 | Ip4 | vlan://111 | 206 | Dhcp | Allocated | Create | DirectNetworkGuru | NULL | 0 | 2015-04-14 09:01:30 | vlan://111 | NULL | 0 | User | 2015-04-14 16:01:30 | NULL | NULL | NULL | 0 | 1 |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
However, the IP address assigned in the nics table doesn't match the one in the vm_instance table:
mysql> select id,name,private_ip_address from vm_instance where name='myhost'; +------+--------------------+--------------------+
| id | name | private_ip_address |
+------+--------------------+--------------------+
| 101 | myhost | 10.1.131.0 |
+------+--------------------+--------------------+
1 row in set (0.00 sec)
Shouldn't that IP have been updated as well?
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, April 13, 2015 9:47 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
Thanks,
Jayapal
On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
> When a vm is in expunging state, NIC table entry should be marked as
> removed in DB. If you are sure about the IP address rendered by CS to
> the VM , you can try adding the entry in db.
>
> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Jayapal,
>> Is it safe for me to attempt to re-add the entry through the
>> database, since the VM is still in use?
>>
>> Also, I've noticed that there are expunged VM's with entries still in
>> the NIC table. Should that be the case? Or should the record have
>> been removed when the VM was deleted?
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Sunday, April 12, 2015 10:24 PM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> Cloudstack won't delete the nic entry, When VM is deleted it marked
>> as removed.
>> It might be deleted manually from the DB.
>>
>> Thanks,
>> Jayapal
>>
>>
>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is there a reason why would the nic entry be empty? When is the
>>> entry
>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 9:01 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> When CS picks the ip for new vm from free ip pool excluding nic
>>> table
>> ip4_adress.
>>> In your case the nic entry for vm is empty due to this the ip will
>>> goes
>> free pool.
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>> wrote:
>>>
>>>> CS would not hand out the IPs of a stopped vm since the lease time
>>>> is infinite. If you are able to reproduce is consistently please
>>>> open a JIRA ticket.
>>>>
>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin
>>>> <ky...@citrix.com>
>> wrote:
>>>>
>>>>> Hi Jayapal,
>>>>> The first query against the nics table returned an empty set,
>>>>> while the second returned the hostname and IP address. Here is
>>>>> the sanitized output from both queries:
>>>>>
>>>>>
>>>>> mysql> select instance_id,ip4_address from nics where
>>>>> mysql> instance_id=<id>;
>>>>> Empty set (0.00 sec)
>>>>>
>>>>>
>>>>> mysql> select id, name, private_ip_address from vm_instance where
>>>>> id=<id>;
>>>>> +------+--------------------+--------------------+
>>>>> | id | name | private_ip_address |
>>>>> +------+--------------------+--------------------+
>>>>> | myid | myhostname | 1.1.1.1 |
>>>>> +------+--------------------+--------------------+
>>>>> 1 row in set (0.00 sec)
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>> To: <us...@cloudstack.apache.org>
>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>
>>>>> Hi Kyle,
>>>>>
>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>> ip4_address set to 'null'.
>>>>> After that I am not able to reproduce the issue.
>>>>> I will keep looking into my setup for this issue.
>>>>>
>>>>> Can you please send the below commands output from your setup.
>>>>>
>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>> vm_instance where id=<id>;
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Jayapal
>>>>>
>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>> wrote:
>>>>>
>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>
>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>> address of
>>>>> a stopped VM to a newly created VM. The existing server was found
>>>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>
>>>>>> mysql> select name,private_ip_address,state from vm_instance
>>>>>> mysql> where name like "<myhost>%";
>>>>>> +--------------------+--------------------+---------+
>>>>>> | name | private_ip_address | state |
>>>>>> +--------------------+--------------------+---------+
>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>> +--------------------+--------------------+---------+
>>>>>>
>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>> well,
>>>>> which caused a conflict in our external host management system.
>>>>>>
>>>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>>>> and
>>>>> then CloudStack is just handing it out again. However, it had
>>>>> previously been explained to me that CloudStack would not hand out
>>>>> IP's of stopped VM's (and I do see the IP address registered to
>>>>> the VM
>> in the database).
>>>>> Is that true and is this a possible bug, or is that the expected
>> behavior?
>>>>>>
>>>>>> -Kyle
>>>>>
>>>>>
>>>
>>
>>
RE: CloudStack hands out IP address of a stopped VM
Posted by Somesh Naidu <So...@citrix.com>.
> In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?)
Are you saying that there was no entry in the nics table for an active VM (removed is null)? If so, that is not the correct behavior.
Also, the output of the query doesn't see right. The where clause specifies instance_id=2071, however, the result returned has instance_id=101.
Note that, for a given VM, the private_ip_address/private_mac_address in vm_instance table and ip4_address/mac_address in nics table should match.
It would be interesting to check if this VM was destroyed at some point and then recovered manually or via recoverVirtualMachine API.
Somesh
CloudPlatform Escalations
Citrix Systems, Inc.
-----Original Message-----
From: Kyle Flavin [mailto:kyle.flavin@citrix.com]
Sent: Tuesday, April 14, 2015 12:10 PM
To: users@cloudstack.apache.org
Subject: RE: CloudStack hands out IP address of a stopped VM
In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?). So I went ahead and added one:
mysql> select * from nics where instance_id='2071';
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| 2793 | 84566500-6a7e-4c3a-8035-d4b98f1ecebe | 101 | 06:50:a2:00:0a:ab | 10.1.131.11 | 255.255.254.0 | 10.1.130.1 | Ip4 | vlan://111 | 206 | Dhcp | Allocated | Create | DirectNetworkGuru | NULL | 0 | 2015-04-14 09:01:30 | vlan://111 | NULL | 0 | User | 2015-04-14 16:01:30 | NULL | NULL | NULL | 0 | 1 |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
However, the IP address assigned in the nics table doesn't match the one in the vm_instance table:
mysql> select id,name,private_ip_address from vm_instance where name='myhost'; +------+--------------------+--------------------+
| id | name | private_ip_address |
+------+--------------------+--------------------+
| 101 | myhost | 10.1.131.0 |
+------+--------------------+--------------------+
1 row in set (0.00 sec)
Shouldn't that IP have been updated as well?
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, April 13, 2015 9:47 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
Thanks,
Jayapal
On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
> When a vm is in expunging state, NIC table entry should be marked as
> removed in DB. If you are sure about the IP address rendered by CS to
> the VM , you can try adding the entry in db.
>
> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Jayapal,
>> Is it safe for me to attempt to re-add the entry through the
>> database, since the VM is still in use?
>>
>> Also, I've noticed that there are expunged VM's with entries still in
>> the NIC table. Should that be the case? Or should the record have
>> been removed when the VM was deleted?
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Sunday, April 12, 2015 10:24 PM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> Cloudstack won't delete the nic entry, When VM is deleted it marked
>> as removed.
>> It might be deleted manually from the DB.
>>
>> Thanks,
>> Jayapal
>>
>>
>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is there a reason why would the nic entry be empty? When is the
>>> entry
>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 9:01 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> When CS picks the ip for new vm from free ip pool excluding nic
>>> table
>> ip4_adress.
>>> In your case the nic entry for vm is empty due to this the ip will
>>> goes
>> free pool.
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>> wrote:
>>>
>>>> CS would not hand out the IPs of a stopped vm since the lease time
>>>> is infinite. If you are able to reproduce is consistently please
>>>> open a JIRA ticket.
>>>>
>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin
>>>> <ky...@citrix.com>
>> wrote:
>>>>
>>>>> Hi Jayapal,
>>>>> The first query against the nics table returned an empty set,
>>>>> while the second returned the hostname and IP address. Here is
>>>>> the sanitized output from both queries:
>>>>>
>>>>>
>>>>> mysql> select instance_id,ip4_address from nics where
>>>>> mysql> instance_id=<id>;
>>>>> Empty set (0.00 sec)
>>>>>
>>>>>
>>>>> mysql> select id, name, private_ip_address from vm_instance
>>>>> mysql> where
>>>>> id=<id>;
>>>>> +------+--------------------+--------------------+
>>>>> | id | name | private_ip_address |
>>>>> +------+--------------------+--------------------+
>>>>> | myid | myhostname | 1.1.1.1 |
>>>>> +------+--------------------+--------------------+
>>>>> 1 row in set (0.00 sec)
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>> To: <us...@cloudstack.apache.org>
>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>
>>>>> Hi Kyle,
>>>>>
>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>> ip4_address set to 'null'.
>>>>> After that I am not able to reproduce the issue.
>>>>> I will keep looking into my setup for this issue.
>>>>>
>>>>> Can you please send the below commands output from your setup.
>>>>>
>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>> vm_instance where id=<id>;
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Jayapal
>>>>>
>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>> wrote:
>>>>>
>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>
>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>> address of
>>>>> a stopped VM to a newly created VM. The existing server was found
>>>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>
>>>>>> mysql> select name,private_ip_address,state from vm_instance
>>>>>> mysql> where name like "<myhost>%";
>>>>>> +--------------------+--------------------+---------+
>>>>>> | name | private_ip_address | state |
>>>>>> +--------------------+--------------------+---------+
>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>> +--------------------+--------------------+---------+
>>>>>>
>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>> well,
>>>>> which caused a conflict in our external host management system.
>>>>>>
>>>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>>>> and
>>>>> then CloudStack is just handing it out again. However, it had
>>>>> previously been explained to me that CloudStack would not hand out
>>>>> IP's of stopped VM's (and I do see the IP address registered to
>>>>> the VM
>> in the database).
>>>>> Is that true and is this a possible bug, or is that the expected
>> behavior?
>>>>>>
>>>>>> -Kyle
>>>>>
>>>>>
>>>
>>
>>
RE: CloudStack hands out IP address of a stopped VM
Posted by Kyle Flavin <ky...@citrix.com>.
In the WebUI, I saw no NIC associated with the machine (maybe the user deleted it?). So I went ahead and added one:
mysql> select * from nics where instance_id='2071';
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| id | uuid | instance_id | mac_address | ip4_address | netmask | gateway | ip_type | broadcast_uri | network_id | mode | state | strategy | reserver_name | reservation_id | device_id | update_time | isolation_uri | ip6_address | default_nic | vm_type | created | removed | ip6_gateway | ip6_cidr | secondary_ip | display_nic |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
| 2793 | 84566500-6a7e-4c3a-8035-d4b98f1ecebe | 101 | 06:50:a2:00:0a:ab | 10.1.131.11 | 255.255.254.0 | 10.1.130.1 | Ip4 | vlan://111 | 206 | Dhcp | Allocated | Create | DirectNetworkGuru | NULL | 0 | 2015-04-14 09:01:30 | vlan://111 | NULL | 0 | User | 2015-04-14 16:01:30 | NULL | NULL | NULL | 0 | 1 |
+------+--------------------------------------+-------------+-------------------+---------------+---------------+--------------+---------+---------------+------------+------+-----------+----------+-------------------+----------------+-----------+---------------------+---------------+-------------+-------------+---------+---------------------+---------+-------------+----------+--------------+-------------+
However, the IP address assigned in the nics table doesn't match the one in the vm_instance table:
mysql> select id,name,private_ip_address from vm_instance where name='myhost'; +------+--------------------+--------------------+
| id | name | private_ip_address |
+------+--------------------+--------------------+
| 101 | myhost | 10.1.131.0 |
+------+--------------------+--------------------+
1 row in set (0.00 sec)
Shouldn't that IP have been updated as well?
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Monday, April 13, 2015 9:47 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
Thanks,
Jayapal
On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
> When a vm is in expunging state, NIC table entry should be marked as
> removed in DB. If you are sure about the IP address rendered by CS to
> the VM , you can try adding the entry in db.
>
> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Jayapal,
>> Is it safe for me to attempt to re-add the entry through the
>> database, since the VM is still in use?
>>
>> Also, I've noticed that there are expunged VM's with entries still in
>> the NIC table. Should that be the case? Or should the record have
>> been removed when the VM was deleted?
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Sunday, April 12, 2015 10:24 PM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> Cloudstack won't delete the nic entry, When VM is deleted it marked
>> as removed.
>> It might be deleted manually from the DB.
>>
>> Thanks,
>> Jayapal
>>
>>
>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is there a reason why would the nic entry be empty? When is the
>>> entry
>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 9:01 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> When CS picks the ip for new vm from free ip pool excluding nic
>>> table
>> ip4_adress.
>>> In your case the nic entry for vm is empty due to this the ip will
>>> goes
>> free pool.
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>> wrote:
>>>
>>>> CS would not hand out the IPs of a stopped vm since the lease time
>>>> is infinite. If you are able to reproduce is consistently please
>>>> open a JIRA ticket.
>>>>
>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin
>>>> <ky...@citrix.com>
>> wrote:
>>>>
>>>>> Hi Jayapal,
>>>>> The first query against the nics table returned an empty set,
>>>>> while the second returned the hostname and IP address. Here is
>>>>> the sanitized output from both queries:
>>>>>
>>>>>
>>>>> mysql> select instance_id,ip4_address from nics where
>>>>> mysql> instance_id=<id>;
>>>>> Empty set (0.00 sec)
>>>>>
>>>>>
>>>>> mysql> select id, name, private_ip_address from vm_instance
>>>>> mysql> where
>>>>> id=<id>;
>>>>> +------+--------------------+--------------------+
>>>>> | id | name | private_ip_address |
>>>>> +------+--------------------+--------------------+
>>>>> | myid | myhostname | 1.1.1.1 |
>>>>> +------+--------------------+--------------------+
>>>>> 1 row in set (0.00 sec)
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>> To: <us...@cloudstack.apache.org>
>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>
>>>>> Hi Kyle,
>>>>>
>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>> ip4_address set to 'null'.
>>>>> After that I am not able to reproduce the issue.
>>>>> I will keep looking into my setup for this issue.
>>>>>
>>>>> Can you please send the below commands output from your setup.
>>>>>
>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>> vm_instance where id=<id>;
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Jayapal
>>>>>
>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>> wrote:
>>>>>
>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>
>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>> address of
>>>>> a stopped VM to a newly created VM. The existing server was found
>>>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>
>>>>>> mysql> select name,private_ip_address,state from vm_instance
>>>>>> mysql> where name like "<myhost>%";
>>>>>> +--------------------+--------------------+---------+
>>>>>> | name | private_ip_address | state |
>>>>>> +--------------------+--------------------+---------+
>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>> +--------------------+--------------------+---------+
>>>>>>
>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>> well,
>>>>> which caused a conflict in our external host management system.
>>>>>>
>>>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>>>> and
>>>>> then CloudStack is just handing it out again. However, it had
>>>>> previously been explained to me that CloudStack would not hand out
>>>>> IP's of stopped VM's (and I do see the IP address registered to
>>>>> the VM
>> in the database).
>>>>> Is that true and is this a possible bug, or is that the expected
>> behavior?
>>>>>>
>>>>>> -Kyle
>>>>>
>>>>>
>>>
>>
>>
Re: CloudStack hands out IP address of a stopped VM
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Kyle,
My suggestion is take nic entry of the vm which is removed (in the same network) and edit it for your VM.
While editing take care of the values of instance_id, ip, mac_address, state, default_nic, removed, display_nic and strategy columns.
Thanks,
Jayapal
On 14-Apr-2015, at 9:18 AM, Sanjeev N <sa...@apache.org> wrote:
> When a vm is in expunging state, NIC table entry should be marked as
> removed in DB. If you are sure about the IP address rendered by CS to the
> VM , you can try adding the entry in db.
>
> On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Jayapal,
>> Is it safe for me to attempt to re-add the entry through the database,
>> since the VM is still in use?
>>
>> Also, I've noticed that there are expunged VM's with entries still in the
>> NIC table. Should that be the case? Or should the record have been
>> removed when the VM was deleted?
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Sunday, April 12, 2015 10:24 PM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> Cloudstack won't delete the nic entry, When VM is deleted it marked as
>> removed.
>> It might be deleted manually from the DB.
>>
>> Thanks,
>> Jayapal
>>
>>
>> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Jayapal,
>>> Is there a reason why would the nic entry be empty? When is the entry
>> cleared? It looks like it must have been cleared after the VM was stopped.
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 9:01 PM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> When CS picks the ip for new vm from free ip pool excluding nic table
>> ip4_adress.
>>> In your case the nic entry for vm is empty due to this the ip will goes
>> free pool.
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
>>> wrote:
>>>
>>>> CS would not hand out the IPs of a stopped vm since the lease time is
>>>> infinite. If you are able to reproduce is consistently please open a
>>>> JIRA ticket.
>>>>
>>>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin <ky...@citrix.com>
>> wrote:
>>>>
>>>>> Hi Jayapal,
>>>>> The first query against the nics table returned an empty set, while
>>>>> the second returned the hostname and IP address. Here is the
>>>>> sanitized output from both queries:
>>>>>
>>>>>
>>>>> mysql> select instance_id,ip4_address from nics where
>>>>> mysql> instance_id=<id>;
>>>>> Empty set (0.00 sec)
>>>>>
>>>>>
>>>>> mysql> select id, name, private_ip_address from vm_instance where
>>>>> id=<id>;
>>>>> +------+--------------------+--------------------+
>>>>> | id | name | private_ip_address |
>>>>> +------+--------------------+--------------------+
>>>>> | myid | myhostname | 1.1.1.1 |
>>>>> +------+--------------------+--------------------+
>>>>> 1 row in set (0.00 sec)
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>>>> Sent: Thursday, April 09, 2015 5:32 AM
>>>>> To: <us...@cloudstack.apache.org>
>>>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>>>
>>>>> Hi Kyle,
>>>>>
>>>>> In my setup I have observed this for stopped VM, the nic table
>>>>> ip4_address set to 'null'.
>>>>> After that I am not able to reproduce the issue.
>>>>> I will keep looking into my setup for this issue.
>>>>>
>>>>> Can you please send the below commands output from your setup.
>>>>>
>>>>> #select instance_id,ip4_address from nics where instance_id=
>>>>> <instance_id>; #select id, name, private_ip_address from
>>>>> vm_instance where id=<id>;
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Jayapal
>>>>>
>>>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
>> wrote:
>>>>>
>>>>>> I'm trying to get some help understanding the following behavior.
>>>>>>
>>>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>>>> address of
>>>>> a stopped VM to a newly created VM. The existing server was found
>>>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>>>
>>>>>> mysql> select name,private_ip_address,state from vm_instance where
>>>>>> mysql> name like "<myhost>%";
>>>>>> +--------------------+--------------------+---------+
>>>>>> | name | private_ip_address | state |
>>>>>> +--------------------+--------------------+---------+
>>>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>>>> +--------------------+--------------------+---------+
>>>>>>
>>>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>>>> well,
>>>>> which caused a conflict in our external host management system.
>>>>>>
>>>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>>>> and
>>>>> then CloudStack is just handing it out again. However, it had
>>>>> previously been explained to me that CloudStack would not hand out
>>>>> IP's of stopped VM's (and I do see the IP address registered to the VM
>> in the database).
>>>>> Is that true and is this a possible bug, or is that the expected
>> behavior?
>>>>>>
>>>>>> -Kyle
>>>>>
>>>>>
>>>
>>
>>
Re: CloudStack hands out IP address of a stopped VM
Posted by Sanjeev N <sa...@apache.org>.
When a vm is in expunging state, NIC table entry should be marked as
removed in DB. If you are sure about the IP address rendered by CS to the
VM , you can try adding the entry in db.
On Tue, Apr 14, 2015 at 2:45 AM, Kyle Flavin <ky...@citrix.com> wrote:
> Jayapal,
> Is it safe for me to attempt to re-add the entry through the database,
> since the VM is still in use?
>
> Also, I've noticed that there are expunged VM's with entries still in the
> NIC table. Should that be the case? Or should the record have been
> removed when the VM was deleted?
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Sunday, April 12, 2015 10:24 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: CloudStack hands out IP address of a stopped VM
>
> Hi Kyle,
>
> Cloudstack won't delete the nic entry, When VM is deleted it marked as
> removed.
> It might be deleted manually from the DB.
>
> Thanks,
> Jayapal
>
>
> On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
> > Jayapal,
> > Is there a reason why would the nic entry be empty? When is the entry
> cleared? It looks like it must have been cleared after the VM was stopped.
> >
> > -----Original Message-----
> > From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> > Sent: Thursday, April 09, 2015 9:01 PM
> > To: <us...@cloudstack.apache.org>
> > Subject: Re: CloudStack hands out IP address of a stopped VM
> >
> > Hi Kyle,
> >
> > When CS picks the ip for new vm from free ip pool excluding nic table
> ip4_adress.
> > In your case the nic entry for vm is empty due to this the ip will goes
> free pool.
> >
> >
> > Thanks,
> > Jayapal
> >
> > On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
> > wrote:
> >
> >> CS would not hand out the IPs of a stopped vm since the lease time is
> >> infinite. If you are able to reproduce is consistently please open a
> >> JIRA ticket.
> >>
> >> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin <ky...@citrix.com>
> wrote:
> >>
> >>> Hi Jayapal,
> >>> The first query against the nics table returned an empty set, while
> >>> the second returned the hostname and IP address. Here is the
> >>> sanitized output from both queries:
> >>>
> >>>
> >>> mysql> select instance_id,ip4_address from nics where
> >>> mysql> instance_id=<id>;
> >>> Empty set (0.00 sec)
> >>>
> >>>
> >>> mysql> select id, name, private_ip_address from vm_instance where
> >>> id=<id>;
> >>> +------+--------------------+--------------------+
> >>> | id | name | private_ip_address |
> >>> +------+--------------------+--------------------+
> >>> | myid | myhostname | 1.1.1.1 |
> >>> +------+--------------------+--------------------+
> >>> 1 row in set (0.00 sec)
> >>>
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> >>> Sent: Thursday, April 09, 2015 5:32 AM
> >>> To: <us...@cloudstack.apache.org>
> >>> Subject: Re: CloudStack hands out IP address of a stopped VM
> >>>
> >>> Hi Kyle,
> >>>
> >>> In my setup I have observed this for stopped VM, the nic table
> >>> ip4_address set to 'null'.
> >>> After that I am not able to reproduce the issue.
> >>> I will keep looking into my setup for this issue.
> >>>
> >>> Can you please send the below commands output from your setup.
> >>>
> >>> #select instance_id,ip4_address from nics where instance_id=
> >>> <instance_id>; #select id, name, private_ip_address from
> >>> vm_instance where id=<id>;
> >>>
> >>>
> >>>
> >>> Thanks,
> >>> Jayapal
> >>>
> >>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com>
> wrote:
> >>>
> >>>> I'm trying to get some help understanding the following behavior.
> >>>>
> >>>> Yesterday we had an instance of CloudStack giving out the IP
> >>>> address of
> >>> a stopped VM to a newly created VM. The existing server was found
> >>> in the MySQL database with the assigned IP (sanitized outputs):
> >>>>
> >>>> mysql> select name,private_ip_address,state from vm_instance where
> >>>> mysql> name like "<myhost>%";
> >>>> +--------------------+--------------------+---------+
> >>>> | name | private_ip_address | state |
> >>>> +--------------------+--------------------+---------+
> >>>> | <myhost> | 1.1.1.1 | Stopped |
> >>>> +--------------------+--------------------+---------+
> >>>>
> >>>> The new server booted up, and was given that same 1.1.1.1 IP as
> >>>> well,
> >>> which caused a conflict in our external host management system.
> >>>>
> >>>> It looks to me like the DHCP lease is expiring on the stopped VM,
> >>>> and
> >>> then CloudStack is just handing it out again. However, it had
> >>> previously been explained to me that CloudStack would not hand out
> >>> IP's of stopped VM's (and I do see the IP address registered to the VM
> in the database).
> >>> Is that true and is this a possible bug, or is that the expected
> behavior?
> >>>>
> >>>> -Kyle
> >>>
> >>>
> >
>
>
RE: CloudStack hands out IP address of a stopped VM
Posted by Kyle Flavin <ky...@citrix.com>.
Jayapal,
Is it safe for me to attempt to re-add the entry through the database, since the VM is still in use?
Also, I've noticed that there are expunged VM's with entries still in the NIC table. Should that be the case? Or should the record have been removed when the VM was deleted?
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Sunday, April 12, 2015 10:24 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
Cloudstack won't delete the nic entry, When VM is deleted it marked as removed.
It might be deleted manually from the DB.
Thanks,
Jayapal
On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
> Jayapal,
> Is there a reason why would the nic entry be empty? When is the entry cleared? It looks like it must have been cleared after the VM was stopped.
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Thursday, April 09, 2015 9:01 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: CloudStack hands out IP address of a stopped VM
>
> Hi Kyle,
>
> When CS picks the ip for new vm from free ip pool excluding nic table ip4_adress.
> In your case the nic entry for vm is empty due to this the ip will goes free pool.
>
>
> Thanks,
> Jayapal
>
> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
> wrote:
>
>> CS would not hand out the IPs of a stopped vm since the lease time is
>> infinite. If you are able to reproduce is consistently please open a
>> JIRA ticket.
>>
>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Hi Jayapal,
>>> The first query against the nics table returned an empty set, while
>>> the second returned the hostname and IP address. Here is the
>>> sanitized output from both queries:
>>>
>>>
>>> mysql> select instance_id,ip4_address from nics where
>>> mysql> instance_id=<id>;
>>> Empty set (0.00 sec)
>>>
>>>
>>> mysql> select id, name, private_ip_address from vm_instance where
>>> id=<id>;
>>> +------+--------------------+--------------------+
>>> | id | name | private_ip_address |
>>> +------+--------------------+--------------------+
>>> | myid | myhostname | 1.1.1.1 |
>>> +------+--------------------+--------------------+
>>> 1 row in set (0.00 sec)
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 5:32 AM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> In my setup I have observed this for stopped VM, the nic table
>>> ip4_address set to 'null'.
>>> After that I am not able to reproduce the issue.
>>> I will keep looking into my setup for this issue.
>>>
>>> Can you please send the below commands output from your setup.
>>>
>>> #select instance_id,ip4_address from nics where instance_id=
>>> <instance_id>; #select id, name, private_ip_address from
>>> vm_instance where id=<id>;
>>>
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>>
>>>> I'm trying to get some help understanding the following behavior.
>>>>
>>>> Yesterday we had an instance of CloudStack giving out the IP
>>>> address of
>>> a stopped VM to a newly created VM. The existing server was found
>>> in the MySQL database with the assigned IP (sanitized outputs):
>>>>
>>>> mysql> select name,private_ip_address,state from vm_instance where
>>>> mysql> name like "<myhost>%";
>>>> +--------------------+--------------------+---------+
>>>> | name | private_ip_address | state |
>>>> +--------------------+--------------------+---------+
>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>> +--------------------+--------------------+---------+
>>>>
>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>> well,
>>> which caused a conflict in our external host management system.
>>>>
>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>> and
>>> then CloudStack is just handing it out again. However, it had
>>> previously been explained to me that CloudStack would not hand out
>>> IP's of stopped VM's (and I do see the IP address registered to the VM in the database).
>>> Is that true and is this a possible bug, or is that the expected behavior?
>>>>
>>>> -Kyle
>>>
>>>
>
Re: CloudStack hands out IP address of a stopped VM
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Kyle,
Cloudstack won't delete the nic entry, When VM is deleted it marked as removed.
It might be deleted manually from the DB.
Thanks,
Jayapal
On 11-Apr-2015, at 1:49 AM, Kyle Flavin <ky...@citrix.com> wrote:
> Jayapal,
> Is there a reason why would the nic entry be empty? When is the entry cleared? It looks like it must have been cleared after the VM was stopped.
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Thursday, April 09, 2015 9:01 PM
> To: <us...@cloudstack.apache.org>
> Subject: Re: CloudStack hands out IP address of a stopped VM
>
> Hi Kyle,
>
> When CS picks the ip for new vm from free ip pool excluding nic table ip4_adress.
> In your case the nic entry for vm is empty due to this the ip will goes free pool.
>
>
> Thanks,
> Jayapal
>
> On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
> wrote:
>
>> CS would not hand out the IPs of a stopped vm since the lease time is
>> infinite. If you are able to reproduce is consistently please open a
>> JIRA ticket.
>>
>> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> Hi Jayapal,
>>> The first query against the nics table returned an empty set, while
>>> the second returned the hostname and IP address. Here is the
>>> sanitized output from both queries:
>>>
>>>
>>> mysql> select instance_id,ip4_address from nics where
>>> mysql> instance_id=<id>;
>>> Empty set (0.00 sec)
>>>
>>>
>>> mysql> select id, name, private_ip_address from vm_instance where
>>> id=<id>;
>>> +------+--------------------+--------------------+
>>> | id | name | private_ip_address |
>>> +------+--------------------+--------------------+
>>> | myid | myhostname | 1.1.1.1 |
>>> +------+--------------------+--------------------+
>>> 1 row in set (0.00 sec)
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>>> Sent: Thursday, April 09, 2015 5:32 AM
>>> To: <us...@cloudstack.apache.org>
>>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>>
>>> Hi Kyle,
>>>
>>> In my setup I have observed this for stopped VM, the nic table
>>> ip4_address set to 'null'.
>>> After that I am not able to reproduce the issue.
>>> I will keep looking into my setup for this issue.
>>>
>>> Can you please send the below commands output from your setup.
>>>
>>> #select instance_id,ip4_address from nics where instance_id=
>>> <instance_id>; #select id, name, private_ip_address from vm_instance
>>> where id=<id>;
>>>
>>>
>>>
>>> Thanks,
>>> Jayapal
>>>
>>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>>
>>>> I'm trying to get some help understanding the following behavior.
>>>>
>>>> Yesterday we had an instance of CloudStack giving out the IP address
>>>> of
>>> a stopped VM to a newly created VM. The existing server was found in
>>> the MySQL database with the assigned IP (sanitized outputs):
>>>>
>>>> mysql> select name,private_ip_address,state from vm_instance where
>>>> mysql> name like "<myhost>%";
>>>> +--------------------+--------------------+---------+
>>>> | name | private_ip_address | state |
>>>> +--------------------+--------------------+---------+
>>>> | <myhost> | 1.1.1.1 | Stopped |
>>>> +--------------------+--------------------+---------+
>>>>
>>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>>> well,
>>> which caused a conflict in our external host management system.
>>>>
>>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>>> and
>>> then CloudStack is just handing it out again. However, it had
>>> previously been explained to me that CloudStack would not hand out
>>> IP's of stopped VM's (and I do see the IP address registered to the VM in the database).
>>> Is that true and is this a possible bug, or is that the expected behavior?
>>>>
>>>> -Kyle
>>>
>>>
>
RE: CloudStack hands out IP address of a stopped VM
Posted by Kyle Flavin <ky...@citrix.com>.
Jayapal,
Is there a reason why would the nic entry be empty? When is the entry cleared? It looks like it must have been cleared after the VM was stopped.
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Thursday, April 09, 2015 9:01 PM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
When CS picks the ip for new vm from free ip pool excluding nic table ip4_adress.
In your case the nic entry for vm is empty due to this the ip will goes free pool.
Thanks,
Jayapal
On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
wrote:
> CS would not hand out the IPs of a stopped vm since the lease time is
> infinite. If you are able to reproduce is consistently please open a
> JIRA ticket.
>
> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Hi Jayapal,
>> The first query against the nics table returned an empty set, while
>> the second returned the hostname and IP address. Here is the
>> sanitized output from both queries:
>>
>>
>> mysql> select instance_id,ip4_address from nics where
>> mysql> instance_id=<id>;
>> Empty set (0.00 sec)
>>
>>
>> mysql> select id, name, private_ip_address from vm_instance where
>> id=<id>;
>> +------+--------------------+--------------------+
>> | id | name | private_ip_address |
>> +------+--------------------+--------------------+
>> | myid | myhostname | 1.1.1.1 |
>> +------+--------------------+--------------------+
>> 1 row in set (0.00 sec)
>>
>>
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Thursday, April 09, 2015 5:32 AM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> In my setup I have observed this for stopped VM, the nic table
>> ip4_address set to 'null'.
>> After that I am not able to reproduce the issue.
>> I will keep looking into my setup for this issue.
>>
>> Can you please send the below commands output from your setup.
>>
>> #select instance_id,ip4_address from nics where instance_id=
>> <instance_id>; #select id, name, private_ip_address from vm_instance
>> where id=<id>;
>>
>>
>>
>> Thanks,
>> Jayapal
>>
>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> I'm trying to get some help understanding the following behavior.
>>>
>>> Yesterday we had an instance of CloudStack giving out the IP address
>>> of
>> a stopped VM to a newly created VM. The existing server was found in
>> the MySQL database with the assigned IP (sanitized outputs):
>>>
>>> mysql> select name,private_ip_address,state from vm_instance where
>>> mysql> name like "<myhost>%";
>>> +--------------------+--------------------+---------+
>>> | name | private_ip_address | state |
>>> +--------------------+--------------------+---------+
>>> | <myhost> | 1.1.1.1 | Stopped |
>>> +--------------------+--------------------+---------+
>>>
>>> The new server booted up, and was given that same 1.1.1.1 IP as
>>> well,
>> which caused a conflict in our external host management system.
>>>
>>> It looks to me like the DHCP lease is expiring on the stopped VM,
>>> and
>> then CloudStack is just handing it out again. However, it had
>> previously been explained to me that CloudStack would not hand out
>> IP's of stopped VM's (and I do see the IP address registered to the VM in the database).
>> Is that true and is this a possible bug, or is that the expected behavior?
>>>
>>> -Kyle
>>
>>
Re: CloudStack hands out IP address of a stopped VM
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Kyle,
When CS picks the ip for new vm from free ip pool excluding nic table ip4_adress.
In your case the nic entry for vm is empty due to this the ip will goes free pool.
Thanks,
Jayapal
On 10-Apr-2015, at 9:20 AM, Sanjeev N <sa...@apache.org>
wrote:
> CS would not hand out the IPs of a stopped vm since the lease time is
> infinite. If you are able to reproduce is consistently please open a JIRA
> ticket.
>
> On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
>> Hi Jayapal,
>> The first query against the nics table returned an empty set, while the
>> second returned the hostname and IP address. Here is the sanitized output
>> from both queries:
>>
>>
>> mysql> select instance_id,ip4_address from nics where instance_id=<id>;
>> Empty set (0.00 sec)
>>
>>
>> mysql> select id, name, private_ip_address from vm_instance where
>> id=<id>;
>> +------+--------------------+--------------------+
>> | id | name | private_ip_address |
>> +------+--------------------+--------------------+
>> | myid | myhostname | 1.1.1.1 |
>> +------+--------------------+--------------------+
>> 1 row in set (0.00 sec)
>>
>>
>>
>> -----Original Message-----
>> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
>> Sent: Thursday, April 09, 2015 5:32 AM
>> To: <us...@cloudstack.apache.org>
>> Subject: Re: CloudStack hands out IP address of a stopped VM
>>
>> Hi Kyle,
>>
>> In my setup I have observed this for stopped VM, the nic table ip4_address
>> set to 'null'.
>> After that I am not able to reproduce the issue.
>> I will keep looking into my setup for this issue.
>>
>> Can you please send the below commands output from your setup.
>>
>> #select instance_id,ip4_address from nics where instance_id=
>> <instance_id>; #select id, name, private_ip_address from vm_instance
>> where id=<id>;
>>
>>
>>
>> Thanks,
>> Jayapal
>>
>> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com> wrote:
>>
>>> I'm trying to get some help understanding the following behavior.
>>>
>>> Yesterday we had an instance of CloudStack giving out the IP address of
>> a stopped VM to a newly created VM. The existing server was found in the
>> MySQL database with the assigned IP (sanitized outputs):
>>>
>>> mysql> select name,private_ip_address,state from vm_instance where
>>> mysql> name like "<myhost>%";
>>> +--------------------+--------------------+---------+
>>> | name | private_ip_address | state |
>>> +--------------------+--------------------+---------+
>>> | <myhost> | 1.1.1.1 | Stopped |
>>> +--------------------+--------------------+---------+
>>>
>>> The new server booted up, and was given that same 1.1.1.1 IP as well,
>> which caused a conflict in our external host management system.
>>>
>>> It looks to me like the DHCP lease is expiring on the stopped VM, and
>> then CloudStack is just handing it out again. However, it had previously
>> been explained to me that CloudStack would not hand out IP's of stopped
>> VM's (and I do see the IP address registered to the VM in the database).
>> Is that true and is this a possible bug, or is that the expected behavior?
>>>
>>> -Kyle
>>
>>
Re: CloudStack hands out IP address of a stopped VM
Posted by Sanjeev N <sa...@apache.org>.
CS would not hand out the IPs of a stopped vm since the lease time is
infinite. If you are able to reproduce is consistently please open a JIRA
ticket.
On Fri, Apr 10, 2015 at 4:17 AM, Kyle Flavin <ky...@citrix.com> wrote:
> Hi Jayapal,
> The first query against the nics table returned an empty set, while the
> second returned the hostname and IP address. Here is the sanitized output
> from both queries:
>
>
> mysql> select instance_id,ip4_address from nics where instance_id=<id>;
> Empty set (0.00 sec)
>
>
> mysql> select id, name, private_ip_address from vm_instance where
> id=<id>;
> +------+--------------------+--------------------+
> | id | name | private_ip_address |
> +------+--------------------+--------------------+
> | myid | myhostname | 1.1.1.1 |
> +------+--------------------+--------------------+
> 1 row in set (0.00 sec)
>
>
>
> -----Original Message-----
> From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> Sent: Thursday, April 09, 2015 5:32 AM
> To: <us...@cloudstack.apache.org>
> Subject: Re: CloudStack hands out IP address of a stopped VM
>
> Hi Kyle,
>
> In my setup I have observed this for stopped VM, the nic table ip4_address
> set to 'null'.
> After that I am not able to reproduce the issue.
> I will keep looking into my setup for this issue.
>
> Can you please send the below commands output from your setup.
>
> #select instance_id,ip4_address from nics where instance_id=
> <instance_id>; #select id, name, private_ip_address from vm_instance
> where id=<id>;
>
>
>
> Thanks,
> Jayapal
>
> On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com> wrote:
>
> > I'm trying to get some help understanding the following behavior.
> >
> > Yesterday we had an instance of CloudStack giving out the IP address of
> a stopped VM to a newly created VM. The existing server was found in the
> MySQL database with the assigned IP (sanitized outputs):
> >
> > mysql> select name,private_ip_address,state from vm_instance where
> > mysql> name like "<myhost>%";
> > +--------------------+--------------------+---------+
> > | name | private_ip_address | state |
> > +--------------------+--------------------+---------+
> > | <myhost> | 1.1.1.1 | Stopped |
> > +--------------------+--------------------+---------+
> >
> > The new server booted up, and was given that same 1.1.1.1 IP as well,
> which caused a conflict in our external host management system.
> >
> > It looks to me like the DHCP lease is expiring on the stopped VM, and
> then CloudStack is just handing it out again. However, it had previously
> been explained to me that CloudStack would not hand out IP's of stopped
> VM's (and I do see the IP address registered to the VM in the database).
> Is that true and is this a possible bug, or is that the expected behavior?
> >
> > -Kyle
>
>
RE: CloudStack hands out IP address of a stopped VM
Posted by Kyle Flavin <ky...@citrix.com>.
Hi Jayapal,
The first query against the nics table returned an empty set, while the second returned the hostname and IP address. Here is the sanitized output from both queries:
mysql> select instance_id,ip4_address from nics where instance_id=<id>;
Empty set (0.00 sec)
mysql> select id, name, private_ip_address from vm_instance where id=<id>;
+------+--------------------+--------------------+
| id | name | private_ip_address |
+------+--------------------+--------------------+
| myid | myhostname | 1.1.1.1 |
+------+--------------------+--------------------+
1 row in set (0.00 sec)
-----Original Message-----
From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
Sent: Thursday, April 09, 2015 5:32 AM
To: <us...@cloudstack.apache.org>
Subject: Re: CloudStack hands out IP address of a stopped VM
Hi Kyle,
In my setup I have observed this for stopped VM, the nic table ip4_address set to 'null'.
After that I am not able to reproduce the issue.
I will keep looking into my setup for this issue.
Can you please send the below commands output from your setup.
#select instance_id,ip4_address from nics where instance_id= <instance_id>; #select id, name, private_ip_address from vm_instance where id=<id>;
Thanks,
Jayapal
On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com> wrote:
> I'm trying to get some help understanding the following behavior.
>
> Yesterday we had an instance of CloudStack giving out the IP address of a stopped VM to a newly created VM. The existing server was found in the MySQL database with the assigned IP (sanitized outputs):
>
> mysql> select name,private_ip_address,state from vm_instance where
> mysql> name like "<myhost>%";
> +--------------------+--------------------+---------+
> | name | private_ip_address | state |
> +--------------------+--------------------+---------+
> | <myhost> | 1.1.1.1 | Stopped |
> +--------------------+--------------------+---------+
>
> The new server booted up, and was given that same 1.1.1.1 IP as well, which caused a conflict in our external host management system.
>
> It looks to me like the DHCP lease is expiring on the stopped VM, and then CloudStack is just handing it out again. However, it had previously been explained to me that CloudStack would not hand out IP's of stopped VM's (and I do see the IP address registered to the VM in the database). Is that true and is this a possible bug, or is that the expected behavior?
>
> -Kyle
Re: CloudStack hands out IP address of a stopped VM
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Kyle,
In my setup I have observed this for stopped VM, the nic table ip4_address set to 'null'.
After that I am not able to reproduce the issue.
I will keep looking into my setup for this issue.
Can you please send the below commands output from your setup.
#select instance_id,ip4_address from nics where instance_id= <instance_id>;
#select id, name, private_ip_address from vm_instance where id=<id>;
Thanks,
Jayapal
On 09-Apr-2015, at 6:10 AM, Kyle Flavin <ky...@citrix.com> wrote:
> I'm trying to get some help understanding the following behavior.
>
> Yesterday we had an instance of CloudStack giving out the IP address of a stopped VM to a newly created VM. The existing server was found in the MySQL database with the assigned IP (sanitized outputs):
>
> mysql> select name,private_ip_address,state from vm_instance where name like "<myhost>%";
> +--------------------+--------------------+---------+
> | name | private_ip_address | state |
> +--------------------+--------------------+---------+
> | <myhost> | 1.1.1.1 | Stopped |
> +--------------------+--------------------+---------+
>
> The new server booted up, and was given that same 1.1.1.1 IP as well, which caused a conflict in our external host management system.
>
> It looks to me like the DHCP lease is expiring on the stopped VM, and then CloudStack is just handing it out again. However, it had previously been explained to me that CloudStack would not hand out IP’s of stopped VM’s (and I do see the IP address registered to the VM in the database). Is that true and is this a possible bug, or is that the expected behavior?
>
> -Kyle