You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Özhan Rüzgar Karaman <or...@gmail.com> on 2018/01/22 09:07:13 UTC
[4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Hi;
Today we noticed that one of our new provisioned instance did not get IP
from VR. When we dig into the issue we find that one different mac is
written in dnsmasq.leases file holds new instances IP address.
We checked this mac address from db and we noticed that this mac is used
for old expunged instance.
So from this point we realised that when we destroy an instance its mac did
not removed from dnsmasq.leases file so if we use this ip for a new
instance then we have a problem, our instance could not get IP from VR.
We have one host on our lab environment and its Ubuntu 16.04.3 KVM. Today
we made a HA test and we crashed the host so VR and SystemVM's are rebooted
after we boot host back. I do not think this issue is related to VR reboot
but i like to give information about our environment.
We need to manage dnsmasq.leases file when we expunge an instance.
Thanks
Özhan
Logs are below:
root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
192.168.18.186 because it is leased to 1e:00:25:00:00:b9
Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:00:00:00:b9 no
address available
Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
192.168.18.187 because it is leased to 1e:00:80:00:00:ba
Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:9f:00:00:ba no
address available
root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
mysql> select name,state,private_mac_address,private_ip_address from
vm_instance;
+---------------+-----------+---------------------+--------------------+
| name | state | private_mac_address | private_ip_address |
+---------------+-----------+---------------------+--------------------+
| s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 |
| v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 |
| Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 |
| r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 |
| Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 |
| Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 |
| deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 |
| snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 |
| TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 |
| TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 |
| test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 |
| test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 |
| sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 |
| sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 |
| sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 |
+---------------+-----------+---------------------+--------------------+
15 rows in set (0.00 sec)
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
- https://github.com/apache/cloudstack/pull/2366
-https://github.com/apache/cloudstack/pull/2393
22 янв. 2018 г. 16:32 пользователь "Ivan Kudryavtsev" <
kudryavtsev_ia@bw-sw.com> написал:
> Hi, Ozhan. MACs are not removed upon vm removal, but they are overriden
> upon vm creation with same ip (or same hostname). It should work fine,
> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the case
> when IP is reused.
>
> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> oruzgarkaraman@gmail.com> написал:
>
> Hi;
> Today we noticed that one of our new provisioned instance did not get IP
> from VR. When we dig into the issue we find that one different mac is
> written in dnsmasq.leases file holds new instances IP address.
>
> We checked this mac address from db and we noticed that this mac is used
> for old expunged instance.
>
> So from this point we realised that when we destroy an instance its mac did
> not removed from dnsmasq.leases file so if we use this ip for a new
> instance then we have a problem, our instance could not get IP from VR.
>
> We have one host on our lab environment and its Ubuntu 16.04.3 KVM. Today
> we made a HA test and we crashed the host so VR and SystemVM's are rebooted
> after we boot host back. I do not think this issue is related to VR reboot
> but i like to give information about our environment.
>
> We need to manage dnsmasq.leases file when we expunge an instance.
>
> Thanks
> Özhan
>
> Logs are below:
>
> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:00:00:00:b9 no
> address available
> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:9f:00:00:ba no
> address available
>
> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
>
> mysql> select name,state,private_mac_address,private_ip_address from
> vm_instance;
> +---------------+-----------+---------------------+--------------------+
> | name | state | private_mac_address | private_ip_address |
> +---------------+-----------+---------------------+--------------------+
> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 |
> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 |
> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 |
> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 |
> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 |
> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 |
> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 |
> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 |
> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 |
> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 |
> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 |
> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 |
> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 |
> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 |
> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 |
> +---------------+-----------+---------------------+--------------------+
> 15 rows in set (0.00 sec)
>
>
>
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Daan Hoogland <da...@gmail.com>.
tnx Özhan,
that sounds like a feasible workaround. The bigger issue is that
dhcp_release should be called on instance removal. There is code to delete
the dhcp leases file already. I wonder why this isn't called in your case.
We actually have complaints that it is being cleaned to rigorously as well.
On Tue, Jan 23, 2018 at 1:01 PM, Özhan Rüzgar Karaman <
oruzgarkaraman@gmail.com> wrote:
> Hi;
> We made some more tests to find the root cause of the problem. The problem
> occur because we crashed(power off) VR for a 4.11 HA test. After rebooting
> VR, active VM's dhcp lease datas are stucked
> in /var/lib/misc/dnsmasq.leases file and this creates problems.
>
> Our solution is to clear/flush dnsmasq file using dhcp_release command.
> After dhcp_release command problem disappears for new VMs with same ip
> address.
>
> So i think we need to add some code on VR startup to flush dnsmasq.leases
> file before starting dnsmasq.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
> oruzgarkaraman@gmail.com> wrote:
>
>
--
Daan
Re: [4.11] VR Problem on Releasing Expunged Instance
IP from dnsmasq.leases file
Posted by Rohit Yadav <ro...@shapeblue.com>.
Thanks Ozhan for confirming, the fix will make its way to RC2, which we'll hopefully cut by EOD tomorrow.
- Rohit
<https://cloudstack.apache.org>
________________________________
From: Özhan Rüzgar Karaman <or...@gmail.com>
Sent: Thursday, January 25, 2018 1:12:07 PM
To: dev@cloudstack.apache.org
Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file
Hi Rohit;
I made the test again on a fresh VR and your solution fixed the issue.
Thanks
Özhan
On Wed, Jan 24, 2018 at 11:26 AM, Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi Ozhan,
>
>
> With my fix, whenever dnsmasq needs to be restarted the leases file will
> be removed, and dnsmasq will be restarted whenever /etc/dhcphosts.txt or
> /etc/dnsmasq.d/cloud.conf change, otherwise it will be reloaded.
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Özhan Rüzgar Karaman <or...@gmail.com>
> Sent: Wednesday, January 24, 2018 7:40:18 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> dnsmasq.leases file
>
> Hi Rohit;
> Today i am short in time but tomorrow i will create a new network and test
> your fix over this fresh VR.
>
> I have one more question, with your current code fix do we still continue
> reloading dnsmasq on normal operations and we only flush leases on
> start/restart operations or after this fix we start to use restart the
> dnsmasq instead of reloading it on all our operations? Thanks for all your
> help.
>
> Özhan
>
> On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <ro...@shapeblue.com>
> wrote:
>
> > Hi Ozhan,
> >
> >
> > During the 4.11-systemvmtemplate migration work (to debian9 based
> > template), I refactored the code to reload dnsmasq instead of restart it.
> > Based on your feedback, I've created a fix that will remove the leases
> file
> > everytime dnsmasq needs to be restarted.
> >
> >
> > Can you help test/verify it:
> >
> > https://github.com/apache/cloudstack/pull/2427/files
> >
> >
> >
> > - Rohit
> >
> > <https://cloudstack.apache.org>
> >
> >
> >
> > ________________________________
> > From: Özhan Rüzgar Karaman <or...@gmail.com>
> > Sent: Tuesday, January 23, 2018 1:01:16 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> > dnsmasq.leases file
> >
> > Hi;
> > We made some more tests to find the root cause of the problem. The
> problem
> > occur because we crashed(power off) VR for a 4.11 HA test. After
> rebooting
> > VR, active VM's dhcp lease datas are stucked
> > in /var/lib/misc/dnsmasq.leases file and this creates problems.
> >
> > Our solution is to clear/flush dnsmasq file using dhcp_release command.
> > After dhcp_release command problem disappears for new VMs with same ip
> > address.
> >
> > So i think we need to add some code on VR startup to flush dnsmasq.leases
> > file before starting dnsmasq.
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
> > oruzgarkaraman@gmail.com> wrote:
> >
> > > Hi Ivan;
> > > I am not sure PR 2393 directly points to my findings, i only tested
> this
> > > scenario on 4.11rc1.
> > >
> > > I am not a developer so i will not submit a fix, i am only testing
> 4.11rc
> > > because its a LTS release and its quality is very important.
> > >
> > > Please check the issue on your environment, all details and issue
> > > reproducing steps are written on my first email, but if you want i will
> > > create a PR only to report & record the situation, just send me message
> > if
> > > you want.
> > >
> > > Thanks
> > > Özhan
> > >
> > > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> > > kudryavtsev_ia@bw-sw.com> wrote:
> > >
> > >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
> > >> have
> > >> patched code? Also, describe testing scenario, I'll try to look at it
> in
> > >> my
> > >> patched 4.10.
> > >>
> > >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
> > >> oruzgarkaraman@gmail.com> написал:
> > >>
> > >> > Hi Ivan;
> > >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still
> exists
> > >> on my
> > >> > environment. When a new vm uses IP from old expunged vm then leases
> > file
> > >> > creates problem. Please check the logs that i submitted on first
> > email,
> > >> > issue is clear there and in my opinion it still exists on 4.11rc1.
> > >> >
> > >> > By the way 2393 is about VM's IP Changing progress, maybe it does
> not
> > >> cover
> > >> > my scenario.
> > >> >
> > >> > Thanks
> > >> > Özhan
> > >> >
> > >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> > >> > oruzgarkaraman@gmail.com> wrote:
> > >> >
> > >> > > Hi Ivan;
> > >> > > I made several tests with same scenario on 4.11rc1 and got same
> > >> results,
> > >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing
> > or
> > >> it
> > >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
> > >> > problem
> > >> > >
> > >> > > Thanks
> > >> > > Özhan
> > >> > >
> > >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> > >> > > kudryavtsev_ia@bw-sw.com> wrote:
> > >> > >
> > >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
> > >> overriden
> > >> > >> upon vm creation with same ip (or same hostname). It should work
> > >> fine,
> > >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested
> > the
> > >> > case
> > >> > >> when IP is reused.
> > >> > >>
> > >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> > >> > >> oruzgarkaraman@gmail.com> написал:
> > >> > >>
> > >> > >> Hi;
> > >> > >> Today we noticed that one of our new provisioned instance did not
> > >> get IP
> > >> > >> from VR. When we dig into the issue we find that one different
> mac
> > is
> > >> > >> written in dnsmasq.leases file holds new instances IP address.
> > >> > >>
> > >> > >> We checked this mac address from db and we noticed that this mac
> is
> > >> used
> > >> > >> for old expunged instance.
> > >> > >>
> > >> > >> So from this point we realised that when we destroy an instance
> its
> > >> mac
> > >> > >> did
> > >> > >> not removed from dnsmasq.leases file so if we use this ip for a
> new
> > >> > >> instance then we have a problem, our instance could not get IP
> from
> > >> VR.
> > >> > >>
> > >> > >> We have one host on our lab environment and its Ubuntu 16.04.3
> KVM.
> > >> > Today
> > >> > >> we made a HA test and we crashed the host so VR and SystemVM's
> are
> > >> > >> rebooted
> > >> > >> after we boot host back. I do not think this issue is related to
> VR
> > >> > reboot
> > >> > >> but i like to give information about our environment.
> > >> > >>
> > >> > >> We need to manage dnsmasq.leases file when we expunge an
> instance.
> > >> > >>
> > >> > >> Thanks
> > >> > >> Özhan
> > >> > >>
> > >> > >> Logs are below:
> > >> > >>
> > >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> > >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> > >> 1e:00:00:00:00:b9
> > >> > no
> > >> > >> address available
> > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> > >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> > >> 1e:00:9f:00:00:ba
> > >> > no
> > >> > >> address available
> > >> > >>
> > >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> > >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> > >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> > >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> > >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> > >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> > >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> > >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> > >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> > >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> > >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> > >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> > >> > >>
> > >> > >> mysql> select name,state,private_mac_address,private_ip_address
> > from
> > >> > >> vm_instance;
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> | name | state | private_mac_address |
> > >> private_ip_address |
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143
> > >> |
> > >> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146
> > >> |
> > >> > >> | Tolga | Expunging | 1e:00:50:00:00:bc |
> 192.168.18.189
> > >> |
> > >> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14
> > >> |
> > >> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf |
> 192.168.18.192
> > >> |
> > >> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb |
> 192.168.18.188
> > >> |
> > >> > >> | deneme | Expunging | 1e:00:80:00:00:ba |
> 192.168.18.187
> > >> |
> > >> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc |
> 192.168.18.189
> > >> |
> > >> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf |
> 192.168.18.192
> > >> |
> > >> > >> | test411rc1 | Running | 1e:00:40:00:00:bb |
> 192.168.18.188
> > >> |
> > >> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba |
> 192.168.18.187
> > >> |
> > >> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> 15 rows in set (0.00 sec)
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
> > rohit.yadav@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Özhan Rüzgar Karaman <or...@gmail.com>.
Hi Rohit;
I made the test again on a fresh VR and your solution fixed the issue.
Thanks
Özhan
On Wed, Jan 24, 2018 at 11:26 AM, Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi Ozhan,
>
>
> With my fix, whenever dnsmasq needs to be restarted the leases file will
> be removed, and dnsmasq will be restarted whenever /etc/dhcphosts.txt or
> /etc/dnsmasq.d/cloud.conf change, otherwise it will be reloaded.
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Özhan Rüzgar Karaman <or...@gmail.com>
> Sent: Wednesday, January 24, 2018 7:40:18 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> dnsmasq.leases file
>
> Hi Rohit;
> Today i am short in time but tomorrow i will create a new network and test
> your fix over this fresh VR.
>
> I have one more question, with your current code fix do we still continue
> reloading dnsmasq on normal operations and we only flush leases on
> start/restart operations or after this fix we start to use restart the
> dnsmasq instead of reloading it on all our operations? Thanks for all your
> help.
>
> Özhan
>
> On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <ro...@shapeblue.com>
> wrote:
>
> > Hi Ozhan,
> >
> >
> > During the 4.11-systemvmtemplate migration work (to debian9 based
> > template), I refactored the code to reload dnsmasq instead of restart it.
> > Based on your feedback, I've created a fix that will remove the leases
> file
> > everytime dnsmasq needs to be restarted.
> >
> >
> > Can you help test/verify it:
> >
> > https://github.com/apache/cloudstack/pull/2427/files
> >
> >
> >
> > - Rohit
> >
> > <https://cloudstack.apache.org>
> >
> >
> >
> > ________________________________
> > From: Özhan Rüzgar Karaman <or...@gmail.com>
> > Sent: Tuesday, January 23, 2018 1:01:16 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> > dnsmasq.leases file
> >
> > Hi;
> > We made some more tests to find the root cause of the problem. The
> problem
> > occur because we crashed(power off) VR for a 4.11 HA test. After
> rebooting
> > VR, active VM's dhcp lease datas are stucked
> > in /var/lib/misc/dnsmasq.leases file and this creates problems.
> >
> > Our solution is to clear/flush dnsmasq file using dhcp_release command.
> > After dhcp_release command problem disappears for new VMs with same ip
> > address.
> >
> > So i think we need to add some code on VR startup to flush dnsmasq.leases
> > file before starting dnsmasq.
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
> > oruzgarkaraman@gmail.com> wrote:
> >
> > > Hi Ivan;
> > > I am not sure PR 2393 directly points to my findings, i only tested
> this
> > > scenario on 4.11rc1.
> > >
> > > I am not a developer so i will not submit a fix, i am only testing
> 4.11rc
> > > because its a LTS release and its quality is very important.
> > >
> > > Please check the issue on your environment, all details and issue
> > > reproducing steps are written on my first email, but if you want i will
> > > create a PR only to report & record the situation, just send me message
> > if
> > > you want.
> > >
> > > Thanks
> > > Özhan
> > >
> > > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> > > kudryavtsev_ia@bw-sw.com> wrote:
> > >
> > >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
> > >> have
> > >> patched code? Also, describe testing scenario, I'll try to look at it
> in
> > >> my
> > >> patched 4.10.
> > >>
> > >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
> > >> oruzgarkaraman@gmail.com> написал:
> > >>
> > >> > Hi Ivan;
> > >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still
> exists
> > >> on my
> > >> > environment. When a new vm uses IP from old expunged vm then leases
> > file
> > >> > creates problem. Please check the logs that i submitted on first
> > email,
> > >> > issue is clear there and in my opinion it still exists on 4.11rc1.
> > >> >
> > >> > By the way 2393 is about VM's IP Changing progress, maybe it does
> not
> > >> cover
> > >> > my scenario.
> > >> >
> > >> > Thanks
> > >> > Özhan
> > >> >
> > >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> > >> > oruzgarkaraman@gmail.com> wrote:
> > >> >
> > >> > > Hi Ivan;
> > >> > > I made several tests with same scenario on 4.11rc1 and got same
> > >> results,
> > >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing
> > or
> > >> it
> > >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
> > >> > problem
> > >> > >
> > >> > > Thanks
> > >> > > Özhan
> > >> > >
> > >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> > >> > > kudryavtsev_ia@bw-sw.com> wrote:
> > >> > >
> > >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
> > >> overriden
> > >> > >> upon vm creation with same ip (or same hostname). It should work
> > >> fine,
> > >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested
> > the
> > >> > case
> > >> > >> when IP is reused.
> > >> > >>
> > >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> > >> > >> oruzgarkaraman@gmail.com> написал:
> > >> > >>
> > >> > >> Hi;
> > >> > >> Today we noticed that one of our new provisioned instance did not
> > >> get IP
> > >> > >> from VR. When we dig into the issue we find that one different
> mac
> > is
> > >> > >> written in dnsmasq.leases file holds new instances IP address.
> > >> > >>
> > >> > >> We checked this mac address from db and we noticed that this mac
> is
> > >> used
> > >> > >> for old expunged instance.
> > >> > >>
> > >> > >> So from this point we realised that when we destroy an instance
> its
> > >> mac
> > >> > >> did
> > >> > >> not removed from dnsmasq.leases file so if we use this ip for a
> new
> > >> > >> instance then we have a problem, our instance could not get IP
> from
> > >> VR.
> > >> > >>
> > >> > >> We have one host on our lab environment and its Ubuntu 16.04.3
> KVM.
> > >> > Today
> > >> > >> we made a HA test and we crashed the host so VR and SystemVM's
> are
> > >> > >> rebooted
> > >> > >> after we boot host back. I do not think this issue is related to
> VR
> > >> > reboot
> > >> > >> but i like to give information about our environment.
> > >> > >>
> > >> > >> We need to manage dnsmasq.leases file when we expunge an
> instance.
> > >> > >>
> > >> > >> Thanks
> > >> > >> Özhan
> > >> > >>
> > >> > >> Logs are below:
> > >> > >>
> > >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> > >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> > >> 1e:00:00:00:00:b9
> > >> > no
> > >> > >> address available
> > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> > >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> > >> 1e:00:9f:00:00:ba
> > >> > no
> > >> > >> address available
> > >> > >>
> > >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> > >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> > >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> > >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> > >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> > >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> > >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> > >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> > >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> > >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> > >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> > >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> > >> > >>
> > >> > >> mysql> select name,state,private_mac_address,private_ip_address
> > from
> > >> > >> vm_instance;
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> | name | state | private_mac_address |
> > >> private_ip_address |
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143
> > >> |
> > >> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146
> > >> |
> > >> > >> | Tolga | Expunging | 1e:00:50:00:00:bc |
> 192.168.18.189
> > >> |
> > >> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14
> > >> |
> > >> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf |
> 192.168.18.192
> > >> |
> > >> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb |
> 192.168.18.188
> > >> |
> > >> > >> | deneme | Expunging | 1e:00:80:00:00:ba |
> 192.168.18.187
> > >> |
> > >> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc |
> 192.168.18.189
> > >> |
> > >> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf |
> 192.168.18.192
> > >> |
> > >> > >> | test411rc1 | Running | 1e:00:40:00:00:bb |
> 192.168.18.188
> > >> |
> > >> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba |
> 192.168.18.187
> > >> |
> > >> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 |
> 192.168.18.186
> > >> |
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> 15 rows in set (0.00 sec)
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
> > rohit.yadav@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
Re: [4.11] VR Problem on Releasing Expunged Instance
IP from dnsmasq.leases file
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Ozhan,
With my fix, whenever dnsmasq needs to be restarted the leases file will be removed, and dnsmasq will be restarted whenever /etc/dhcphosts.txt or /etc/dnsmasq.d/cloud.conf change, otherwise it will be reloaded.
- Rohit
<https://cloudstack.apache.org>
________________________________
From: Özhan Rüzgar Karaman <or...@gmail.com>
Sent: Wednesday, January 24, 2018 7:40:18 AM
To: dev@cloudstack.apache.org
Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file
Hi Rohit;
Today i am short in time but tomorrow i will create a new network and test
your fix over this fresh VR.
I have one more question, with your current code fix do we still continue
reloading dnsmasq on normal operations and we only flush leases on
start/restart operations or after this fix we start to use restart the
dnsmasq instead of reloading it on all our operations? Thanks for all your
help.
Özhan
On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi Ozhan,
>
>
> During the 4.11-systemvmtemplate migration work (to debian9 based
> template), I refactored the code to reload dnsmasq instead of restart it.
> Based on your feedback, I've created a fix that will remove the leases file
> everytime dnsmasq needs to be restarted.
>
>
> Can you help test/verify it:
>
> https://github.com/apache/cloudstack/pull/2427/files
>
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Özhan Rüzgar Karaman <or...@gmail.com>
> Sent: Tuesday, January 23, 2018 1:01:16 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> dnsmasq.leases file
>
> Hi;
> We made some more tests to find the root cause of the problem. The problem
> occur because we crashed(power off) VR for a 4.11 HA test. After rebooting
> VR, active VM's dhcp lease datas are stucked
> in /var/lib/misc/dnsmasq.leases file and this creates problems.
>
> Our solution is to clear/flush dnsmasq file using dhcp_release command.
> After dhcp_release command problem disappears for new VMs with same ip
> address.
>
> So i think we need to add some code on VR startup to flush dnsmasq.leases
> file before starting dnsmasq.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
> oruzgarkaraman@gmail.com> wrote:
>
> > Hi Ivan;
> > I am not sure PR 2393 directly points to my findings, i only tested this
> > scenario on 4.11rc1.
> >
> > I am not a developer so i will not submit a fix, i am only testing 4.11rc
> > because its a LTS release and its quality is very important.
> >
> > Please check the issue on your environment, all details and issue
> > reproducing steps are written on my first email, but if you want i will
> > create a PR only to report & record the situation, just send me message
> if
> > you want.
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com> wrote:
> >
> >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
> >> have
> >> patched code? Also, describe testing scenario, I'll try to look at it in
> >> my
> >> patched 4.10.
> >>
> >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
> >> oruzgarkaraman@gmail.com> написал:
> >>
> >> > Hi Ivan;
> >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists
> >> on my
> >> > environment. When a new vm uses IP from old expunged vm then leases
> file
> >> > creates problem. Please check the logs that i submitted on first
> email,
> >> > issue is clear there and in my opinion it still exists on 4.11rc1.
> >> >
> >> > By the way 2393 is about VM's IP Changing progress, maybe it does not
> >> cover
> >> > my scenario.
> >> >
> >> > Thanks
> >> > Özhan
> >> >
> >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> >> > oruzgarkaraman@gmail.com> wrote:
> >> >
> >> > > Hi Ivan;
> >> > > I made several tests with same scenario on 4.11rc1 and got same
> >> results,
> >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing
> or
> >> it
> >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
> >> > problem
> >> > >
> >> > > Thanks
> >> > > Özhan
> >> > >
> >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> >> > > kudryavtsev_ia@bw-sw.com> wrote:
> >> > >
> >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
> >> overriden
> >> > >> upon vm creation with same ip (or same hostname). It should work
> >> fine,
> >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested
> the
> >> > case
> >> > >> when IP is reused.
> >> > >>
> >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> >> > >> oruzgarkaraman@gmail.com> написал:
> >> > >>
> >> > >> Hi;
> >> > >> Today we noticed that one of our new provisioned instance did not
> >> get IP
> >> > >> from VR. When we dig into the issue we find that one different mac
> is
> >> > >> written in dnsmasq.leases file holds new instances IP address.
> >> > >>
> >> > >> We checked this mac address from db and we noticed that this mac is
> >> used
> >> > >> for old expunged instance.
> >> > >>
> >> > >> So from this point we realised that when we destroy an instance its
> >> mac
> >> > >> did
> >> > >> not removed from dnsmasq.leases file so if we use this ip for a new
> >> > >> instance then we have a problem, our instance could not get IP from
> >> VR.
> >> > >>
> >> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM.
> >> > Today
> >> > >> we made a HA test and we crashed the host so VR and SystemVM's are
> >> > >> rebooted
> >> > >> after we boot host back. I do not think this issue is related to VR
> >> > reboot
> >> > >> but i like to give information about our environment.
> >> > >>
> >> > >> We need to manage dnsmasq.leases file when we expunge an instance.
> >> > >>
> >> > >> Thanks
> >> > >> Özhan
> >> > >>
> >> > >> Logs are below:
> >> > >>
> >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> >> 1e:00:00:00:00:b9
> >> > no
> >> > >> address available
> >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> >> 1e:00:9f:00:00:ba
> >> > no
> >> > >> address available
> >> > >>
> >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> >> > >>
> >> > >> mysql> select name,state,private_mac_address,private_ip_address
> from
> >> > >> vm_instance;
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> | name | state | private_mac_address |
> >> private_ip_address |
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143
> >> |
> >> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146
> >> |
> >> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189
> >> |
> >> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14
> >> |
> >> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192
> >> |
> >> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188
> >> |
> >> > >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187
> >> |
> >> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186
> >> |
> >> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189
> >> |
> >> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192
> >> |
> >> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188
> >> |
> >> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187
> >> |
> >> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186
> >> |
> >> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186
> >> |
> >> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186
> >> |
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> 15 rows in set (0.00 sec)
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Özhan Rüzgar Karaman <or...@gmail.com>.
Hi Rohit;
Today i am short in time but tomorrow i will create a new network and test
your fix over this fresh VR.
I have one more question, with your current code fix do we still continue
reloading dnsmasq on normal operations and we only flush leases on
start/restart operations or after this fix we start to use restart the
dnsmasq instead of reloading it on all our operations? Thanks for all your
help.
Özhan
On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <ro...@shapeblue.com>
wrote:
> Hi Ozhan,
>
>
> During the 4.11-systemvmtemplate migration work (to debian9 based
> template), I refactored the code to reload dnsmasq instead of restart it.
> Based on your feedback, I've created a fix that will remove the leases file
> everytime dnsmasq needs to be restarted.
>
>
> Can you help test/verify it:
>
> https://github.com/apache/cloudstack/pull/2427/files
>
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Özhan Rüzgar Karaman <or...@gmail.com>
> Sent: Tuesday, January 23, 2018 1:01:16 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> dnsmasq.leases file
>
> Hi;
> We made some more tests to find the root cause of the problem. The problem
> occur because we crashed(power off) VR for a 4.11 HA test. After rebooting
> VR, active VM's dhcp lease datas are stucked
> in /var/lib/misc/dnsmasq.leases file and this creates problems.
>
> Our solution is to clear/flush dnsmasq file using dhcp_release command.
> After dhcp_release command problem disappears for new VMs with same ip
> address.
>
> So i think we need to add some code on VR startup to flush dnsmasq.leases
> file before starting dnsmasq.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
> oruzgarkaraman@gmail.com> wrote:
>
> > Hi Ivan;
> > I am not sure PR 2393 directly points to my findings, i only tested this
> > scenario on 4.11rc1.
> >
> > I am not a developer so i will not submit a fix, i am only testing 4.11rc
> > because its a LTS release and its quality is very important.
> >
> > Please check the issue on your environment, all details and issue
> > reproducing steps are written on my first email, but if you want i will
> > create a PR only to report & record the situation, just send me message
> if
> > you want.
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com> wrote:
> >
> >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
> >> have
> >> patched code? Also, describe testing scenario, I'll try to look at it in
> >> my
> >> patched 4.10.
> >>
> >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
> >> oruzgarkaraman@gmail.com> написал:
> >>
> >> > Hi Ivan;
> >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists
> >> on my
> >> > environment. When a new vm uses IP from old expunged vm then leases
> file
> >> > creates problem. Please check the logs that i submitted on first
> email,
> >> > issue is clear there and in my opinion it still exists on 4.11rc1.
> >> >
> >> > By the way 2393 is about VM's IP Changing progress, maybe it does not
> >> cover
> >> > my scenario.
> >> >
> >> > Thanks
> >> > Özhan
> >> >
> >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> >> > oruzgarkaraman@gmail.com> wrote:
> >> >
> >> > > Hi Ivan;
> >> > > I made several tests with same scenario on 4.11rc1 and got same
> >> results,
> >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing
> or
> >> it
> >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
> >> > problem
> >> > >
> >> > > Thanks
> >> > > Özhan
> >> > >
> >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> >> > > kudryavtsev_ia@bw-sw.com> wrote:
> >> > >
> >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
> >> overriden
> >> > >> upon vm creation with same ip (or same hostname). It should work
> >> fine,
> >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested
> the
> >> > case
> >> > >> when IP is reused.
> >> > >>
> >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> >> > >> oruzgarkaraman@gmail.com> написал:
> >> > >>
> >> > >> Hi;
> >> > >> Today we noticed that one of our new provisioned instance did not
> >> get IP
> >> > >> from VR. When we dig into the issue we find that one different mac
> is
> >> > >> written in dnsmasq.leases file holds new instances IP address.
> >> > >>
> >> > >> We checked this mac address from db and we noticed that this mac is
> >> used
> >> > >> for old expunged instance.
> >> > >>
> >> > >> So from this point we realised that when we destroy an instance its
> >> mac
> >> > >> did
> >> > >> not removed from dnsmasq.leases file so if we use this ip for a new
> >> > >> instance then we have a problem, our instance could not get IP from
> >> VR.
> >> > >>
> >> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM.
> >> > Today
> >> > >> we made a HA test and we crashed the host so VR and SystemVM's are
> >> > >> rebooted
> >> > >> after we boot host back. I do not think this issue is related to VR
> >> > reboot
> >> > >> but i like to give information about our environment.
> >> > >>
> >> > >> We need to manage dnsmasq.leases file when we expunge an instance.
> >> > >>
> >> > >> Thanks
> >> > >> Özhan
> >> > >>
> >> > >> Logs are below:
> >> > >>
> >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> >> 1e:00:00:00:00:b9
> >> > no
> >> > >> address available
> >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> >> 1e:00:9f:00:00:ba
> >> > no
> >> > >> address available
> >> > >>
> >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> >> > >>
> >> > >> mysql> select name,state,private_mac_address,private_ip_address
> from
> >> > >> vm_instance;
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> | name | state | private_mac_address |
> >> private_ip_address |
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143
> >> |
> >> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146
> >> |
> >> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189
> >> |
> >> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14
> >> |
> >> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192
> >> |
> >> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188
> >> |
> >> > >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187
> >> |
> >> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186
> >> |
> >> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189
> >> |
> >> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192
> >> |
> >> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188
> >> |
> >> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187
> >> |
> >> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186
> >> |
> >> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186
> >> |
> >> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186
> >> |
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> 15 rows in set (0.00 sec)
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
Re: [4.11] VR Problem on Releasing Expunged Instance
IP from dnsmasq.leases file
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Ozhan,
During the 4.11-systemvmtemplate migration work (to debian9 based template), I refactored the code to reload dnsmasq instead of restart it. Based on your feedback, I've created a fix that will remove the leases file everytime dnsmasq needs to be restarted.
Can you help test/verify it:
https://github.com/apache/cloudstack/pull/2427/files
- Rohit
<https://cloudstack.apache.org>
________________________________
From: Özhan Rüzgar Karaman <or...@gmail.com>
Sent: Tuesday, January 23, 2018 1:01:16 PM
To: dev@cloudstack.apache.org
Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file
Hi;
We made some more tests to find the root cause of the problem. The problem
occur because we crashed(power off) VR for a 4.11 HA test. After rebooting
VR, active VM's dhcp lease datas are stucked
in /var/lib/misc/dnsmasq.leases file and this creates problems.
Our solution is to clear/flush dnsmasq file using dhcp_release command.
After dhcp_release command problem disappears for new VMs with same ip
address.
So i think we need to add some code on VR startup to flush dnsmasq.leases
file before starting dnsmasq.
Thanks
Özhan
On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
oruzgarkaraman@gmail.com> wrote:
> Hi Ivan;
> I am not sure PR 2393 directly points to my findings, i only tested this
> scenario on 4.11rc1.
>
> I am not a developer so i will not submit a fix, i am only testing 4.11rc
> because its a LTS release and its quality is very important.
>
> Please check the issue on your environment, all details and issue
> reproducing steps are written on my first email, but if you want i will
> create a PR only to report & record the situation, just send me message if
> you want.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com> wrote:
>
>> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
>> have
>> patched code? Also, describe testing scenario, I'll try to look at it in
>> my
>> patched 4.10.
>>
>> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
>> oruzgarkaraman@gmail.com> написал:
>>
>> > Hi Ivan;
>> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists
>> on my
>> > environment. When a new vm uses IP from old expunged vm then leases file
>> > creates problem. Please check the logs that i submitted on first email,
>> > issue is clear there and in my opinion it still exists on 4.11rc1.
>> >
>> > By the way 2393 is about VM's IP Changing progress, maybe it does not
>> cover
>> > my scenario.
>> >
>> > Thanks
>> > Özhan
>> >
>> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
>> > oruzgarkaraman@gmail.com> wrote:
>> >
>> > > Hi Ivan;
>> > > I made several tests with same scenario on 4.11rc1 and got same
>> results,
>> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing or
>> it
>> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
>> > problem
>> > >
>> > > Thanks
>> > > Özhan
>> > >
>> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
>> > > kudryavtsev_ia@bw-sw.com> wrote:
>> > >
>> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
>> overriden
>> > >> upon vm creation with same ip (or same hostname). It should work
>> fine,
>> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the
>> > case
>> > >> when IP is reused.
>> > >>
>> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
>> > >> oruzgarkaraman@gmail.com> написал:
>> > >>
>> > >> Hi;
>> > >> Today we noticed that one of our new provisioned instance did not
>> get IP
>> > >> from VR. When we dig into the issue we find that one different mac is
>> > >> written in dnsmasq.leases file holds new instances IP address.
>> > >>
>> > >> We checked this mac address from db and we noticed that this mac is
>> used
>> > >> for old expunged instance.
>> > >>
>> > >> So from this point we realised that when we destroy an instance its
>> mac
>> > >> did
>> > >> not removed from dnsmasq.leases file so if we use this ip for a new
>> > >> instance then we have a problem, our instance could not get IP from
>> VR.
>> > >>
>> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM.
>> > Today
>> > >> we made a HA test and we crashed the host so VR and SystemVM's are
>> > >> rebooted
>> > >> after we boot host back. I do not think this issue is related to VR
>> > reboot
>> > >> but i like to give information about our environment.
>> > >>
>> > >> We need to manage dnsmasq.leases file when we expunge an instance.
>> > >>
>> > >> Thanks
>> > >> Özhan
>> > >>
>> > >> Logs are below:
>> > >>
>> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
>> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
>> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
>> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
>> 1e:00:00:00:00:b9
>> > no
>> > >> address available
>> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
>> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
>> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
>> 1e:00:9f:00:00:ba
>> > no
>> > >> address available
>> > >>
>> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
>> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
>> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
>> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
>> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
>> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
>> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
>> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
>> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
>> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
>> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
>> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
>> > >>
>> > >> mysql> select name,state,private_mac_address,private_ip_address from
>> > >> vm_instance;
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> | name | state | private_mac_address |
>> private_ip_address |
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143
>> |
>> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146
>> |
>> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189
>> |
>> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14
>> |
>> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192
>> |
>> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188
>> |
>> > >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187
>> |
>> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186
>> |
>> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189
>> |
>> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192
>> |
>> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188
>> |
>> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187
>> |
>> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186
>> |
>> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186
>> |
>> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186
>> |
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> 15 rows in set (0.00 sec)
>> > >>
>> > >
>> > >
>> >
>>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Özhan Rüzgar Karaman <or...@gmail.com>.
Hi;
We made some more tests to find the root cause of the problem. The problem
occur because we crashed(power off) VR for a 4.11 HA test. After rebooting
VR, active VM's dhcp lease datas are stucked
in /var/lib/misc/dnsmasq.leases file and this creates problems.
Our solution is to clear/flush dnsmasq file using dhcp_release command.
After dhcp_release command problem disappears for new VMs with same ip
address.
So i think we need to add some code on VR startup to flush dnsmasq.leases
file before starting dnsmasq.
Thanks
Özhan
On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
oruzgarkaraman@gmail.com> wrote:
> Hi Ivan;
> I am not sure PR 2393 directly points to my findings, i only tested this
> scenario on 4.11rc1.
>
> I am not a developer so i will not submit a fix, i am only testing 4.11rc
> because its a LTS release and its quality is very important.
>
> Please check the issue on your environment, all details and issue
> reproducing steps are written on my first email, but if you want i will
> create a PR only to report & record the situation, just send me message if
> you want.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com> wrote:
>
>> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
>> have
>> patched code? Also, describe testing scenario, I'll try to look at it in
>> my
>> patched 4.10.
>>
>> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
>> oruzgarkaraman@gmail.com> написал:
>>
>> > Hi Ivan;
>> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists
>> on my
>> > environment. When a new vm uses IP from old expunged vm then leases file
>> > creates problem. Please check the logs that i submitted on first email,
>> > issue is clear there and in my opinion it still exists on 4.11rc1.
>> >
>> > By the way 2393 is about VM's IP Changing progress, maybe it does not
>> cover
>> > my scenario.
>> >
>> > Thanks
>> > Özhan
>> >
>> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
>> > oruzgarkaraman@gmail.com> wrote:
>> >
>> > > Hi Ivan;
>> > > I made several tests with same scenario on 4.11rc1 and got same
>> results,
>> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing or
>> it
>> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
>> > problem
>> > >
>> > > Thanks
>> > > Özhan
>> > >
>> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
>> > > kudryavtsev_ia@bw-sw.com> wrote:
>> > >
>> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
>> overriden
>> > >> upon vm creation with same ip (or same hostname). It should work
>> fine,
>> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the
>> > case
>> > >> when IP is reused.
>> > >>
>> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
>> > >> oruzgarkaraman@gmail.com> написал:
>> > >>
>> > >> Hi;
>> > >> Today we noticed that one of our new provisioned instance did not
>> get IP
>> > >> from VR. When we dig into the issue we find that one different mac is
>> > >> written in dnsmasq.leases file holds new instances IP address.
>> > >>
>> > >> We checked this mac address from db and we noticed that this mac is
>> used
>> > >> for old expunged instance.
>> > >>
>> > >> So from this point we realised that when we destroy an instance its
>> mac
>> > >> did
>> > >> not removed from dnsmasq.leases file so if we use this ip for a new
>> > >> instance then we have a problem, our instance could not get IP from
>> VR.
>> > >>
>> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM.
>> > Today
>> > >> we made a HA test and we crashed the host so VR and SystemVM's are
>> > >> rebooted
>> > >> after we boot host back. I do not think this issue is related to VR
>> > reboot
>> > >> but i like to give information about our environment.
>> > >>
>> > >> We need to manage dnsmasq.leases file when we expunge an instance.
>> > >>
>> > >> Thanks
>> > >> Özhan
>> > >>
>> > >> Logs are below:
>> > >>
>> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
>> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
>> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
>> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
>> 1e:00:00:00:00:b9
>> > no
>> > >> address available
>> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
>> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
>> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
>> 1e:00:9f:00:00:ba
>> > no
>> > >> address available
>> > >>
>> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
>> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
>> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
>> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
>> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
>> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
>> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
>> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
>> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
>> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
>> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
>> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
>> > >>
>> > >> mysql> select name,state,private_mac_address,private_ip_address from
>> > >> vm_instance;
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> | name | state | private_mac_address |
>> private_ip_address |
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143
>> |
>> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146
>> |
>> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189
>> |
>> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14
>> |
>> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192
>> |
>> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188
>> |
>> > >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187
>> |
>> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186
>> |
>> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189
>> |
>> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192
>> |
>> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188
>> |
>> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187
>> |
>> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186
>> |
>> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186
>> |
>> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186
>> |
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> 15 rows in set (0.00 sec)
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Özhan Rüzgar Karaman <or...@gmail.com>.
Hi Ivan;
I am not sure PR 2393 directly points to my findings, i only tested this
scenario on 4.11rc1.
I am not a developer so i will not submit a fix, i am only testing 4.11rc
because its a LTS release and its quality is very important.
Please check the issue on your environment, all details and issue
reproducing steps are written on my first email, but if you want i will
create a PR only to report & record the situation, just send me message if
you want.
Thanks
Özhan
On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <ku...@bw-sw.com>
wrote:
> Reopen issue, do a PR with fix if you can, could it be that VR doesn't have
> patched code? Also, describe testing scenario, I'll try to look at it in my
> patched 4.10.
>
> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
> oruzgarkaraman@gmail.com> написал:
>
> > Hi Ivan;
> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists on
> my
> > environment. When a new vm uses IP from old expunged vm then leases file
> > creates problem. Please check the logs that i submitted on first email,
> > issue is clear there and in my opinion it still exists on 4.11rc1.
> >
> > By the way 2393 is about VM's IP Changing progress, maybe it does not
> cover
> > my scenario.
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> > oruzgarkaraman@gmail.com> wrote:
> >
> > > Hi Ivan;
> > > I made several tests with same scenario on 4.11rc1 and got same
> results,
> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing or
> it
> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
> > problem
> > >
> > > Thanks
> > > Özhan
> > >
> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> > > kudryavtsev_ia@bw-sw.com> wrote:
> > >
> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
> overriden
> > >> upon vm creation with same ip (or same hostname). It should work fine,
> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the
> > case
> > >> when IP is reused.
> > >>
> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> > >> oruzgarkaraman@gmail.com> написал:
> > >>
> > >> Hi;
> > >> Today we noticed that one of our new provisioned instance did not get
> IP
> > >> from VR. When we dig into the issue we find that one different mac is
> > >> written in dnsmasq.leases file holds new instances IP address.
> > >>
> > >> We checked this mac address from db and we noticed that this mac is
> used
> > >> for old expunged instance.
> > >>
> > >> So from this point we realised that when we destroy an instance its
> mac
> > >> did
> > >> not removed from dnsmasq.leases file so if we use this ip for a new
> > >> instance then we have a problem, our instance could not get IP from
> VR.
> > >>
> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM.
> > Today
> > >> we made a HA test and we crashed the host so VR and SystemVM's are
> > >> rebooted
> > >> after we boot host back. I do not think this issue is related to VR
> > reboot
> > >> but i like to give information about our environment.
> > >>
> > >> We need to manage dnsmasq.leases file when we expunge an instance.
> > >>
> > >> Thanks
> > >> Özhan
> > >>
> > >> Logs are below:
> > >>
> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> 1e:00:00:00:00:b9
> > no
> > >> address available
> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> 1e:00:9f:00:00:ba
> > no
> > >> address available
> > >>
> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> > >>
> > >> mysql> select name,state,private_mac_address,private_ip_address from
> > >> vm_instance;
> > >> +---------------+-----------+---------------------+---------
> > -----------+
> > >> | name | state | private_mac_address |
> private_ip_address |
> > >> +---------------+-----------+---------------------+---------
> > -----------+
> > >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143
> |
> > >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146
> |
> > >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189
> |
> > >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14
> |
> > >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192
> |
> > >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188
> |
> > >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187
> |
> > >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186
> |
> > >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189
> |
> > >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192
> |
> > >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188
> |
> > >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187
> |
> > >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186
> |
> > >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186
> |
> > >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186
> |
> > >> +---------------+-----------+---------------------+---------
> > -----------+
> > >> 15 rows in set (0.00 sec)
> > >>
> > >
> > >
> >
>
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Reopen issue, do a PR with fix if you can, could it be that VR doesn't have
patched code? Also, describe testing scenario, I'll try to look at it in my
patched 4.10.
22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
oruzgarkaraman@gmail.com> написал:
> Hi Ivan;
> I checked 2 PR's and they are exist on 4.11rc1 but issue still exists on my
> environment. When a new vm uses IP from old expunged vm then leases file
> creates problem. Please check the logs that i submitted on first email,
> issue is clear there and in my opinion it still exists on 4.11rc1.
>
> By the way 2393 is about VM's IP Changing progress, maybe it does not cover
> my scenario.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> oruzgarkaraman@gmail.com> wrote:
>
> > Hi Ivan;
> > I made several tests with same scenario on 4.11rc1 and got same results,
> > did your 2 PR's currently exists on 4.11 rc1 in which i am testing or it
> > will exist on future rc2? If they exists on 4.11rc1 then we have a
> problem
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com> wrote:
> >
> >> Hi, Ozhan. MACs are not removed upon vm removal, but they are overriden
> >> upon vm creation with same ip (or same hostname). It should work fine,
> >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the
> case
> >> when IP is reused.
> >>
> >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> >> oruzgarkaraman@gmail.com> написал:
> >>
> >> Hi;
> >> Today we noticed that one of our new provisioned instance did not get IP
> >> from VR. When we dig into the issue we find that one different mac is
> >> written in dnsmasq.leases file holds new instances IP address.
> >>
> >> We checked this mac address from db and we noticed that this mac is used
> >> for old expunged instance.
> >>
> >> So from this point we realised that when we destroy an instance its mac
> >> did
> >> not removed from dnsmasq.leases file so if we use this ip for a new
> >> instance then we have a problem, our instance could not get IP from VR.
> >>
> >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM.
> Today
> >> we made a HA test and we crashed the host so VR and SystemVM's are
> >> rebooted
> >> after we boot host back. I do not think this issue is related to VR
> reboot
> >> but i like to give information about our environment.
> >>
> >> We need to manage dnsmasq.leases file when we expunge an instance.
> >>
> >> Thanks
> >> Özhan
> >>
> >> Logs are below:
> >>
> >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:00:00:00:b9
> no
> >> address available
> >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:9f:00:00:ba
> no
> >> address available
> >>
> >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> >>
> >> mysql> select name,state,private_mac_address,private_ip_address from
> >> vm_instance;
> >> +---------------+-----------+---------------------+---------
> -----------+
> >> | name | state | private_mac_address | private_ip_address |
> >> +---------------+-----------+---------------------+---------
> -----------+
> >> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 |
> >> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 |
> >> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 |
> >> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 |
> >> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 |
> >> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 |
> >> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 |
> >> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 |
> >> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 |
> >> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 |
> >> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 |
> >> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 |
> >> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 |
> >> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 |
> >> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 |
> >> +---------------+-----------+---------------------+---------
> -----------+
> >> 15 rows in set (0.00 sec)
> >>
> >
> >
>
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Özhan Rüzgar Karaman <or...@gmail.com>.
Hi Ivan;
I checked 2 PR's and they are exist on 4.11rc1 but issue still exists on my
environment. When a new vm uses IP from old expunged vm then leases file
creates problem. Please check the logs that i submitted on first email,
issue is clear there and in my opinion it still exists on 4.11rc1.
By the way 2393 is about VM's IP Changing progress, maybe it does not cover
my scenario.
Thanks
Özhan
On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
oruzgarkaraman@gmail.com> wrote:
> Hi Ivan;
> I made several tests with same scenario on 4.11rc1 and got same results,
> did your 2 PR's currently exists on 4.11 rc1 in which i am testing or it
> will exist on future rc2? If they exists on 4.11rc1 then we have a problem
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com> wrote:
>
>> Hi, Ozhan. MACs are not removed upon vm removal, but they are overriden
>> upon vm creation with same ip (or same hostname). It should work fine,
>> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the case
>> when IP is reused.
>>
>> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
>> oruzgarkaraman@gmail.com> написал:
>>
>> Hi;
>> Today we noticed that one of our new provisioned instance did not get IP
>> from VR. When we dig into the issue we find that one different mac is
>> written in dnsmasq.leases file holds new instances IP address.
>>
>> We checked this mac address from db and we noticed that this mac is used
>> for old expunged instance.
>>
>> So from this point we realised that when we destroy an instance its mac
>> did
>> not removed from dnsmasq.leases file so if we use this ip for a new
>> instance then we have a problem, our instance could not get IP from VR.
>>
>> We have one host on our lab environment and its Ubuntu 16.04.3 KVM. Today
>> we made a HA test and we crashed the host so VR and SystemVM's are
>> rebooted
>> after we boot host back. I do not think this issue is related to VR reboot
>> but i like to give information about our environment.
>>
>> We need to manage dnsmasq.leases file when we expunge an instance.
>>
>> Thanks
>> Özhan
>>
>> Logs are below:
>>
>> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
>> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
>> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
>> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:00:00:00:b9 no
>> address available
>> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
>> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
>> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:9f:00:00:ba no
>> address available
>>
>> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
>> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
>> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
>> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
>> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
>> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
>> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
>> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
>> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
>> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
>> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
>> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
>>
>> mysql> select name,state,private_mac_address,private_ip_address from
>> vm_instance;
>> +---------------+-----------+---------------------+--------------------+
>> | name | state | private_mac_address | private_ip_address |
>> +---------------+-----------+---------------------+--------------------+
>> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 |
>> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 |
>> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 |
>> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 |
>> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 |
>> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 |
>> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 |
>> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 |
>> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 |
>> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 |
>> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 |
>> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 |
>> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 |
>> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 |
>> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 |
>> +---------------+-----------+---------------------+--------------------+
>> 15 rows in set (0.00 sec)
>>
>
>
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Özhan Rüzgar Karaman <or...@gmail.com>.
Hi Ivan;
I made several tests with same scenario on 4.11rc1 and got same results,
did your 2 PR's currently exists on 4.11 rc1 in which i am testing or it
will exist on future rc2? If they exists on 4.11rc1 then we have a problem
Thanks
Özhan
On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com
> wrote:
> Hi, Ozhan. MACs are not removed upon vm removal, but they are overriden
> upon vm creation with same ip (or same hostname). It should work fine,
> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the case
> when IP is reused.
>
> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
> oruzgarkaraman@gmail.com> написал:
>
> Hi;
> Today we noticed that one of our new provisioned instance did not get IP
> from VR. When we dig into the issue we find that one different mac is
> written in dnsmasq.leases file holds new instances IP address.
>
> We checked this mac address from db and we noticed that this mac is used
> for old expunged instance.
>
> So from this point we realised that when we destroy an instance its mac did
> not removed from dnsmasq.leases file so if we use this ip for a new
> instance then we have a problem, our instance could not get IP from VR.
>
> We have one host on our lab environment and its Ubuntu 16.04.3 KVM. Today
> we made a HA test and we crashed the host so VR and SystemVM's are rebooted
> after we boot host back. I do not think this issue is related to VR reboot
> but i like to give information about our environment.
>
> We need to manage dnsmasq.leases file when we expunge an instance.
>
> Thanks
> Özhan
>
> Logs are below:
>
> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:00:00:00:b9 no
> address available
> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:9f:00:00:ba no
> address available
>
> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
>
> mysql> select name,state,private_mac_address,private_ip_address from
> vm_instance;
> +---------------+-----------+---------------------+--------------------+
> | name | state | private_mac_address | private_ip_address |
> +---------------+-----------+---------------------+--------------------+
> | s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 |
> | v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 |
> | Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 |
> | r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 |
> | Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 |
> | Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 |
> | deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 |
> | snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 |
> | TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 |
> | TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 |
> | test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 |
> | test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 |
> | sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 |
> | sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 |
> | sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 |
> +---------------+-----------+---------------------+--------------------+
> 15 rows in set (0.00 sec)
>
Re: [4.11] VR Problem on Releasing Expunged Instance IP from
dnsmasq.leases file
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Hi, Ozhan. MACs are not removed upon vm removal, but they are overriden
upon vm creation with same ip (or same hostname). It should work fine,
4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the case
when IP is reused.
22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar Karaman" <
oruzgarkaraman@gmail.com> написал:
Hi;
Today we noticed that one of our new provisioned instance did not get IP
from VR. When we dig into the issue we find that one different mac is
written in dnsmasq.leases file holds new instances IP address.
We checked this mac address from db and we noticed that this mac is used
for old expunged instance.
So from this point we realised that when we destroy an instance its mac did
not removed from dnsmasq.leases file so if we use this ip for a new
instance then we have a problem, our instance could not get IP from VR.
We have one host on our lab environment and its Ubuntu 16.04.3 KVM. Today
we made a HA test and we crashed the host so VR and SystemVM's are rebooted
after we boot host back. I do not think this issue is related to VR reboot
but i like to give information about our environment.
We need to manage dnsmasq.leases file when we expunge an instance.
Thanks
Özhan
Logs are below:
root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
192.168.18.186 because it is leased to 1e:00:25:00:00:b9
Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:00:00:00:b9 no
address available
Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
192.168.18.187 because it is leased to 1e:00:80:00:00:ba
Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0) 1e:00:9f:00:00:ba no
address available
root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
mysql> select name,state,private_mac_address,private_ip_address from
vm_instance;
+---------------+-----------+---------------------+--------------------+
| name | state | private_mac_address | private_ip_address |
+---------------+-----------+---------------------+--------------------+
| s-1-VM | Running | 1e:00:34:00:01:00 | 172.16.50.143 |
| v-2-VM | Running | 1e:00:81:00:01:03 | 172.16.50.146 |
| Tolga | Expunging | 1e:00:50:00:00:bc | 192.168.18.189 |
| r-4-VM | Running | 0e:00:a9:fe:03:0e | 169.254.3.14 |
| Tolga02 | Expunging | 1e:00:b4:00:00:bf | 192.168.18.192 |
| Tolga03 | Expunging | 1e:00:99:00:00:bb | 192.168.18.188 |
| deneme | Expunging | 1e:00:80:00:00:ba | 192.168.18.187 |
| snpvmtolga02 | Expunging | 1e:00:69:00:00:b9 | 192.168.18.186 |
| TolgaTest | Stopped | 1e:00:90:00:00:bc | 192.168.18.189 |
| TolgaTest02 | Stopped | 1e:00:96:00:00:bf | 192.168.18.192 |
| test411rc1 | Running | 1e:00:40:00:00:bb | 192.168.18.188 |
| test411rc1mac | Running | 1e:00:9f:00:00:ba | 192.168.18.187 |
| sil1 | Expunging | 1e:00:25:00:00:b9 | 192.168.18.186 |
| sil2sameip | Expunging | 1e:00:14:00:00:b9 | 192.168.18.186 |
| sil3sameip | Running | 1e:00:00:00:00:b9 | 192.168.18.186 |
+---------------+-----------+---------------------+--------------------+
15 rows in set (0.00 sec)