You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Joris van Lieshout <jv...@schubergphilis.com> on 2014/05/21 17:30:49 UTC
Review Request 21773: sysctl improvements regarding nf_conntrack_max,
panic and nonlocal_bind
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/
-----------------------------------------------------------
Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
Repository: cloudstack-git
Description
-------
net.ipv4.ip_nonlocal_bind = 1
For smooth transition on redundant routers in case of a failover
kernel.panic = 10
kernel.panic_on_oops = 1
vm.panic_on_oom = 1
This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
net.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_max = 1000000
also up the limit of nf_conntrack for more performance
Diffs
-----
systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
Diff: https://reviews.apache.org/r/21773/diff/
Testing
-------
we are running these settings in our beta and prod env for months allready, with good results.
Thanks,
Joris van Lieshout
Re: Review Request 21773: sysctl improvements regarding nf_conntrack_max,
panic and nonlocal_bind
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
> On May 27, 2014, 4:59 p.m., Chiradeep Vittal wrote:
> > I like the idea of taking action on OOM in the VR/systemvm. However there is a missing (very important) piece: the VR stays unconfigured. So, it should actually stop, not reboot in order for the management server to take notice and perform HA.
>
> Joris van Lieshout wrote:
> As far as I know Debian does not have a way to halt after a panic. I agree that a reboot of the SVM is not the best solution but it is still better then having the OOM killer randomly kill processes ending up partially functional. What's you though on this: have cloud-early-config detect a non-ACS initiated reboot, f.i. a panic, and stop the SVM?
how about adding something like this to cloud-early-config?
local DIDICRASH=$(last -x -4 | grep crash)
[ -n "$DIDICRASH" ] && log_it "Last boot due to crash, shutting down." && shutdown -h now
- Joris
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/#review43985
-----------------------------------------------------------
On May 26, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21773/
> -----------------------------------------------------------
>
> (Updated May 26, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> net.ipv4.ip_nonlocal_bind = 1
> For smooth transition on redundant routers in case of a failover
>
> kernel.panic = 10
> kernel.panic_on_oops = 1
> vm.panic_on_oom = 1
> This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
>
> net.nf_conntrack_max = 1000000
> net.netfilter.nf_conntrack_max = 1000000
> also up the limit of nf_conntrack for more performance
>
>
> Diffs
> -----
>
> systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
>
> Diff: https://reviews.apache.org/r/21773/diff/
>
>
> Testing
> -------
>
> we are running these settings in our beta and prod env for months allready, with good results.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21773: sysctl improvements regarding
nf_conntrack_max, panic and nonlocal_bind
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
> On May 27, 2014, 4:59 p.m., Chiradeep Vittal wrote:
> > I like the idea of taking action on OOM in the VR/systemvm. However there is a missing (very important) piece: the VR stays unconfigured. So, it should actually stop, not reboot in order for the management server to take notice and perform HA.
>
> Joris van Lieshout wrote:
> As far as I know Debian does not have a way to halt after a panic. I agree that a reboot of the SVM is not the best solution but it is still better then having the OOM killer randomly kill processes ending up partially functional. What's you though on this: have cloud-early-config detect a non-ACS initiated reboot, f.i. a panic, and stop the SVM?
>
> Joris van Lieshout wrote:
> how about adding something like this to cloud-early-config?
>
> local DIDICRASH=$(last -x -4 | grep crash)
> [ -n "$DIDICRASH" ] && log_it "Last boot due to crash, shutting down." && shutdown -h now
I just had a chat with the guys refacturing the rrvm code and what I understood is that the new code will be able to deal with a rvm that has been rebooted preventing it from getting in an unconfigured state.
- Joris
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/#review43985
-----------------------------------------------------------
On May 26, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21773/
> -----------------------------------------------------------
>
> (Updated May 26, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> net.ipv4.ip_nonlocal_bind = 1
> For smooth transition on redundant routers in case of a failover
>
> kernel.panic = 10
> kernel.panic_on_oops = 1
> vm.panic_on_oom = 1
> This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
>
> net.nf_conntrack_max = 1000000
> net.netfilter.nf_conntrack_max = 1000000
> also up the limit of nf_conntrack for more performance
>
>
> Diffs
> -----
>
> systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
>
> Diff: https://reviews.apache.org/r/21773/diff/
>
>
> Testing
> -------
>
> we are running these settings in our beta and prod env for months allready, with good results.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21773: sysctl improvements regarding nf_conntrack_max,
panic and nonlocal_bind
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
> On May 27, 2014, 4:59 p.m., Chiradeep Vittal wrote:
> > I like the idea of taking action on OOM in the VR/systemvm. However there is a missing (very important) piece: the VR stays unconfigured. So, it should actually stop, not reboot in order for the management server to take notice and perform HA.
As far as I know Debian does not have a way to halt after a panic. I agree that a reboot of the SVM is not the best solution but it is still better then having the OOM killer randomly kill processes ending up partially functional. What's you though on this: have cloud-early-config detect a non-ACS initiated reboot, f.i. a panic, and stop the SVM?
- Joris
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/#review43985
-----------------------------------------------------------
On May 26, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21773/
> -----------------------------------------------------------
>
> (Updated May 26, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> net.ipv4.ip_nonlocal_bind = 1
> For smooth transition on redundant routers in case of a failover
>
> kernel.panic = 10
> kernel.panic_on_oops = 1
> vm.panic_on_oom = 1
> This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
>
> net.nf_conntrack_max = 1000000
> net.netfilter.nf_conntrack_max = 1000000
> also up the limit of nf_conntrack for more performance
>
>
> Diffs
> -----
>
> systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
>
> Diff: https://reviews.apache.org/r/21773/diff/
>
>
> Testing
> -------
>
> we are running these settings in our beta and prod env for months allready, with good results.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21773: sysctl improvements regarding nf_conntrack_max,
panic and nonlocal_bind
Posted by Chiradeep Vittal <ch...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/#review43985
-----------------------------------------------------------
I like the idea of taking action on OOM in the VR/systemvm. However there is a missing (very important) piece: the VR stays unconfigured. So, it should actually stop, not reboot in order for the management server to take notice and perform HA.
- Chiradeep Vittal
On May 26, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21773/
> -----------------------------------------------------------
>
> (Updated May 26, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> net.ipv4.ip_nonlocal_bind = 1
> For smooth transition on redundant routers in case of a failover
>
> kernel.panic = 10
> kernel.panic_on_oops = 1
> vm.panic_on_oom = 1
> This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
>
> net.nf_conntrack_max = 1000000
> net.netfilter.nf_conntrack_max = 1000000
> also up the limit of nf_conntrack for more performance
>
>
> Diffs
> -----
>
> systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
>
> Diff: https://reviews.apache.org/r/21773/diff/
>
>
> Testing
> -------
>
> we are running these settings in our beta and prod env for months allready, with good results.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21773: sysctl improvements regarding
nf_conntrack_max, panic and nonlocal_bind
Posted by daan Hoogland <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/#review50576
-----------------------------------------------------------
45deade1df171d9ae5562111bf011ee913ff9dc0 on 4.3
c4d1bf7f2426bfddd96ab9415fe793647e67a5a4 on 4.4
c90bf1231f2baabff329f19bc195d1b626f9ce94 on master
- daan Hoogland
On May 26, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21773/
> -----------------------------------------------------------
>
> (Updated May 26, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> net.ipv4.ip_nonlocal_bind = 1
> For smooth transition on redundant routers in case of a failover
>
> kernel.panic = 10
> kernel.panic_on_oops = 1
> vm.panic_on_oom = 1
> This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
>
> net.nf_conntrack_max = 1000000
> net.netfilter.nf_conntrack_max = 1000000
> also up the limit of nf_conntrack for more performance
>
>
> Diffs
> -----
>
> systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
>
> Diff: https://reviews.apache.org/r/21773/diff/
>
>
> Testing
> -------
>
> we are running these settings in our beta and prod env for months allready, with good results.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21773: sysctl improvements regarding nf_conntrack_max,
panic and nonlocal_bind
Posted by Jayapal Reddy <ja...@citrix.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/#review46616
-----------------------------------------------------------
But in debian wheezy sysctl.conf is not got loaded on boot time.
There is bug https://bugs.launchpad.net/ubuntu/+source/procps/+bug/50093
- Jayapal Reddy
On May 26, 2014, 2:34 p.m., Joris van Lieshout wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21773/
> -----------------------------------------------------------
>
> (Updated May 26, 2014, 2:34 p.m.)
>
>
> Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
>
>
> Repository: cloudstack-git
>
>
> Description
> -------
>
> net.ipv4.ip_nonlocal_bind = 1
> For smooth transition on redundant routers in case of a failover
>
> kernel.panic = 10
> kernel.panic_on_oops = 1
> vm.panic_on_oom = 1
> This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
>
> net.nf_conntrack_max = 1000000
> net.netfilter.nf_conntrack_max = 1000000
> also up the limit of nf_conntrack for more performance
>
>
> Diffs
> -----
>
> systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
>
> Diff: https://reviews.apache.org/r/21773/diff/
>
>
> Testing
> -------
>
> we are running these settings in our beta and prod env for months allready, with good results.
>
>
> Thanks,
>
> Joris van Lieshout
>
>
Re: Review Request 21773: sysctl improvements regarding nf_conntrack_max,
panic and nonlocal_bind
Posted by Joris van Lieshout <jv...@schubergphilis.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21773/
-----------------------------------------------------------
(Updated May 26, 2014, 2:34 p.m.)
Review request for cloudstack, Chiradeep Vittal, daan Hoogland, edison su, Hugo Trippaers, and Sheng Yang.
Changes
-------
sugesting branches to add it to
Repository: cloudstack-git
Description
-------
net.ipv4.ip_nonlocal_bind = 1
For smooth transition on redundant routers in case of a failover
kernel.panic = 10
kernel.panic_on_oops = 1
vm.panic_on_oom = 1
This way, if the instance oops/ooms, it dies in a way that ACS sees it and can act on it
net.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_max = 1000000
also up the limit of nf_conntrack for more performance
Diffs
-----
systemvm/patches/debian/config/etc/sysctl.conf 586d5bd
Diff: https://reviews.apache.org/r/21773/diff/
Testing
-------
we are running these settings in our beta and prod env for months allready, with good results.
Thanks,
Joris van Lieshout