You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Laurent Steff <La...@inria.fr> on 2014/03/18 15:22:08 UTC
Virtual Router VM issues
Hi,
As we are since one full week in big issues with our cloudstack implementation (unable to create vm, start a
vm, ..., see "Unable to create a deployment for vm" from Pierre Benard on this list),
we are investigating our VirtualRouter Vm and it appears that :
- it appears as a "great green" on cloudstack web interface (running)
- there's full on it's /var (due to below)
- it's /var/log/kern.log is filled with thoses messages :
---
Mar 18 14:11:57 r-3452-VM kernel: [ 78.384476] nf_conntrack: table full, dropping packet.
Mar 18 14:11:57 r-3452-VM kernel: [ 78.422114] nf_conntrack: table full, dropping packet.
Mar 18 14:11:57 r-3452-VM kernel: [ 78.453980] nf_conntrack: table full, dropping packet.
Mar 18 14:11:58 r-3452-VM kernel: [ 79.506306] nf_conntrack: table full, dropping packet.
Mar 18 14:11:58 r-3452-VM kernel: [ 79.509123] nf_conntrack: table full, dropping packet.
Mar 18 14:11:58 r-3452-VM kernel: [ 79.522540] nf_conntrack: table full, dropping packet.
---
What could we do (and how ?)
- Increase RAM of the VirtualRouter VM ?
- Scale out on the services provided by the VirtualRouter VM ?
Thanks in advance,
--
Laurent Steff
DSI/SESI
INRIA
Tél. : +33 1 39 63 50 81
Port. : +33 6 87 66 77 85
http://www.inria.fr/
Re: Virtual Router VM issues
Posted by Shanker Balan <sh...@shapeblue.com>.
Comments inline.
On 19-Mar-2014, at 3:38 pm, Laurent Steff <La...@inria.fr> wrote:
> Hi,
>
> Thanks for your response, answers below.
>
> ----- Mail original -----
>>
>> From what I know and have read on the Internet, this usually
>> happens when your Linux NAT box hits the IPtables connection
>> tracking limit. This limit is set via /proc.
>>
>> A quick search on the web reveals many sites that tell
>> you how to increase the conntrack limits.
>
> Yes, but every tunning I'll do will be lost when VirtualRouter VM
> will be re-instanciated.
You can stick in your tunable into rc.local. You will need
to re-add it in case the VR is recreated.
> As all monitoring we could manually had on this VM.
> I've heard there's are request for enhancement concerning VirtualRouter VM
> real-life monitoring (and not juste, it's ok, I'm running :)).
> I will try to +1 them.
Yes, I think 64bit VRs will give a larger address space. Not sure if
the conntrack limits would be enhanced in the process as well.
>
>> However, what would be interesting to know why you hit the limit
>> in the first place. Any signs of network abuse? Lots of connections
>> in TIME_WAIT state?
>
> Yes, and the only way to find it was to tcpdump on VirtualRouter VM.
> 2 hosts were making accidentaly to much DNS requests (hundred by seconds).
Are the VMs directly queries any external DNS server? If so, reconfigure
them to use the internal DNS server and block all external DNS traffic
using the egress firewall settings.
> As soon as the "DoS" on the VirtualRouter VM was stopped, we could again
> create/start virtual machines on cloudstack.
I think you should be able to clean up the network by filtering egress traffic.
Hth.
--
@shankerbalan
M: +91 98860 60539 | O: +91 (80) 67935867
shanker.balan@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the best 24/7 SLA for CloudStack Environments.
Apache CloudStack Bootcamp training courses
**NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
18th-19th February 2014, Brazil. Classroom<http://shapeblue.com/cloudstack-training/>
17th-23rd March 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
24th-28th March 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
16th-20th June 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
23rd-27th June 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.
Re: Virtual Router VM issues
Posted by Laurent Steff <La...@inria.fr>.
Hi,
Thanks for your response, answers below.
----- Mail original -----
>
> From what I know and have read on the Internet, this usually
> happens when your Linux NAT box hits the IPtables connection
> tracking limit. This limit is set via /proc.
>
> A quick search on the web reveals many sites that tell
> you how to increase the conntrack limits.
Yes, but every tunning I'll do will be lost when VirtualRouter VM
will be re-instanciated.
As all monitoring we could manually had on this VM.
I've heard there's are request for enhancement concerning VirtualRouter VM
real-life monitoring (and not juste, it's ok, I'm running :)).
I will try to +1 them.
>
> However, what would be interesting to know why you hit the limit
> in the first place. Any signs of network abuse? Lots of connections
> in TIME_WAIT state?
Yes, and the only way to find it was to tcpdump on VirtualRouter VM.
2 hosts were making accidentaly to much DNS requests (hundred by seconds).
As soon as the "DoS" on the VirtualRouter VM was stopped, we could again
create/start virtual machines on cloudstack.
Regards,
--
Laurent Steff
DSI/SESI
INRIA
Tél. : +33 1 39 63 50 81
Port. : +33 6 87 66 77 85
http://www.inria.fr/
Re: Virtual Router VM issues
Posted by Shanker Balan <sh...@shapeblue.com>.
Comments inline.
On 18-Mar-2014, at 7:52 pm, Laurent Steff <La...@inria.fr> wrote:
> Hi,
>
> As we are since one full week in big issues with our cloudstack implementation (unable to create vm, start a
> vm, ..., see "Unable to create a deployment for vm" from Pierre Benard on this list),
> we are investigating our VirtualRouter Vm and it appears that :
>
> - it appears as a "great green" on cloudstack web interface (running)
> - there's full on it's /var (due to below)
> - it's /var/log/kern.log is filled with thoses messages :
>
> ---
> Mar 18 14:11:57 r-3452-VM kernel: [ 78.384476] nf_conntrack: table full, dropping packet.
> Mar 18 14:11:57 r-3452-VM kernel: [ 78.422114] nf_conntrack: table full, dropping packet.
> Mar 18 14:11:57 r-3452-VM kernel: [ 78.453980] nf_conntrack: table full, dropping packet.
> Mar 18 14:11:58 r-3452-VM kernel: [ 79.506306] nf_conntrack: table full, dropping packet.
> Mar 18 14:11:58 r-3452-VM kernel: [ 79.509123] nf_conntrack: table full, dropping packet.
> Mar 18 14:11:58 r-3452-VM kernel: [ 79.522540] nf_conntrack: table full, dropping packet.
> ---
>
> What could we do (and how ?)
>
> - Increase RAM of the VirtualRouter VM ?
> - Scale out on the services provided by the VirtualRouter VM ?
>From what I know and have read on the Internet, this usually
happens when your Linux NAT box hits the IPtables connection
tracking limit. This limit is set via /proc.
A quick search on the web reveals many sites that tell
you how to increase the conntrack limits.
However, what would be interesting to know why you hit the limit
in the first place. Any signs of network abuse? Lots of connections
in TIME_WAIT state?
Hth.
--
@shankerbalan
M: +91 98860 60539 | O: +91 (80) 67935867
shanker.balan@shapeblue.com | www.shapeblue.com | Twitter:@shapeblue
ShapeBlue Services India LLP, 22nd floor, Unit 2201A, World Trade Centre, Bangalore - 560 055
Need Enterprise Grade Support for Apache CloudStack?
Our CloudStack Infrastructure Support<http://shapeblue.com/cloudstack-infrastructure-support/> offers the best 24/7 SLA for CloudStack Environments.
Apache CloudStack Bootcamp training courses
**NEW!** CloudStack 4.2.1 training<http://shapeblue.com/cloudstack-training/>
18th-19th February 2014, Brazil. Classroom<http://shapeblue.com/cloudstack-training/>
17th-23rd March 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
24th-28th March 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
16th-20th June 2014, Region A. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
23rd-27th June 2014, Region B. Instructor led, On-line<http://shapeblue.com/cloudstack-training/>
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Shape Blue Ltd or related companies. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error. Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue Services India LLP is a company incorporated in India and is operated under license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is operated under license from Shape Blue Ltd. ShapeBlue is a registered trademark.