You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Rene Moser <ma...@renemoser.net> on 2015/03/17 17:19:26 UTC
irqbalancer on VR not working
Hi
We experienced packet drops and high CPU on one core (only) on a
multicore VR on high volume traffic.
After extensive investigation, we identified the irq balancing is not
working properly, as a result only one core was used for IRQ handing of
TCP for all 3 interfaces:
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
72: 2704675402 0 0 0 PCI-MSI-edge eth0-rxtx-0
73: 1272554833 0 0 0 PCI-MSI-edge eth0-rxtx-1
74: 1277628455 0 0 0 PCI-MSI-edge eth0-rxtx-2
75: 1278457419 0 0 0 PCI-MSI-edge eth0-rxtx-3
76: 0 0 0 0 PCI-MSI-edge eth0-event-4
77: 450819 0 0 0 PCI-MSI-edge eth1-rxtx-0
78: 370442 0 0 0 PCI-MSI-edge eth1-rxtx-1
79: 317069 0 0 0 PCI-MSI-edge eth1-rxtx-2
80: 401197 0 0 0 PCI-MSI-edge eth1-rxtx-3
81: 0 0 0 0 PCI-MSI-edge eth1-event-4
82: 2768559207 0 0 0 PCI-MSI-edge eth2-rxtx-0
83: 1328122336 0 0 0 PCI-MSI-edge eth2-rxtx-1
84: 1328017908 0 0 0 PCI-MSI-edge eth2-rxtx-2
85: 1322146180 0 0 0 PCI-MSI-edge
Restarting /etc/init.d/ircbalance did not help, but you will find some
log entries in /var/log/daemon.log
irqbalance: WARNING: MSI interrupts found in /proc/interrupts
irqbalance: But none found in sysfs, you need to update your kernel
irqbalance: Until then, IRQs will be improperly classified
It turns out, this is a known issue on Debian Wheezy irqbalance version
1.0.3-3 https://lists.debian.org/debian-user/2013/05/msg00512.html and
was patched in 1.0.4-1.
But the good know is, there is 1.0.6 in wheezy-backports
aptitude update
aptitude install -t wheezy-backports irqbalance
That helped.
Can this be included in the SystemVM templates as well?
Yours
René
Re: irqbalancer on VR not working
Posted by Ahmad Emneina <ae...@gmail.com>.
This would be a great bug to file... that way developers can get this
addressed. File here:
https://issues.apache.org/
On Tue, Mar 17, 2015 at 9:19 AM, Rene Moser <ma...@renemoser.net> wrote:
> Hi
>
> We experienced packet drops and high CPU on one core (only) on a
> multicore VR on high volume traffic.
>
> After extensive investigation, we identified the irq balancing is not
> working properly, as a result only one core was used for IRQ handing of
> TCP for all 3 interfaces:
>
> # cat /proc/interrupts
> CPU0 CPU1 CPU2 CPU3
> 72: 2704675402 0 0 0 PCI-MSI-edge eth0-rxtx-0
> 73: 1272554833 0 0 0 PCI-MSI-edge eth0-rxtx-1
> 74: 1277628455 0 0 0 PCI-MSI-edge eth0-rxtx-2
> 75: 1278457419 0 0 0 PCI-MSI-edge eth0-rxtx-3
> 76: 0 0 0 0 PCI-MSI-edge eth0-event-4
> 77: 450819 0 0 0 PCI-MSI-edge eth1-rxtx-0
> 78: 370442 0 0 0 PCI-MSI-edge eth1-rxtx-1
> 79: 317069 0 0 0 PCI-MSI-edge eth1-rxtx-2
> 80: 401197 0 0 0 PCI-MSI-edge eth1-rxtx-3
> 81: 0 0 0 0 PCI-MSI-edge eth1-event-4
> 82: 2768559207 0 0 0 PCI-MSI-edge eth2-rxtx-0
> 83: 1328122336 0 0 0 PCI-MSI-edge eth2-rxtx-1
> 84: 1328017908 0 0 0 PCI-MSI-edge eth2-rxtx-2
> 85: 1322146180 0 0 0 PCI-MSI-edge
>
> Restarting /etc/init.d/ircbalance did not help, but you will find some
> log entries in /var/log/daemon.log
>
> irqbalance: WARNING: MSI interrupts found in /proc/interrupts
> irqbalance: But none found in sysfs, you need to update your kernel
> irqbalance: Until then, IRQs will be improperly classified
>
> It turns out, this is a known issue on Debian Wheezy irqbalance version
> 1.0.3-3 https://lists.debian.org/debian-user/2013/05/msg00512.html and
> was patched in 1.0.4-1.
>
> But the good know is, there is 1.0.6 in wheezy-backports
>
> aptitude update
> aptitude install -t wheezy-backports irqbalance
>
> That helped.
>
> Can this be included in the SystemVM templates as well?
>
> Yours
> René
>
>
>
Re: irqbalancer on VR not working
Posted by Rene Moser <ma...@renemoser.net>.
Hi
On 18.03.2015 05:34, Jayapal Reddy Uradi wrote:
> On 64 bit templates 1.0.6 not worked.
After a few days, those VRs on which I experianced updating to 1.0.6 had
no affect and which are VRs having lots of traffic, also started
balancing the interrupts. Yey!
IMHO updating to 1.0.6 worked! Commit
https://github.com/apache/cloudstack/commit/6a881b93373f26af2f0786435022771faedd0d10
now installs 1.0.6 from backports and the fix is also backported into 4.5.
Yours
René
Re: irqbalancer on VR not working
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Rene,
On 64 bit templates 1.0.6 not worked.
Please see the below irqbalance logs on VR for 1.0.7
http://pastebin.com/zm6Tbep8
Thanks,
Jayapal
On 18-Mar-2015, at 12:34 AM, Rene Moser <ma...@renemoser.net> wrote:
> Hi
>
> On 03/17/2015 05:35 PM, Jayapal Reddy Uradi wrote:
>> Did you test irqbalance 1.0.6 ? It won't work.
>> irqbalance 1.0.7 works.
>
> On most VRs it worked (somehow on a few VRs it doesn't). But this is a yes I guess.
>
> I also checked the Debian changelog https://sources.debian.net/src/irqbalance/1.0.6-3/debian/changelog/ and IMHO the relevant fix is "Improve rescan ability to catch a timing race early in irqbalance setup" and/or "Added ability to force irqbalance to rescan irqs on reception of a SIGHUP." in 1.0.4-1
>
> Did you have other experiences?
>
> Yours
> René
>
>
>
Re: irqbalancer on VR not working
Posted by Rene Moser <ma...@renemoser.net>.
Hi
On 03/17/2015 05:35 PM, Jayapal Reddy Uradi wrote:
> Did you test irqbalance 1.0.6 ? It won't work.
> irqbalance 1.0.7 works.
On most VRs it worked (somehow on a few VRs it doesn't). But this is a
yes I guess.
I also checked the Debian changelog
https://sources.debian.net/src/irqbalance/1.0.6-3/debian/changelog/ and
IMHO the relevant fix is "Improve rescan ability to catch a timing race
early in irqbalance setup" and/or "Added ability to force irqbalance to
rescan irqs on reception of a SIGHUP." in 1.0.4-1
Did you have other experiences?
Yours
René
Re: irqbalancer on VR not working
Posted by Jayapal Reddy Uradi <ja...@citrix.com>.
Hi Rene,
Did you test irqbalance 1.0.6 ? It won't work.
irqbalance 1.0.7 works.
Compile 1.0.7 and get irqbalance binary.
Install 1.0.6 from repo and replace binary with 1.0.7 binary.
Thanks,
Jayapal
On 17-Mar-2015, at 9:49 PM, Rene Moser <ma...@renemoser.net> wrote:
> Hi
>
> We experienced packet drops and high CPU on one core (only) on a
> multicore VR on high volume traffic.
>
> After extensive investigation, we identified the irq balancing is not
> working properly, as a result only one core was used for IRQ handing of
> TCP for all 3 interfaces:
>
> # cat /proc/interrupts
> CPU0 CPU1 CPU2 CPU3
> 72: 2704675402 0 0 0 PCI-MSI-edge eth0-rxtx-0
> 73: 1272554833 0 0 0 PCI-MSI-edge eth0-rxtx-1
> 74: 1277628455 0 0 0 PCI-MSI-edge eth0-rxtx-2
> 75: 1278457419 0 0 0 PCI-MSI-edge eth0-rxtx-3
> 76: 0 0 0 0 PCI-MSI-edge eth0-event-4
> 77: 450819 0 0 0 PCI-MSI-edge eth1-rxtx-0
> 78: 370442 0 0 0 PCI-MSI-edge eth1-rxtx-1
> 79: 317069 0 0 0 PCI-MSI-edge eth1-rxtx-2
> 80: 401197 0 0 0 PCI-MSI-edge eth1-rxtx-3
> 81: 0 0 0 0 PCI-MSI-edge eth1-event-4
> 82: 2768559207 0 0 0 PCI-MSI-edge eth2-rxtx-0
> 83: 1328122336 0 0 0 PCI-MSI-edge eth2-rxtx-1
> 84: 1328017908 0 0 0 PCI-MSI-edge eth2-rxtx-2
> 85: 1322146180 0 0 0 PCI-MSI-edge
>
> Restarting /etc/init.d/ircbalance did not help, but you will find some
> log entries in /var/log/daemon.log
>
> irqbalance: WARNING: MSI interrupts found in /proc/interrupts
> irqbalance: But none found in sysfs, you need to update your kernel
> irqbalance: Until then, IRQs will be improperly classified
>
> It turns out, this is a known issue on Debian Wheezy irqbalance version
> 1.0.3-3 https://lists.debian.org/debian-user/2013/05/msg00512.html and
> was patched in 1.0.4-1.
>
> But the good know is, there is 1.0.6 in wheezy-backports
>
> aptitude update
> aptitude install -t wheezy-backports irqbalance
>
> That helped.
>
> Can this be included in the SystemVM templates as well?
>
> Yours
> René
>
>