You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Simon Weller <sw...@ena.com.INVALID> on 2022/07/01 15:54:44 UTC
Re: Asymmetric traffic issues?
Unfortunately, I can't open the xml via the mailing list, but based on the template you're using, it's not setup for Virtio-blk or virtio-scsi. Change the template to Virtio-SCSI 64bit and the correct interface drive type will be specified in the XML.
________________________________
From: S.Fuller <st...@gmail.com>
Sent: Thursday, June 30, 2022 12:43 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Asymmetric traffic issues?
I'm using the Windows Server 2016 profile for the template. After the VM came up, I installed the virtio drivers from the latest ISO, and then added a new NIC so that it used the virtio network drivers vs the default E1000 drivers.
XML dump is attached
On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sw...@ena.com.invalid> wrote:
Can you provide a redacted dump of your libvirt xml file for the VM?
Also, which OS profile are you using in Cloudstack for this? The Windows specific ones don't use virtio if I remember correctly, so you will need to select Virtio-SCSI 64bit in order for the xml to be built correctly.
-Si
________________________________
From: S.Fuller <st...@gmail.com>>
Sent: Thursday, June 30, 2022 11:38 AM
To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>>
Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message.
Neither the block nor the scsi drivers appear to be running as far as I can
tell.
- Steve
On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sw...@ena.com.invalid>
wrote:
> Steve,
>
> Are you running the virtio-block or virtio-scsi drivers?
>
> -Si
>
>
> ________________________________
> From: S.Fuller <st...@gmail.com>>
> Sent: Thursday, June 30, 2022 8:52 AM
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <us...@cloudstack.apache.org>>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Well, I checked everything that I could for any QOS settings and there is
> nothing configured there. What is curious is that Idon't see this behavior
> when the transmitting host is running Linux, or if I'm using the E1000
> drivers (although with the E1000 driver, the overall throughput is lower).
> It really feels like I'mrunning into some weird issue with the virtio
> drivers on Windows. My Windows hosts are (to my knowledge) using the latest
> version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
>
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
> On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <st...@gmail.com>> wrote:
>
> > Vivek,
> >
> > Thanks for the reply. I am using the KVM hypervisor. I'll will review the
> > QoS on the hypervisor for both of the nodes.
> >
> > I'm checking throughput between two different VMs that running on two
> > different hosts within the same cluster. As of right now, I'm receiving
> > similar results using both iperf3 and nuttcp as the testing tools. We are
> > only seeing this issue when the VM is not on the same host as the vrouter
> > for its isolated network.
> >
> > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > <vi...@indiqus.com>.invalid> wrote:
> >
> >> Hey Fuller,
> >>
> >> What hypervisor are you using ? I know you have checked all bandwidth
> >> limit on templates and global settings, but it’s worth to check the QoS
> on
> >> the hypervisor level, because at the end it’s the hypervisor which
> manages
> >> all. And from where are you trying to check the network throughout,
> >> between client and server ?
> >>
> >>
> >> Vivek Kumar
> >> Sr. Manager - Cloud & DevOps
> >> TechOps | Indiqus Technologies
> >>
> >> + 91 7503460090 <tel:++91+7503460090>
> >> vivek.kumar@indiqus.com<ma...@indiqus.com> <ma...@indiqus.com>>
> >> www.indiqus.com<http://www.indiqus.com><http://www.indiqus.com> <
> https://www.indiqus.com/>
> >>
> >>
> >>
> >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <st...@gmail.com>> wrote:
> >> >
> >> > Environment:
> >> >
> >> > Two physical hosts
> >> > - Cloudstack 4.11.3
> >> > - Verified that there are no bandwidth limits in place on any of the
> >> > templates or in global settings.
> >> >
> >> > Two isolated networks ("Client" and "Server")
> >> > - Each has a vrouter with a public and private address
> >> > - One Windows 2016 VM on each network (running the latest virtio
> >> drivers)
> >> > - each node running latest version of Iperf3 to test throughput
> >> >
> >> > Testing/Observation:
> >> >
> >> > If the Client VM and the vrouter for the isolated Client network are
> on
> >> the
> >> > same physical host, we see symmetrical throughput in the 2 Gbps range,
> >> > whether we run iperf in regular mode or in reverse mode (iperf -R).
> >> >
> >> > If the Client VM and the vrouter for the isolated Client network are
> on
> >> > different physical hosts, we are seeing 25% of the throughput running
> >> iperf
> >> > in regular mode vs running it in reverse mode.
> >> >
> >> > Has anyone encountered this issue before? If we change the Client VM
> to
> >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we see
> >> > symmetrical throughput in our tests, no matter where the vrouter is in
> >> > relation to the Client VM.
> >> >
> >> > --
> >> > Steve Fuller
> >> > stevefuller@gmail.com<ma...@gmail.com>
> >>
> >>
> >> --
> >> This message is intended only for the use of the individual or entity to
> >> which it is addressed and may contain confidential and/or privileged
> >> information. If you are not the intended recipient, please delete the
> >> original message and any copy of it from your computer system. You are
> >> hereby notified that any dissemination, distribution or copying of this
> >> communication is strictly prohibited unless proper authorization has
> been
> >> obtained for such action. If you have received this communication in
> >> error,
> >> please notify the sender immediately. Although IndiQus attempts to sweep
> >> e-mail and attachments for viruses, it does not guarantee that both are
> >> virus-free and accepts no liability for any damage sustained as a result
> >> of
> >> viruses.
> >>
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
--
Steve Fuller
stevefuller@gmail.com<ma...@gmail.com>
--
Steve Fuller
stevefuller@gmail.com<ma...@gmail.com>
Re: Asymmetric traffic issues?
Posted by Simon Weller <sw...@ena.com.INVALID>.
Good to hear. Thanks for providing a follow up Steve.
-Si
________________________________
From: S.Fuller <st...@gmail.com>
Sent: Friday, July 15, 2022 1:08 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message.
Update on this, we were able to enable the same performance changes by
using ethtool to disable gro on the private and public interfaces on the
virtual routers.
Steve
On Fri, Jul 8, 2022 at 2:14 PM Simon Weller <sw...@ena.com.invalid> wrote:
> That's great news! Glad it all worked out.
>
> -Si
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Friday, July 8, 2022 12:49 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Holy cow Simon!!! That did the trick!!!
>
> Thank you so much! I feel like I should send you coffee or beer or
> something :)
>
> Steve
>
> On Fri, Jul 8, 2022 at 11:48 AM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > I recall an issue a while back where the Windows Virtio drivers didn't
> > support generic support offload (GRO) and it created some performance
> > problems with Cisco NICs.
> > This was particular to Windows and didn't occur for Linux VMs.
> >
> > Can you try disabling GRO on your NICs and retesting?
> > ethtool -K <interface> gro off
> >
> > -Si
> > ________________________________
> > From: S.Fuller <st...@gmail.com>
> > Sent: Thursday, July 7, 2022 9:00 AM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > The servers in question are in a Cisco UCS blade chassis. The physical
> NIC
> > card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
> > itself multiple virtual NICs to the server. Those virtual NICs use the
> > Cisco enic driver.
> >
> > Steve
> >
> > On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> >
> > > ok, I managed to access your xml file via the mailing list archives.
> > >
> > > <interface type="bridge">
> > > <mac address="02:00:69:fe:00:17"/>
> > > <source bridge="breth2-315"/>
> > > <target dev="vnet4"/>
> > > <model type="virtio"/>
> > > <link state="up"/>
> > > <alias name="net2"/>
> > > <address type="pci" domain="0x0000" bus="0x00" slot="0x03"
> > function="0x0"/>
> > > </interface>
> > >
> > > So, it does look correct in terms of virtio-net. One thing of note is
> > that
> > > kvm guests can't transmit or recieve packets in parallel, since
> > virtio-net
> > > only has one TX and RX queue. This tends to reduce performance on
> Windows
> > > guests.
> > > This can be addressed by enabling multi-queue (you assign a queue for
> > > vCPU). There was talk of this back in 2021 on the list, but I'm not
> sure
> > > whether it was implemented or not.
> > >
> > > What brand and chipset are your NICs on the servers?
> > >
> > >
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Simon Weller <sw...@ena.com.INVALID>
> > > Sent: Friday, July 1, 2022 2:34 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > I'm assuming the para-virtualized network driver is probably not being
> > > used either based on the template.
> > > If you take a close look at the xml for the interface, you should be
> able
> > > to confirm that or not.
> > >
> > > The templates tend to package all the para-virtualized drivers
> together,
> > > so they're normally in use, or they're not.
> > >
> > >
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>
> > > Sent: Friday, July 1, 2022 1:52 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > Will give that a try. I'm curious why you think changing the disk
> drivers
> > > over to virtio will have an effect on the network throughput.
> > >
> > > - Steve
> > >
> > > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
> > > wrote:
> > >
> > > > Unfortunately, I can't open the xml via the mailing list, but based
> on
> > > the
> > > > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> > > Change
> > > > the template to Virtio-SCSI 64bit and the correct interface drive
> type
> > > will
> > > > be specified in the XML.
> > > >
> > > > ________________________________
> > > > From: S.Fuller <st...@gmail.com>
> > > > Sent: Thursday, June 30, 2022 12:43 PM
> > > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > > Subject: Re: Asymmetric traffic issues?
> > > >
> > > >
> > > > I'm using the Windows Server 2016 profile for the template. After the
> > VM
> > > > came up, I installed the virtio drivers from the latest ISO, and then
> > > added
> > > > a new NIC so that it used the virtio network drivers vs the default
> > E1000
> > > > drivers.
> > > >
> > > > XML dump is attached
> > > >
> > > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller
> <sweller@ena.com.invalid
> > >
> > > > wrote:
> > > > Can you provide a redacted dump of your libvirt xml file for the VM?
> > > > Also, which OS profile are you using in Cloudstack for this? The
> > Windows
> > > > specific ones don't use virtio if I remember correctly, so you will
> > need
> > > to
> > > > select Virtio-SCSI 64bit in order for the xml to be built correctly.
> > > >
> > > > -Si
> > > > ________________________________
> > > > From: S.Fuller <st...@gmail.com>>
> > > > Sent: Thursday, June 30, 2022 11:38 AM
> > > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> <
> > > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > > Subject: Re: Asymmetric traffic issues?
> > > >
> > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> > when
> > > > clicking links, opening attachments, or complying with requests.
> Click
> > > the
> > > > "Phish Alert Report" button above the email, or contact MIS,
> regarding
> > > any
> > > > suspicious message.
> > > >
> > > >
> > > >
> > > > Neither the block nor the scsi drivers appear to be running as far
> as I
> > > can
> > > > tell.
> > > >
> > > > - Steve
> > > >
> > > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller
> <sweller@ena.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Steve,
> > > > >
> > > > > Are you running the virtio-block or virtio-scsi drivers?
> > > > >
> > > > > -Si
> > > > >
> > > > >
> > > > > ________________________________
> > > > > From: S.Fuller <stevefuller@gmail.com<mailto:stevefuller@gmail.com
> >>
> > > > > Sent: Thursday, June 30, 2022 8:52 AM
> > > > > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org
> >
> > <
> > > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > > > Subject: Re: Asymmetric traffic issues?
> > > > >
> > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> > > when
> > > > > clicking links, opening attachments, or complying with requests.
> > Click
> > > > the
> > > > > "Phish Alert Report" button above the email, or contact MIS,
> > regarding
> > > > any
> > > > > suspicious message.
> > > > >
> > > > >
> > > > >
> > > > > Well, I checked everything that I could for any QOS settings and
> > there
> > > is
> > > > > nothing configured there. What is curious is that Idon't see this
> > > > behavior
> > > > > when the transmitting host is running Linux, or if I'm using the
> > E1000
> > > > > drivers (although with the E1000 driver, the overall throughput is
> > > > lower).
> > > > > It really feels like I'mrunning into some weird issue with the
> virtio
> > > > > drivers on Windows. My Windows hosts are (to my knowledge) using
> the
> > > > latest
> > > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> > > > >
> > > > > Steve Fuller
> > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > >
> > > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com
> > > <mailto:
> > > > stevefuller@gmail.com>> wrote:
> > > > >
> > > > > > Vivek,
> > > > > >
> > > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will
> > review
> > > > the
> > > > > > QoS on the hypervisor for both of the nodes.
> > > > > >
> > > > > > I'm checking throughput between two different VMs that running on
> > two
> > > > > > different hosts within the same cluster. As of right now, I'm
> > > receiving
> > > > > > similar results using both iperf3 and nuttcp as the testing
> tools.
> > We
> > > > are
> > > > > > only seeing this issue when the VM is not on the same host as the
> > > > vrouter
> > > > > > for its isolated network.
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > > > > <vivek.kumar@indiqus.com<mailto:vivek.kumar@indiqus.com
> >.invalid>
> > > > wrote:
> > > > > >
> > > > > >> Hey Fuller,
> > > > > >>
> > > > > >> What hypervisor are you using ? I know you have checked all
> > > bandwidth
> > > > > >> limit on templates and global settings, but it’s worth to check
> > the
> > > > QoS
> > > > > on
> > > > > >> the hypervisor level, because at the end it’s the hypervisor
> which
> > > > > manages
> > > > > >> all. And from where are you trying to check the network
> > throughout,
> > > > > >> between client and server ?
> > > > > >>
> > > > > >>
> > > > > >> Vivek Kumar
> > > > > >> Sr. Manager - Cloud & DevOps
> > > > > >> TechOps | Indiqus Technologies
> > > > > >>
> > > > > >> + 91 7503460090 <tel:++91+7503460090>
> > > > > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> > > > <ma...@indiqus.com>>
> > > > > >> www.indiqus.com<http://www.indiqus.com><
> > > > http://www.indiqus.com> <
> > > > > https://www.indiqus.com/>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com
> > > <mailto:
> > > > stevefuller@gmail.com>> wrote:
> > > > > >> >
> > > > > >> > Environment:
> > > > > >> >
> > > > > >> > Two physical hosts
> > > > > >> > - Cloudstack 4.11.3
> > > > > >> > - Verified that there are no bandwidth limits in place on any
> of
> > > the
> > > > > >> > templates or in global settings.
> > > > > >> >
> > > > > >> > Two isolated networks ("Client" and "Server")
> > > > > >> > - Each has a vrouter with a public and private address
> > > > > >> > - One Windows 2016 VM on each network (running the latest
> virtio
> > > > > >> drivers)
> > > > > >> > - each node running latest version of Iperf3 to test
> throughput
> > > > > >> >
> > > > > >> > Testing/Observation:
> > > > > >> >
> > > > > >> > If the Client VM and the vrouter for the isolated Client
> network
> > > are
> > > > > on
> > > > > >> the
> > > > > >> > same physical host, we see symmetrical throughput in the 2
> Gbps
> > > > range,
> > > > > >> > whether we run iperf in regular mode or in reverse mode (iperf
> > > -R).
> > > > > >> >
> > > > > >> > If the Client VM and the vrouter for the isolated Client
> network
> > > are
> > > > > on
> > > > > >> > different physical hosts, we are seeing 25% of the throughput
> > > > running
> > > > > >> iperf
> > > > > >> > in regular mode vs running it in reverse mode.
> > > > > >> >
> > > > > >> > Has anyone encountered this issue before? If we change the
> > Client
> > > VM
> > > > > to
> > > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver,
> we
> > > see
> > > > > >> > symmetrical throughput in our tests, no matter where the
> vrouter
> > > is
> > > > in
> > > > > >> > relation to the Client VM.
> > > > > >> >
> > > > > >> > --
> > > > > >> > Steve Fuller
> > > > > >> > stevefuller@gmail.com<ma...@gmail.com>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> This message is intended only for the use of the individual or
> > > entity
> > > > to
> > > > > >> which it is addressed and may contain confidential and/or
> > privileged
> > > > > >> information. If you are not the intended recipient, please
> delete
> > > the
> > > > > >> original message and any copy of it from your computer system.
> You
> > > are
> > > > > >> hereby notified that any dissemination, distribution or copying
> of
> > > > this
> > > > > >> communication is strictly prohibited unless proper authorization
> > has
> > > > > been
> > > > > >> obtained for such action. If you have received this
> communication
> > in
> > > > > >> error,
> > > > > >> please notify the sender immediately. Although IndiQus attempts
> to
> > > > sweep
> > > > > >> e-mail and attachments for viruses, it does not guarantee that
> > both
> > > > are
> > > > > >> virus-free and accepts no liability for any damage sustained as
> a
> > > > result
> > > > > >> of
> > > > > >> viruses.
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Steve Fuller
> > > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Steve Fuller
> > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > >
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by "S.Fuller" <st...@gmail.com>.
Update on this, we were able to enable the same performance changes by
using ethtool to disable gro on the private and public interfaces on the
virtual routers.
Steve
On Fri, Jul 8, 2022 at 2:14 PM Simon Weller <sw...@ena.com.invalid> wrote:
> That's great news! Glad it all worked out.
>
> -Si
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Friday, July 8, 2022 12:49 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Holy cow Simon!!! That did the trick!!!
>
> Thank you so much! I feel like I should send you coffee or beer or
> something :)
>
> Steve
>
> On Fri, Jul 8, 2022 at 11:48 AM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > I recall an issue a while back where the Windows Virtio drivers didn't
> > support generic support offload (GRO) and it created some performance
> > problems with Cisco NICs.
> > This was particular to Windows and didn't occur for Linux VMs.
> >
> > Can you try disabling GRO on your NICs and retesting?
> > ethtool -K <interface> gro off
> >
> > -Si
> > ________________________________
> > From: S.Fuller <st...@gmail.com>
> > Sent: Thursday, July 7, 2022 9:00 AM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > The servers in question are in a Cisco UCS blade chassis. The physical
> NIC
> > card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
> > itself multiple virtual NICs to the server. Those virtual NICs use the
> > Cisco enic driver.
> >
> > Steve
> >
> > On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> >
> > > ok, I managed to access your xml file via the mailing list archives.
> > >
> > > <interface type="bridge">
> > > <mac address="02:00:69:fe:00:17"/>
> > > <source bridge="breth2-315"/>
> > > <target dev="vnet4"/>
> > > <model type="virtio"/>
> > > <link state="up"/>
> > > <alias name="net2"/>
> > > <address type="pci" domain="0x0000" bus="0x00" slot="0x03"
> > function="0x0"/>
> > > </interface>
> > >
> > > So, it does look correct in terms of virtio-net. One thing of note is
> > that
> > > kvm guests can't transmit or recieve packets in parallel, since
> > virtio-net
> > > only has one TX and RX queue. This tends to reduce performance on
> Windows
> > > guests.
> > > This can be addressed by enabling multi-queue (you assign a queue for
> > > vCPU). There was talk of this back in 2021 on the list, but I'm not
> sure
> > > whether it was implemented or not.
> > >
> > > What brand and chipset are your NICs on the servers?
> > >
> > >
> > >
> > >
> > >
> > >
> > > ________________________________
> > > From: Simon Weller <sw...@ena.com.INVALID>
> > > Sent: Friday, July 1, 2022 2:34 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > I'm assuming the para-virtualized network driver is probably not being
> > > used either based on the template.
> > > If you take a close look at the xml for the interface, you should be
> able
> > > to confirm that or not.
> > >
> > > The templates tend to package all the para-virtualized drivers
> together,
> > > so they're normally in use, or they're not.
> > >
> > >
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>
> > > Sent: Friday, July 1, 2022 1:52 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > Will give that a try. I'm curious why you think changing the disk
> drivers
> > > over to virtio will have an effect on the network throughput.
> > >
> > > - Steve
> > >
> > > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
> > > wrote:
> > >
> > > > Unfortunately, I can't open the xml via the mailing list, but based
> on
> > > the
> > > > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> > > Change
> > > > the template to Virtio-SCSI 64bit and the correct interface drive
> type
> > > will
> > > > be specified in the XML.
> > > >
> > > > ________________________________
> > > > From: S.Fuller <st...@gmail.com>
> > > > Sent: Thursday, June 30, 2022 12:43 PM
> > > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > > Subject: Re: Asymmetric traffic issues?
> > > >
> > > >
> > > > I'm using the Windows Server 2016 profile for the template. After the
> > VM
> > > > came up, I installed the virtio drivers from the latest ISO, and then
> > > added
> > > > a new NIC so that it used the virtio network drivers vs the default
> > E1000
> > > > drivers.
> > > >
> > > > XML dump is attached
> > > >
> > > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller
> <sweller@ena.com.invalid
> > >
> > > > wrote:
> > > > Can you provide a redacted dump of your libvirt xml file for the VM?
> > > > Also, which OS profile are you using in Cloudstack for this? The
> > Windows
> > > > specific ones don't use virtio if I remember correctly, so you will
> > need
> > > to
> > > > select Virtio-SCSI 64bit in order for the xml to be built correctly.
> > > >
> > > > -Si
> > > > ________________________________
> > > > From: S.Fuller <st...@gmail.com>>
> > > > Sent: Thursday, June 30, 2022 11:38 AM
> > > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> <
> > > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > > Subject: Re: Asymmetric traffic issues?
> > > >
> > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> > when
> > > > clicking links, opening attachments, or complying with requests.
> Click
> > > the
> > > > "Phish Alert Report" button above the email, or contact MIS,
> regarding
> > > any
> > > > suspicious message.
> > > >
> > > >
> > > >
> > > > Neither the block nor the scsi drivers appear to be running as far
> as I
> > > can
> > > > tell.
> > > >
> > > > - Steve
> > > >
> > > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller
> <sweller@ena.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Steve,
> > > > >
> > > > > Are you running the virtio-block or virtio-scsi drivers?
> > > > >
> > > > > -Si
> > > > >
> > > > >
> > > > > ________________________________
> > > > > From: S.Fuller <stevefuller@gmail.com<mailto:stevefuller@gmail.com
> >>
> > > > > Sent: Thursday, June 30, 2022 8:52 AM
> > > > > To: users@cloudstack.apache.org<mailto:users@cloudstack.apache.org
> >
> > <
> > > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > > > Subject: Re: Asymmetric traffic issues?
> > > > >
> > > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> > > when
> > > > > clicking links, opening attachments, or complying with requests.
> > Click
> > > > the
> > > > > "Phish Alert Report" button above the email, or contact MIS,
> > regarding
> > > > any
> > > > > suspicious message.
> > > > >
> > > > >
> > > > >
> > > > > Well, I checked everything that I could for any QOS settings and
> > there
> > > is
> > > > > nothing configured there. What is curious is that Idon't see this
> > > > behavior
> > > > > when the transmitting host is running Linux, or if I'm using the
> > E1000
> > > > > drivers (although with the E1000 driver, the overall throughput is
> > > > lower).
> > > > > It really feels like I'mrunning into some weird issue with the
> virtio
> > > > > drivers on Windows. My Windows hosts are (to my knowledge) using
> the
> > > > latest
> > > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> > > > >
> > > > > Steve Fuller
> > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > >
> > > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com
> > > <mailto:
> > > > stevefuller@gmail.com>> wrote:
> > > > >
> > > > > > Vivek,
> > > > > >
> > > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will
> > review
> > > > the
> > > > > > QoS on the hypervisor for both of the nodes.
> > > > > >
> > > > > > I'm checking throughput between two different VMs that running on
> > two
> > > > > > different hosts within the same cluster. As of right now, I'm
> > > receiving
> > > > > > similar results using both iperf3 and nuttcp as the testing
> tools.
> > We
> > > > are
> > > > > > only seeing this issue when the VM is not on the same host as the
> > > > vrouter
> > > > > > for its isolated network.
> > > > > >
> > > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > > > > <vivek.kumar@indiqus.com<mailto:vivek.kumar@indiqus.com
> >.invalid>
> > > > wrote:
> > > > > >
> > > > > >> Hey Fuller,
> > > > > >>
> > > > > >> What hypervisor are you using ? I know you have checked all
> > > bandwidth
> > > > > >> limit on templates and global settings, but it’s worth to check
> > the
> > > > QoS
> > > > > on
> > > > > >> the hypervisor level, because at the end it’s the hypervisor
> which
> > > > > manages
> > > > > >> all. And from where are you trying to check the network
> > throughout,
> > > > > >> between client and server ?
> > > > > >>
> > > > > >>
> > > > > >> Vivek Kumar
> > > > > >> Sr. Manager - Cloud & DevOps
> > > > > >> TechOps | Indiqus Technologies
> > > > > >>
> > > > > >> + 91 7503460090 <tel:++91+7503460090>
> > > > > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> > > > <ma...@indiqus.com>>
> > > > > >> www.indiqus.com<http://www.indiqus.com><
> > > > http://www.indiqus.com> <
> > > > > https://www.indiqus.com/>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com
> > > <mailto:
> > > > stevefuller@gmail.com>> wrote:
> > > > > >> >
> > > > > >> > Environment:
> > > > > >> >
> > > > > >> > Two physical hosts
> > > > > >> > - Cloudstack 4.11.3
> > > > > >> > - Verified that there are no bandwidth limits in place on any
> of
> > > the
> > > > > >> > templates or in global settings.
> > > > > >> >
> > > > > >> > Two isolated networks ("Client" and "Server")
> > > > > >> > - Each has a vrouter with a public and private address
> > > > > >> > - One Windows 2016 VM on each network (running the latest
> virtio
> > > > > >> drivers)
> > > > > >> > - each node running latest version of Iperf3 to test
> throughput
> > > > > >> >
> > > > > >> > Testing/Observation:
> > > > > >> >
> > > > > >> > If the Client VM and the vrouter for the isolated Client
> network
> > > are
> > > > > on
> > > > > >> the
> > > > > >> > same physical host, we see symmetrical throughput in the 2
> Gbps
> > > > range,
> > > > > >> > whether we run iperf in regular mode or in reverse mode (iperf
> > > -R).
> > > > > >> >
> > > > > >> > If the Client VM and the vrouter for the isolated Client
> network
> > > are
> > > > > on
> > > > > >> > different physical hosts, we are seeing 25% of the throughput
> > > > running
> > > > > >> iperf
> > > > > >> > in regular mode vs running it in reverse mode.
> > > > > >> >
> > > > > >> > Has anyone encountered this issue before? If we change the
> > Client
> > > VM
> > > > > to
> > > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver,
> we
> > > see
> > > > > >> > symmetrical throughput in our tests, no matter where the
> vrouter
> > > is
> > > > in
> > > > > >> > relation to the Client VM.
> > > > > >> >
> > > > > >> > --
> > > > > >> > Steve Fuller
> > > > > >> > stevefuller@gmail.com<ma...@gmail.com>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> This message is intended only for the use of the individual or
> > > entity
> > > > to
> > > > > >> which it is addressed and may contain confidential and/or
> > privileged
> > > > > >> information. If you are not the intended recipient, please
> delete
> > > the
> > > > > >> original message and any copy of it from your computer system.
> You
> > > are
> > > > > >> hereby notified that any dissemination, distribution or copying
> of
> > > > this
> > > > > >> communication is strictly prohibited unless proper authorization
> > has
> > > > > been
> > > > > >> obtained for such action. If you have received this
> communication
> > in
> > > > > >> error,
> > > > > >> please notify the sender immediately. Although IndiQus attempts
> to
> > > > sweep
> > > > > >> e-mail and attachments for viruses, it does not guarantee that
> > both
> > > > are
> > > > > >> virus-free and accepts no liability for any damage sustained as
> a
> > > > result
> > > > > >> of
> > > > > >> viruses.
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Steve Fuller
> > > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Steve Fuller
> > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > >
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by Simon Weller <sw...@ena.com.INVALID>.
That's great news! Glad it all worked out.
-Si
________________________________
From: S.Fuller <st...@gmail.com>
Sent: Friday, July 8, 2022 12:49 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message.
Holy cow Simon!!! That did the trick!!!
Thank you so much! I feel like I should send you coffee or beer or
something :)
Steve
On Fri, Jul 8, 2022 at 11:48 AM Simon Weller <sw...@ena.com.invalid>
wrote:
> I recall an issue a while back where the Windows Virtio drivers didn't
> support generic support offload (GRO) and it created some performance
> problems with Cisco NICs.
> This was particular to Windows and didn't occur for Linux VMs.
>
> Can you try disabling GRO on your NICs and retesting?
> ethtool -K <interface> gro off
>
> -Si
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Thursday, July 7, 2022 9:00 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> The servers in question are in a Cisco UCS blade chassis. The physical NIC
> card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
> itself multiple virtual NICs to the server. Those virtual NICs use the
> Cisco enic driver.
>
> Steve
>
> On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > ok, I managed to access your xml file via the mailing list archives.
> >
> > <interface type="bridge">
> > <mac address="02:00:69:fe:00:17"/>
> > <source bridge="breth2-315"/>
> > <target dev="vnet4"/>
> > <model type="virtio"/>
> > <link state="up"/>
> > <alias name="net2"/>
> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03"
> function="0x0"/>
> > </interface>
> >
> > So, it does look correct in terms of virtio-net. One thing of note is
> that
> > kvm guests can't transmit or recieve packets in parallel, since
> virtio-net
> > only has one TX and RX queue. This tends to reduce performance on Windows
> > guests.
> > This can be addressed by enabling multi-queue (you assign a queue for
> > vCPU). There was talk of this back in 2021 on the list, but I'm not sure
> > whether it was implemented or not.
> >
> > What brand and chipset are your NICs on the servers?
> >
> >
> >
> >
> >
> >
> > ________________________________
> > From: Simon Weller <sw...@ena.com.INVALID>
> > Sent: Friday, July 1, 2022 2:34 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > I'm assuming the para-virtualized network driver is probably not being
> > used either based on the template.
> > If you take a close look at the xml for the interface, you should be able
> > to confirm that or not.
> >
> > The templates tend to package all the para-virtualized drivers together,
> > so they're normally in use, or they're not.
> >
> >
> > ________________________________
> > From: S.Fuller <st...@gmail.com>
> > Sent: Friday, July 1, 2022 1:52 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Will give that a try. I'm curious why you think changing the disk drivers
> > over to virtio will have an effect on the network throughput.
> >
> > - Steve
> >
> > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> >
> > > Unfortunately, I can't open the xml via the mailing list, but based on
> > the
> > > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> > Change
> > > the template to Virtio-SCSI 64bit and the correct interface drive type
> > will
> > > be specified in the XML.
> > >
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>
> > > Sent: Thursday, June 30, 2022 12:43 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > >
> > > I'm using the Windows Server 2016 profile for the template. After the
> VM
> > > came up, I installed the virtio drivers from the latest ISO, and then
> > added
> > > a new NIC so that it used the virtio network drivers vs the default
> E1000
> > > drivers.
> > >
> > > XML dump is attached
> > >
> > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sweller@ena.com.invalid
> >
> > > wrote:
> > > Can you provide a redacted dump of your libvirt xml file for the VM?
> > > Also, which OS profile are you using in Cloudstack for this? The
> Windows
> > > specific ones don't use virtio if I remember correctly, so you will
> need
> > to
> > > select Virtio-SCSI 64bit in order for the xml to be built correctly.
> > >
> > > -Si
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>>
> > > Sent: Thursday, June 30, 2022 11:38 AM
> > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > Neither the block nor the scsi drivers appear to be running as far as I
> > can
> > > tell.
> > >
> > > - Steve
> > >
> > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sweller@ena.com.invalid
> >
> > > wrote:
> > >
> > > > Steve,
> > > >
> > > > Are you running the virtio-block or virtio-scsi drivers?
> > > >
> > > > -Si
> > > >
> > > >
> > > > ________________________________
> > > > From: S.Fuller <st...@gmail.com>>
> > > > Sent: Thursday, June 30, 2022 8:52 AM
> > > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> <
> > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > > Subject: Re: Asymmetric traffic issues?
> > > >
> > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> > when
> > > > clicking links, opening attachments, or complying with requests.
> Click
> > > the
> > > > "Phish Alert Report" button above the email, or contact MIS,
> regarding
> > > any
> > > > suspicious message.
> > > >
> > > >
> > > >
> > > > Well, I checked everything that I could for any QOS settings and
> there
> > is
> > > > nothing configured there. What is curious is that Idon't see this
> > > behavior
> > > > when the transmitting host is running Linux, or if I'm using the
> E1000
> > > > drivers (although with the E1000 driver, the overall throughput is
> > > lower).
> > > > It really feels like I'mrunning into some weird issue with the virtio
> > > > drivers on Windows. My Windows hosts are (to my knowledge) using the
> > > latest
> > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> > > >
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com
> > <mailto:
> > > stevefuller@gmail.com>> wrote:
> > > >
> > > > > Vivek,
> > > > >
> > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will
> review
> > > the
> > > > > QoS on the hypervisor for both of the nodes.
> > > > >
> > > > > I'm checking throughput between two different VMs that running on
> two
> > > > > different hosts within the same cluster. As of right now, I'm
> > receiving
> > > > > similar results using both iperf3 and nuttcp as the testing tools.
> We
> > > are
> > > > > only seeing this issue when the VM is not on the same host as the
> > > vrouter
> > > > > for its isolated network.
> > > > >
> > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > > > <vi...@indiqus.com>.invalid>
> > > wrote:
> > > > >
> > > > >> Hey Fuller,
> > > > >>
> > > > >> What hypervisor are you using ? I know you have checked all
> > bandwidth
> > > > >> limit on templates and global settings, but it’s worth to check
> the
> > > QoS
> > > > on
> > > > >> the hypervisor level, because at the end it’s the hypervisor which
> > > > manages
> > > > >> all. And from where are you trying to check the network
> throughout,
> > > > >> between client and server ?
> > > > >>
> > > > >>
> > > > >> Vivek Kumar
> > > > >> Sr. Manager - Cloud & DevOps
> > > > >> TechOps | Indiqus Technologies
> > > > >>
> > > > >> + 91 7503460090 <tel:++91+7503460090>
> > > > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> > > <ma...@indiqus.com>>
> > > > >> www.indiqus.com<http://www.indiqus.com><
> > > http://www.indiqus.com> <
> > > > https://www.indiqus.com/>
> > > > >>
> > > > >>
> > > > >>
> > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com
> > <mailto:
> > > stevefuller@gmail.com>> wrote:
> > > > >> >
> > > > >> > Environment:
> > > > >> >
> > > > >> > Two physical hosts
> > > > >> > - Cloudstack 4.11.3
> > > > >> > - Verified that there are no bandwidth limits in place on any of
> > the
> > > > >> > templates or in global settings.
> > > > >> >
> > > > >> > Two isolated networks ("Client" and "Server")
> > > > >> > - Each has a vrouter with a public and private address
> > > > >> > - One Windows 2016 VM on each network (running the latest virtio
> > > > >> drivers)
> > > > >> > - each node running latest version of Iperf3 to test throughput
> > > > >> >
> > > > >> > Testing/Observation:
> > > > >> >
> > > > >> > If the Client VM and the vrouter for the isolated Client network
> > are
> > > > on
> > > > >> the
> > > > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> > > range,
> > > > >> > whether we run iperf in regular mode or in reverse mode (iperf
> > -R).
> > > > >> >
> > > > >> > If the Client VM and the vrouter for the isolated Client network
> > are
> > > > on
> > > > >> > different physical hosts, we are seeing 25% of the throughput
> > > running
> > > > >> iperf
> > > > >> > in regular mode vs running it in reverse mode.
> > > > >> >
> > > > >> > Has anyone encountered this issue before? If we change the
> Client
> > VM
> > > > to
> > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we
> > see
> > > > >> > symmetrical throughput in our tests, no matter where the vrouter
> > is
> > > in
> > > > >> > relation to the Client VM.
> > > > >> >
> > > > >> > --
> > > > >> > Steve Fuller
> > > > >> > stevefuller@gmail.com<ma...@gmail.com>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> This message is intended only for the use of the individual or
> > entity
> > > to
> > > > >> which it is addressed and may contain confidential and/or
> privileged
> > > > >> information. If you are not the intended recipient, please delete
> > the
> > > > >> original message and any copy of it from your computer system. You
> > are
> > > > >> hereby notified that any dissemination, distribution or copying of
> > > this
> > > > >> communication is strictly prohibited unless proper authorization
> has
> > > > been
> > > > >> obtained for such action. If you have received this communication
> in
> > > > >> error,
> > > > >> please notify the sender immediately. Although IndiQus attempts to
> > > sweep
> > > > >> e-mail and attachments for viruses, it does not guarantee that
> both
> > > are
> > > > >> virus-free and accepts no liability for any damage sustained as a
> > > result
> > > > >> of
> > > > >> viruses.
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Steve Fuller
> > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > >
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by "S.Fuller" <st...@gmail.com>.
Holy cow Simon!!! That did the trick!!!
Thank you so much! I feel like I should send you coffee or beer or
something :)
Steve
On Fri, Jul 8, 2022 at 11:48 AM Simon Weller <sw...@ena.com.invalid>
wrote:
> I recall an issue a while back where the Windows Virtio drivers didn't
> support generic support offload (GRO) and it created some performance
> problems with Cisco NICs.
> This was particular to Windows and didn't occur for Linux VMs.
>
> Can you try disabling GRO on your NICs and retesting?
> ethtool -K <interface> gro off
>
> -Si
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Thursday, July 7, 2022 9:00 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> The servers in question are in a Cisco UCS blade chassis. The physical NIC
> card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
> itself multiple virtual NICs to the server. Those virtual NICs use the
> Cisco enic driver.
>
> Steve
>
> On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > ok, I managed to access your xml file via the mailing list archives.
> >
> > <interface type="bridge">
> > <mac address="02:00:69:fe:00:17"/>
> > <source bridge="breth2-315"/>
> > <target dev="vnet4"/>
> > <model type="virtio"/>
> > <link state="up"/>
> > <alias name="net2"/>
> > <address type="pci" domain="0x0000" bus="0x00" slot="0x03"
> function="0x0"/>
> > </interface>
> >
> > So, it does look correct in terms of virtio-net. One thing of note is
> that
> > kvm guests can't transmit or recieve packets in parallel, since
> virtio-net
> > only has one TX and RX queue. This tends to reduce performance on Windows
> > guests.
> > This can be addressed by enabling multi-queue (you assign a queue for
> > vCPU). There was talk of this back in 2021 on the list, but I'm not sure
> > whether it was implemented or not.
> >
> > What brand and chipset are your NICs on the servers?
> >
> >
> >
> >
> >
> >
> > ________________________________
> > From: Simon Weller <sw...@ena.com.INVALID>
> > Sent: Friday, July 1, 2022 2:34 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > I'm assuming the para-virtualized network driver is probably not being
> > used either based on the template.
> > If you take a close look at the xml for the interface, you should be able
> > to confirm that or not.
> >
> > The templates tend to package all the para-virtualized drivers together,
> > so they're normally in use, or they're not.
> >
> >
> > ________________________________
> > From: S.Fuller <st...@gmail.com>
> > Sent: Friday, July 1, 2022 1:52 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Will give that a try. I'm curious why you think changing the disk drivers
> > over to virtio will have an effect on the network throughput.
> >
> > - Steve
> >
> > On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> >
> > > Unfortunately, I can't open the xml via the mailing list, but based on
> > the
> > > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> > Change
> > > the template to Virtio-SCSI 64bit and the correct interface drive type
> > will
> > > be specified in the XML.
> > >
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>
> > > Sent: Thursday, June 30, 2022 12:43 PM
> > > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > >
> > > I'm using the Windows Server 2016 profile for the template. After the
> VM
> > > came up, I installed the virtio drivers from the latest ISO, and then
> > added
> > > a new NIC so that it used the virtio network drivers vs the default
> E1000
> > > drivers.
> > >
> > > XML dump is attached
> > >
> > > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sweller@ena.com.invalid
> >
> > > wrote:
> > > Can you provide a redacted dump of your libvirt xml file for the VM?
> > > Also, which OS profile are you using in Cloudstack for this? The
> Windows
> > > specific ones don't use virtio if I remember correctly, so you will
> need
> > to
> > > select Virtio-SCSI 64bit in order for the xml to be built correctly.
> > >
> > > -Si
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>>
> > > Sent: Thursday, June 30, 2022 11:38 AM
> > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > Neither the block nor the scsi drivers appear to be running as far as I
> > can
> > > tell.
> > >
> > > - Steve
> > >
> > > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sweller@ena.com.invalid
> >
> > > wrote:
> > >
> > > > Steve,
> > > >
> > > > Are you running the virtio-block or virtio-scsi drivers?
> > > >
> > > > -Si
> > > >
> > > >
> > > > ________________________________
> > > > From: S.Fuller <st...@gmail.com>>
> > > > Sent: Thursday, June 30, 2022 8:52 AM
> > > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> <
> > > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > > Subject: Re: Asymmetric traffic issues?
> > > >
> > > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> > when
> > > > clicking links, opening attachments, or complying with requests.
> Click
> > > the
> > > > "Phish Alert Report" button above the email, or contact MIS,
> regarding
> > > any
> > > > suspicious message.
> > > >
> > > >
> > > >
> > > > Well, I checked everything that I could for any QOS settings and
> there
> > is
> > > > nothing configured there. What is curious is that Idon't see this
> > > behavior
> > > > when the transmitting host is running Linux, or if I'm using the
> E1000
> > > > drivers (although with the E1000 driver, the overall throughput is
> > > lower).
> > > > It really feels like I'mrunning into some weird issue with the virtio
> > > > drivers on Windows. My Windows hosts are (to my knowledge) using the
> > > latest
> > > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> > > >
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com
> > <mailto:
> > > stevefuller@gmail.com>> wrote:
> > > >
> > > > > Vivek,
> > > > >
> > > > > Thanks for the reply. I am using the KVM hypervisor. I'll will
> review
> > > the
> > > > > QoS on the hypervisor for both of the nodes.
> > > > >
> > > > > I'm checking throughput between two different VMs that running on
> two
> > > > > different hosts within the same cluster. As of right now, I'm
> > receiving
> > > > > similar results using both iperf3 and nuttcp as the testing tools.
> We
> > > are
> > > > > only seeing this issue when the VM is not on the same host as the
> > > vrouter
> > > > > for its isolated network.
> > > > >
> > > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > > > <vi...@indiqus.com>.invalid>
> > > wrote:
> > > > >
> > > > >> Hey Fuller,
> > > > >>
> > > > >> What hypervisor are you using ? I know you have checked all
> > bandwidth
> > > > >> limit on templates and global settings, but it’s worth to check
> the
> > > QoS
> > > > on
> > > > >> the hypervisor level, because at the end it’s the hypervisor which
> > > > manages
> > > > >> all. And from where are you trying to check the network
> throughout,
> > > > >> between client and server ?
> > > > >>
> > > > >>
> > > > >> Vivek Kumar
> > > > >> Sr. Manager - Cloud & DevOps
> > > > >> TechOps | Indiqus Technologies
> > > > >>
> > > > >> + 91 7503460090 <tel:++91+7503460090>
> > > > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> > > <ma...@indiqus.com>>
> > > > >> www.indiqus.com<http://www.indiqus.com><
> > > http://www.indiqus.com> <
> > > > https://www.indiqus.com/>
> > > > >>
> > > > >>
> > > > >>
> > > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com
> > <mailto:
> > > stevefuller@gmail.com>> wrote:
> > > > >> >
> > > > >> > Environment:
> > > > >> >
> > > > >> > Two physical hosts
> > > > >> > - Cloudstack 4.11.3
> > > > >> > - Verified that there are no bandwidth limits in place on any of
> > the
> > > > >> > templates or in global settings.
> > > > >> >
> > > > >> > Two isolated networks ("Client" and "Server")
> > > > >> > - Each has a vrouter with a public and private address
> > > > >> > - One Windows 2016 VM on each network (running the latest virtio
> > > > >> drivers)
> > > > >> > - each node running latest version of Iperf3 to test throughput
> > > > >> >
> > > > >> > Testing/Observation:
> > > > >> >
> > > > >> > If the Client VM and the vrouter for the isolated Client network
> > are
> > > > on
> > > > >> the
> > > > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> > > range,
> > > > >> > whether we run iperf in regular mode or in reverse mode (iperf
> > -R).
> > > > >> >
> > > > >> > If the Client VM and the vrouter for the isolated Client network
> > are
> > > > on
> > > > >> > different physical hosts, we are seeing 25% of the throughput
> > > running
> > > > >> iperf
> > > > >> > in regular mode vs running it in reverse mode.
> > > > >> >
> > > > >> > Has anyone encountered this issue before? If we change the
> Client
> > VM
> > > > to
> > > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we
> > see
> > > > >> > symmetrical throughput in our tests, no matter where the vrouter
> > is
> > > in
> > > > >> > relation to the Client VM.
> > > > >> >
> > > > >> > --
> > > > >> > Steve Fuller
> > > > >> > stevefuller@gmail.com<ma...@gmail.com>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> This message is intended only for the use of the individual or
> > entity
> > > to
> > > > >> which it is addressed and may contain confidential and/or
> privileged
> > > > >> information. If you are not the intended recipient, please delete
> > the
> > > > >> original message and any copy of it from your computer system. You
> > are
> > > > >> hereby notified that any dissemination, distribution or copying of
> > > this
> > > > >> communication is strictly prohibited unless proper authorization
> has
> > > > been
> > > > >> obtained for such action. If you have received this communication
> in
> > > > >> error,
> > > > >> please notify the sender immediately. Although IndiQus attempts to
> > > sweep
> > > > >> e-mail and attachments for viruses, it does not guarantee that
> both
> > > are
> > > > >> virus-free and accepts no liability for any damage sustained as a
> > > result
> > > > >> of
> > > > >> viruses.
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Steve Fuller
> > > > > stevefuller@gmail.com<ma...@gmail.com>
> > > > >
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by Simon Weller <sw...@ena.com.INVALID>.
I recall an issue a while back where the Windows Virtio drivers didn't support generic support offload (GRO) and it created some performance problems with Cisco NICs.
This was particular to Windows and didn't occur for Linux VMs.
Can you try disabling GRO on your NICs and retesting?
ethtool -K <interface> gro off
-Si
________________________________
From: S.Fuller <st...@gmail.com>
Sent: Thursday, July 7, 2022 9:00 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message.
The servers in question are in a Cisco UCS blade chassis. The physical NIC
card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
itself multiple virtual NICs to the server. Those virtual NICs use the
Cisco enic driver.
Steve
On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <sw...@ena.com.invalid> wrote:
> ok, I managed to access your xml file via the mailing list archives.
>
> <interface type="bridge">
> <mac address="02:00:69:fe:00:17"/>
> <source bridge="breth2-315"/>
> <target dev="vnet4"/>
> <model type="virtio"/>
> <link state="up"/>
> <alias name="net2"/>
> <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
> </interface>
>
> So, it does look correct in terms of virtio-net. One thing of note is that
> kvm guests can't transmit or recieve packets in parallel, since virtio-net
> only has one TX and RX queue. This tends to reduce performance on Windows
> guests.
> This can be addressed by enabling multi-queue (you assign a queue for
> vCPU). There was talk of this back in 2021 on the list, but I'm not sure
> whether it was implemented or not.
>
> What brand and chipset are your NICs on the servers?
>
>
>
>
>
>
> ________________________________
> From: Simon Weller <sw...@ena.com.INVALID>
> Sent: Friday, July 1, 2022 2:34 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> I'm assuming the para-virtualized network driver is probably not being
> used either based on the template.
> If you take a close look at the xml for the interface, you should be able
> to confirm that or not.
>
> The templates tend to package all the para-virtualized drivers together,
> so they're normally in use, or they're not.
>
>
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Friday, July 1, 2022 1:52 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Will give that a try. I'm curious why you think changing the disk drivers
> over to virtio will have an effect on the network throughput.
>
> - Steve
>
> On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > Unfortunately, I can't open the xml via the mailing list, but based on
> the
> > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> Change
> > the template to Virtio-SCSI 64bit and the correct interface drive type
> will
> > be specified in the XML.
> >
> > ________________________________
> > From: S.Fuller <st...@gmail.com>
> > Sent: Thursday, June 30, 2022 12:43 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> >
> > I'm using the Windows Server 2016 profile for the template. After the VM
> > came up, I installed the virtio drivers from the latest ISO, and then
> added
> > a new NIC so that it used the virtio network drivers vs the default E1000
> > drivers.
> >
> > XML dump is attached
> >
> > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> > Can you provide a redacted dump of your libvirt xml file for the VM?
> > Also, which OS profile are you using in Cloudstack for this? The Windows
> > specific ones don't use virtio if I remember correctly, so you will need
> to
> > select Virtio-SCSI 64bit in order for the xml to be built correctly.
> >
> > -Si
> > ________________________________
> > From: S.Fuller <st...@gmail.com>>
> > Sent: Thursday, June 30, 2022 11:38 AM
> > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Neither the block nor the scsi drivers appear to be running as far as I
> can
> > tell.
> >
> > - Steve
> >
> > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> >
> > > Steve,
> > >
> > > Are you running the virtio-block or virtio-scsi drivers?
> > >
> > > -Si
> > >
> > >
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>>
> > > Sent: Thursday, June 30, 2022 8:52 AM
> > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > Well, I checked everything that I could for any QOS settings and there
> is
> > > nothing configured there. What is curious is that Idon't see this
> > behavior
> > > when the transmitting host is running Linux, or if I'm using the E1000
> > > drivers (although with the E1000 driver, the overall throughput is
> > lower).
> > > It really feels like I'mrunning into some weird issue with the virtio
> > > drivers on Windows. My Windows hosts are (to my knowledge) using the
> > latest
> > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> > >
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com
> <mailto:
> > stevefuller@gmail.com>> wrote:
> > >
> > > > Vivek,
> > > >
> > > > Thanks for the reply. I am using the KVM hypervisor. I'll will review
> > the
> > > > QoS on the hypervisor for both of the nodes.
> > > >
> > > > I'm checking throughput between two different VMs that running on two
> > > > different hosts within the same cluster. As of right now, I'm
> receiving
> > > > similar results using both iperf3 and nuttcp as the testing tools. We
> > are
> > > > only seeing this issue when the VM is not on the same host as the
> > vrouter
> > > > for its isolated network.
> > > >
> > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > > <vi...@indiqus.com>.invalid>
> > wrote:
> > > >
> > > >> Hey Fuller,
> > > >>
> > > >> What hypervisor are you using ? I know you have checked all
> bandwidth
> > > >> limit on templates and global settings, but it’s worth to check the
> > QoS
> > > on
> > > >> the hypervisor level, because at the end it’s the hypervisor which
> > > manages
> > > >> all. And from where are you trying to check the network throughout,
> > > >> between client and server ?
> > > >>
> > > >>
> > > >> Vivek Kumar
> > > >> Sr. Manager - Cloud & DevOps
> > > >> TechOps | Indiqus Technologies
> > > >>
> > > >> + 91 7503460090 <tel:++91+7503460090>
> > > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> > <ma...@indiqus.com>>
> > > >> www.indiqus.com<http://www.indiqus.com><
> > http://www.indiqus.com> <
> > > https://www.indiqus.com/>
> > > >>
> > > >>
> > > >>
> > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com
> <mailto:
> > stevefuller@gmail.com>> wrote:
> > > >> >
> > > >> > Environment:
> > > >> >
> > > >> > Two physical hosts
> > > >> > - Cloudstack 4.11.3
> > > >> > - Verified that there are no bandwidth limits in place on any of
> the
> > > >> > templates or in global settings.
> > > >> >
> > > >> > Two isolated networks ("Client" and "Server")
> > > >> > - Each has a vrouter with a public and private address
> > > >> > - One Windows 2016 VM on each network (running the latest virtio
> > > >> drivers)
> > > >> > - each node running latest version of Iperf3 to test throughput
> > > >> >
> > > >> > Testing/Observation:
> > > >> >
> > > >> > If the Client VM and the vrouter for the isolated Client network
> are
> > > on
> > > >> the
> > > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> > range,
> > > >> > whether we run iperf in regular mode or in reverse mode (iperf
> -R).
> > > >> >
> > > >> > If the Client VM and the vrouter for the isolated Client network
> are
> > > on
> > > >> > different physical hosts, we are seeing 25% of the throughput
> > running
> > > >> iperf
> > > >> > in regular mode vs running it in reverse mode.
> > > >> >
> > > >> > Has anyone encountered this issue before? If we change the Client
> VM
> > > to
> > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we
> see
> > > >> > symmetrical throughput in our tests, no matter where the vrouter
> is
> > in
> > > >> > relation to the Client VM.
> > > >> >
> > > >> > --
> > > >> > Steve Fuller
> > > >> > stevefuller@gmail.com<ma...@gmail.com>
> > > >>
> > > >>
> > > >> --
> > > >> This message is intended only for the use of the individual or
> entity
> > to
> > > >> which it is addressed and may contain confidential and/or privileged
> > > >> information. If you are not the intended recipient, please delete
> the
> > > >> original message and any copy of it from your computer system. You
> are
> > > >> hereby notified that any dissemination, distribution or copying of
> > this
> > > >> communication is strictly prohibited unless proper authorization has
> > > been
> > > >> obtained for such action. If you have received this communication in
> > > >> error,
> > > >> please notify the sender immediately. Although IndiQus attempts to
> > sweep
> > > >> e-mail and attachments for viruses, it does not guarantee that both
> > are
> > > >> virus-free and accepts no liability for any damage sustained as a
> > result
> > > >> of
> > > >> viruses.
> > > >>
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by "S.Fuller" <st...@gmail.com>.
The servers in question are in a Cisco UCS blade chassis. The physical NIC
card is a Cisco UCS VIC 1240, it's a multiport 10gb card that presents
itself multiple virtual NICs to the server. Those virtual NICs use the
Cisco enic driver.
Steve
On Fri, Jul 1, 2022 at 4:05 PM Simon Weller <sw...@ena.com.invalid> wrote:
> ok, I managed to access your xml file via the mailing list archives.
>
> <interface type="bridge">
> <mac address="02:00:69:fe:00:17"/>
> <source bridge="breth2-315"/>
> <target dev="vnet4"/>
> <model type="virtio"/>
> <link state="up"/>
> <alias name="net2"/>
> <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
> </interface>
>
> So, it does look correct in terms of virtio-net. One thing of note is that
> kvm guests can't transmit or recieve packets in parallel, since virtio-net
> only has one TX and RX queue. This tends to reduce performance on Windows
> guests.
> This can be addressed by enabling multi-queue (you assign a queue for
> vCPU). There was talk of this back in 2021 on the list, but I'm not sure
> whether it was implemented or not.
>
> What brand and chipset are your NICs on the servers?
>
>
>
>
>
>
> ________________________________
> From: Simon Weller <sw...@ena.com.INVALID>
> Sent: Friday, July 1, 2022 2:34 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> I'm assuming the para-virtualized network driver is probably not being
> used either based on the template.
> If you take a close look at the xml for the interface, you should be able
> to confirm that or not.
>
> The templates tend to package all the para-virtualized drivers together,
> so they're normally in use, or they're not.
>
>
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Friday, July 1, 2022 1:52 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Will give that a try. I'm curious why you think changing the disk drivers
> over to virtio will have an effect on the network throughput.
>
> - Steve
>
> On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > Unfortunately, I can't open the xml via the mailing list, but based on
> the
> > template you're using, it's not setup for Virtio-blk or virtio-scsi.
> Change
> > the template to Virtio-SCSI 64bit and the correct interface drive type
> will
> > be specified in the XML.
> >
> > ________________________________
> > From: S.Fuller <st...@gmail.com>
> > Sent: Thursday, June 30, 2022 12:43 PM
> > To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> > Subject: Re: Asymmetric traffic issues?
> >
> >
> > I'm using the Windows Server 2016 profile for the template. After the VM
> > came up, I installed the virtio drivers from the latest ISO, and then
> added
> > a new NIC so that it used the virtio network drivers vs the default E1000
> > drivers.
> >
> > XML dump is attached
> >
> > On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> > Can you provide a redacted dump of your libvirt xml file for the VM?
> > Also, which OS profile are you using in Cloudstack for this? The Windows
> > specific ones don't use virtio if I remember correctly, so you will need
> to
> > select Virtio-SCSI 64bit in order for the xml to be built correctly.
> >
> > -Si
> > ________________________________
> > From: S.Fuller <st...@gmail.com>>
> > Sent: Thursday, June 30, 2022 11:38 AM
> > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Neither the block nor the scsi drivers appear to be running as far as I
> can
> > tell.
> >
> > - Steve
> >
> > On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sw...@ena.com.invalid>
> > wrote:
> >
> > > Steve,
> > >
> > > Are you running the virtio-block or virtio-scsi drivers?
> > >
> > > -Si
> > >
> > >
> > > ________________________________
> > > From: S.Fuller <st...@gmail.com>>
> > > Sent: Thursday, June 30, 2022 8:52 AM
> > > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> > users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > > Subject: Re: Asymmetric traffic issues?
> > >
> > > EXTERNAL EMAIL: This message originated outside of ENA. Use caution
> when
> > > clicking links, opening attachments, or complying with requests. Click
> > the
> > > "Phish Alert Report" button above the email, or contact MIS, regarding
> > any
> > > suspicious message.
> > >
> > >
> > >
> > > Well, I checked everything that I could for any QOS settings and there
> is
> > > nothing configured there. What is curious is that Idon't see this
> > behavior
> > > when the transmitting host is running Linux, or if I'm using the E1000
> > > drivers (although with the E1000 driver, the overall throughput is
> > lower).
> > > It really feels like I'mrunning into some weird issue with the virtio
> > > drivers on Windows. My Windows hosts are (to my knowledge) using the
> > latest
> > > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> > >
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> > > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com
> <mailto:
> > stevefuller@gmail.com>> wrote:
> > >
> > > > Vivek,
> > > >
> > > > Thanks for the reply. I am using the KVM hypervisor. I'll will review
> > the
> > > > QoS on the hypervisor for both of the nodes.
> > > >
> > > > I'm checking throughput between two different VMs that running on two
> > > > different hosts within the same cluster. As of right now, I'm
> receiving
> > > > similar results using both iperf3 and nuttcp as the testing tools. We
> > are
> > > > only seeing this issue when the VM is not on the same host as the
> > vrouter
> > > > for its isolated network.
> > > >
> > > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > > <vi...@indiqus.com>.invalid>
> > wrote:
> > > >
> > > >> Hey Fuller,
> > > >>
> > > >> What hypervisor are you using ? I know you have checked all
> bandwidth
> > > >> limit on templates and global settings, but it’s worth to check the
> > QoS
> > > on
> > > >> the hypervisor level, because at the end it’s the hypervisor which
> > > manages
> > > >> all. And from where are you trying to check the network throughout,
> > > >> between client and server ?
> > > >>
> > > >>
> > > >> Vivek Kumar
> > > >> Sr. Manager - Cloud & DevOps
> > > >> TechOps | Indiqus Technologies
> > > >>
> > > >> + 91 7503460090 <tel:++91+7503460090>
> > > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> > <ma...@indiqus.com>>
> > > >> www.indiqus.com<http://www.indiqus.com><
> > http://www.indiqus.com> <
> > > https://www.indiqus.com/>
> > > >>
> > > >>
> > > >>
> > > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com
> <mailto:
> > stevefuller@gmail.com>> wrote:
> > > >> >
> > > >> > Environment:
> > > >> >
> > > >> > Two physical hosts
> > > >> > - Cloudstack 4.11.3
> > > >> > - Verified that there are no bandwidth limits in place on any of
> the
> > > >> > templates or in global settings.
> > > >> >
> > > >> > Two isolated networks ("Client" and "Server")
> > > >> > - Each has a vrouter with a public and private address
> > > >> > - One Windows 2016 VM on each network (running the latest virtio
> > > >> drivers)
> > > >> > - each node running latest version of Iperf3 to test throughput
> > > >> >
> > > >> > Testing/Observation:
> > > >> >
> > > >> > If the Client VM and the vrouter for the isolated Client network
> are
> > > on
> > > >> the
> > > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> > range,
> > > >> > whether we run iperf in regular mode or in reverse mode (iperf
> -R).
> > > >> >
> > > >> > If the Client VM and the vrouter for the isolated Client network
> are
> > > on
> > > >> > different physical hosts, we are seeing 25% of the throughput
> > running
> > > >> iperf
> > > >> > in regular mode vs running it in reverse mode.
> > > >> >
> > > >> > Has anyone encountered this issue before? If we change the Client
> VM
> > > to
> > > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we
> see
> > > >> > symmetrical throughput in our tests, no matter where the vrouter
> is
> > in
> > > >> > relation to the Client VM.
> > > >> >
> > > >> > --
> > > >> > Steve Fuller
> > > >> > stevefuller@gmail.com<ma...@gmail.com>
> > > >>
> > > >>
> > > >> --
> > > >> This message is intended only for the use of the individual or
> entity
> > to
> > > >> which it is addressed and may contain confidential and/or privileged
> > > >> information. If you are not the intended recipient, please delete
> the
> > > >> original message and any copy of it from your computer system. You
> are
> > > >> hereby notified that any dissemination, distribution or copying of
> > this
> > > >> communication is strictly prohibited unless proper authorization has
> > > been
> > > >> obtained for such action. If you have received this communication in
> > > >> error,
> > > >> please notify the sender immediately. Although IndiQus attempts to
> > sweep
> > > >> e-mail and attachments for viruses, it does not guarantee that both
> > are
> > > >> virus-free and accepts no liability for any damage sustained as a
> > result
> > > >> of
> > > >> viruses.
> > > >>
> > > >
> > > >
> > > > --
> > > > Steve Fuller
> > > > stevefuller@gmail.com<ma...@gmail.com>
> > > >
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by Simon Weller <sw...@ena.com.INVALID>.
ok, I managed to access your xml file via the mailing list archives.
<interface type="bridge">
<mac address="02:00:69:fe:00:17"/>
<source bridge="breth2-315"/>
<target dev="vnet4"/>
<model type="virtio"/>
<link state="up"/>
<alias name="net2"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
</interface>
So, it does look correct in terms of virtio-net. One thing of note is that kvm guests can't transmit or recieve packets in parallel, since virtio-net only has one TX and RX queue. This tends to reduce performance on Windows guests.
This can be addressed by enabling multi-queue (you assign a queue for vCPU). There was talk of this back in 2021 on the list, but I'm not sure whether it was implemented or not.
What brand and chipset are your NICs on the servers?
________________________________
From: Simon Weller <sw...@ena.com.INVALID>
Sent: Friday, July 1, 2022 2:34 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message.
I'm assuming the para-virtualized network driver is probably not being used either based on the template.
If you take a close look at the xml for the interface, you should be able to confirm that or not.
The templates tend to package all the para-virtualized drivers together, so they're normally in use, or they're not.
________________________________
From: S.Fuller <st...@gmail.com>
Sent: Friday, July 1, 2022 1:52 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message.
Will give that a try. I'm curious why you think changing the disk drivers
over to virtio will have an effect on the network throughput.
- Steve
On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
wrote:
> Unfortunately, I can't open the xml via the mailing list, but based on the
> template you're using, it's not setup for Virtio-blk or virtio-scsi. Change
> the template to Virtio-SCSI 64bit and the correct interface drive type will
> be specified in the XML.
>
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Thursday, June 30, 2022 12:43 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
>
> I'm using the Windows Server 2016 profile for the template. After the VM
> came up, I installed the virtio drivers from the latest ISO, and then added
> a new NIC so that it used the virtio network drivers vs the default E1000
> drivers.
>
> XML dump is attached
>
> On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sw...@ena.com.invalid>
> wrote:
> Can you provide a redacted dump of your libvirt xml file for the VM?
> Also, which OS profile are you using in Cloudstack for this? The Windows
> specific ones don't use virtio if I remember correctly, so you will need to
> select Virtio-SCSI 64bit in order for the xml to be built correctly.
>
> -Si
> ________________________________
> From: S.Fuller <st...@gmail.com>>
> Sent: Thursday, June 30, 2022 11:38 AM
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Neither the block nor the scsi drivers appear to be running as far as I can
> tell.
>
> - Steve
>
> On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > Steve,
> >
> > Are you running the virtio-block or virtio-scsi drivers?
> >
> > -Si
> >
> >
> > ________________________________
> > From: S.Fuller <st...@gmail.com>>
> > Sent: Thursday, June 30, 2022 8:52 AM
> > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Well, I checked everything that I could for any QOS settings and there is
> > nothing configured there. What is curious is that Idon't see this
> behavior
> > when the transmitting host is running Linux, or if I'm using the E1000
> > drivers (although with the E1000 driver, the overall throughput is
> lower).
> > It really feels like I'mrunning into some weird issue with the virtio
> > drivers on Windows. My Windows hosts are (to my knowledge) using the
> latest
> > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> >
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
> > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com<mailto:
> stevefuller@gmail.com>> wrote:
> >
> > > Vivek,
> > >
> > > Thanks for the reply. I am using the KVM hypervisor. I'll will review
> the
> > > QoS on the hypervisor for both of the nodes.
> > >
> > > I'm checking throughput between two different VMs that running on two
> > > different hosts within the same cluster. As of right now, I'm receiving
> > > similar results using both iperf3 and nuttcp as the testing tools. We
> are
> > > only seeing this issue when the VM is not on the same host as the
> vrouter
> > > for its isolated network.
> > >
> > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > <vi...@indiqus.com>.invalid>
> wrote:
> > >
> > >> Hey Fuller,
> > >>
> > >> What hypervisor are you using ? I know you have checked all bandwidth
> > >> limit on templates and global settings, but it’s worth to check the
> QoS
> > on
> > >> the hypervisor level, because at the end it’s the hypervisor which
> > manages
> > >> all. And from where are you trying to check the network throughout,
> > >> between client and server ?
> > >>
> > >>
> > >> Vivek Kumar
> > >> Sr. Manager - Cloud & DevOps
> > >> TechOps | Indiqus Technologies
> > >>
> > >> + 91 7503460090 <tel:++91+7503460090>
> > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> <ma...@indiqus.com>>
> > >> www.indiqus.com<http://www.indiqus.com><
> http://www.indiqus.com> <
> > https://www.indiqus.com/>
> > >>
> > >>
> > >>
> > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com<mailto:
> stevefuller@gmail.com>> wrote:
> > >> >
> > >> > Environment:
> > >> >
> > >> > Two physical hosts
> > >> > - Cloudstack 4.11.3
> > >> > - Verified that there are no bandwidth limits in place on any of the
> > >> > templates or in global settings.
> > >> >
> > >> > Two isolated networks ("Client" and "Server")
> > >> > - Each has a vrouter with a public and private address
> > >> > - One Windows 2016 VM on each network (running the latest virtio
> > >> drivers)
> > >> > - each node running latest version of Iperf3 to test throughput
> > >> >
> > >> > Testing/Observation:
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> the
> > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> range,
> > >> > whether we run iperf in regular mode or in reverse mode (iperf -R).
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> > different physical hosts, we are seeing 25% of the throughput
> running
> > >> iperf
> > >> > in regular mode vs running it in reverse mode.
> > >> >
> > >> > Has anyone encountered this issue before? If we change the Client VM
> > to
> > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we see
> > >> > symmetrical throughput in our tests, no matter where the vrouter is
> in
> > >> > relation to the Client VM.
> > >> >
> > >> > --
> > >> > Steve Fuller
> > >> > stevefuller@gmail.com<ma...@gmail.com>
> > >>
> > >>
> > >> --
> > >> This message is intended only for the use of the individual or entity
> to
> > >> which it is addressed and may contain confidential and/or privileged
> > >> information. If you are not the intended recipient, please delete the
> > >> original message and any copy of it from your computer system. You are
> > >> hereby notified that any dissemination, distribution or copying of
> this
> > >> communication is strictly prohibited unless proper authorization has
> > been
> > >> obtained for such action. If you have received this communication in
> > >> error,
> > >> please notify the sender immediately. Although IndiQus attempts to
> sweep
> > >> e-mail and attachments for viruses, it does not guarantee that both
> are
> > >> virus-free and accepts no liability for any damage sustained as a
> result
> > >> of
> > >> viruses.
> > >>
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
>
> --
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by Simon Weller <sw...@ena.com.INVALID>.
I'm assuming the para-virtualized network driver is probably not being used either based on the template.
If you take a close look at the xml for the interface, you should be able to confirm that or not.
The templates tend to package all the para-virtualized drivers together, so they're normally in use, or they're not.
________________________________
From: S.Fuller <st...@gmail.com>
Sent: Friday, July 1, 2022 1:52 PM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: Re: Asymmetric traffic issues?
EXTERNAL EMAIL: This message originated outside of ENA. Use caution when clicking links, opening attachments, or complying with requests. Click the "Phish Alert Report" button above the email, or contact MIS, regarding any suspicious message.
Will give that a try. I'm curious why you think changing the disk drivers
over to virtio will have an effect on the network throughput.
- Steve
On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
wrote:
> Unfortunately, I can't open the xml via the mailing list, but based on the
> template you're using, it's not setup for Virtio-blk or virtio-scsi. Change
> the template to Virtio-SCSI 64bit and the correct interface drive type will
> be specified in the XML.
>
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Thursday, June 30, 2022 12:43 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
>
> I'm using the Windows Server 2016 profile for the template. After the VM
> came up, I installed the virtio drivers from the latest ISO, and then added
> a new NIC so that it used the virtio network drivers vs the default E1000
> drivers.
>
> XML dump is attached
>
> On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sw...@ena.com.invalid>
> wrote:
> Can you provide a redacted dump of your libvirt xml file for the VM?
> Also, which OS profile are you using in Cloudstack for this? The Windows
> specific ones don't use virtio if I remember correctly, so you will need to
> select Virtio-SCSI 64bit in order for the xml to be built correctly.
>
> -Si
> ________________________________
> From: S.Fuller <st...@gmail.com>>
> Sent: Thursday, June 30, 2022 11:38 AM
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Neither the block nor the scsi drivers appear to be running as far as I can
> tell.
>
> - Steve
>
> On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > Steve,
> >
> > Are you running the virtio-block or virtio-scsi drivers?
> >
> > -Si
> >
> >
> > ________________________________
> > From: S.Fuller <st...@gmail.com>>
> > Sent: Thursday, June 30, 2022 8:52 AM
> > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Well, I checked everything that I could for any QOS settings and there is
> > nothing configured there. What is curious is that Idon't see this
> behavior
> > when the transmitting host is running Linux, or if I'm using the E1000
> > drivers (although with the E1000 driver, the overall throughput is
> lower).
> > It really feels like I'mrunning into some weird issue with the virtio
> > drivers on Windows. My Windows hosts are (to my knowledge) using the
> latest
> > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> >
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
> > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com<mailto:
> stevefuller@gmail.com>> wrote:
> >
> > > Vivek,
> > >
> > > Thanks for the reply. I am using the KVM hypervisor. I'll will review
> the
> > > QoS on the hypervisor for both of the nodes.
> > >
> > > I'm checking throughput between two different VMs that running on two
> > > different hosts within the same cluster. As of right now, I'm receiving
> > > similar results using both iperf3 and nuttcp as the testing tools. We
> are
> > > only seeing this issue when the VM is not on the same host as the
> vrouter
> > > for its isolated network.
> > >
> > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > <vi...@indiqus.com>.invalid>
> wrote:
> > >
> > >> Hey Fuller,
> > >>
> > >> What hypervisor are you using ? I know you have checked all bandwidth
> > >> limit on templates and global settings, but it’s worth to check the
> QoS
> > on
> > >> the hypervisor level, because at the end it’s the hypervisor which
> > manages
> > >> all. And from where are you trying to check the network throughout,
> > >> between client and server ?
> > >>
> > >>
> > >> Vivek Kumar
> > >> Sr. Manager - Cloud & DevOps
> > >> TechOps | Indiqus Technologies
> > >>
> > >> + 91 7503460090 <tel:++91+7503460090>
> > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> <ma...@indiqus.com>>
> > >> www.indiqus.com<http://www.indiqus.com><
> http://www.indiqus.com> <
> > https://www.indiqus.com/>
> > >>
> > >>
> > >>
> > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com<mailto:
> stevefuller@gmail.com>> wrote:
> > >> >
> > >> > Environment:
> > >> >
> > >> > Two physical hosts
> > >> > - Cloudstack 4.11.3
> > >> > - Verified that there are no bandwidth limits in place on any of the
> > >> > templates or in global settings.
> > >> >
> > >> > Two isolated networks ("Client" and "Server")
> > >> > - Each has a vrouter with a public and private address
> > >> > - One Windows 2016 VM on each network (running the latest virtio
> > >> drivers)
> > >> > - each node running latest version of Iperf3 to test throughput
> > >> >
> > >> > Testing/Observation:
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> the
> > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> range,
> > >> > whether we run iperf in regular mode or in reverse mode (iperf -R).
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> > different physical hosts, we are seeing 25% of the throughput
> running
> > >> iperf
> > >> > in regular mode vs running it in reverse mode.
> > >> >
> > >> > Has anyone encountered this issue before? If we change the Client VM
> > to
> > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we see
> > >> > symmetrical throughput in our tests, no matter where the vrouter is
> in
> > >> > relation to the Client VM.
> > >> >
> > >> > --
> > >> > Steve Fuller
> > >> > stevefuller@gmail.com<ma...@gmail.com>
> > >>
> > >>
> > >> --
> > >> This message is intended only for the use of the individual or entity
> to
> > >> which it is addressed and may contain confidential and/or privileged
> > >> information. If you are not the intended recipient, please delete the
> > >> original message and any copy of it from your computer system. You are
> > >> hereby notified that any dissemination, distribution or copying of
> this
> > >> communication is strictly prohibited unless proper authorization has
> > been
> > >> obtained for such action. If you have received this communication in
> > >> error,
> > >> please notify the sender immediately. Although IndiQus attempts to
> sweep
> > >> e-mail and attachments for viruses, it does not guarantee that both
> are
> > >> virus-free and accepts no liability for any damage sustained as a
> result
> > >> of
> > >> viruses.
> > >>
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
>
> --
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
--
Steve Fuller
stevefuller@gmail.com
Re: Asymmetric traffic issues?
Posted by "S.Fuller" <st...@gmail.com>.
Will give that a try. I'm curious why you think changing the disk drivers
over to virtio will have an effect on the network throughput.
- Steve
On Fri, Jul 1, 2022 at 10:55 AM Simon Weller <sw...@ena.com.invalid>
wrote:
> Unfortunately, I can't open the xml via the mailing list, but based on the
> template you're using, it's not setup for Virtio-blk or virtio-scsi. Change
> the template to Virtio-SCSI 64bit and the correct interface drive type will
> be specified in the XML.
>
> ________________________________
> From: S.Fuller <st...@gmail.com>
> Sent: Thursday, June 30, 2022 12:43 PM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: Re: Asymmetric traffic issues?
>
>
> I'm using the Windows Server 2016 profile for the template. After the VM
> came up, I installed the virtio drivers from the latest ISO, and then added
> a new NIC so that it used the virtio network drivers vs the default E1000
> drivers.
>
> XML dump is attached
>
> On Thu, Jun 30, 2022 at 12:03 PM Simon Weller <sw...@ena.com.invalid>
> wrote:
> Can you provide a redacted dump of your libvirt xml file for the VM?
> Also, which OS profile are you using in Cloudstack for this? The Windows
> specific ones don't use virtio if I remember correctly, so you will need to
> select Virtio-SCSI 64bit in order for the xml to be built correctly.
>
> -Si
> ________________________________
> From: S.Fuller <st...@gmail.com>>
> Sent: Thursday, June 30, 2022 11:38 AM
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> Subject: Re: Asymmetric traffic issues?
>
> EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> clicking links, opening attachments, or complying with requests. Click the
> "Phish Alert Report" button above the email, or contact MIS, regarding any
> suspicious message.
>
>
>
> Neither the block nor the scsi drivers appear to be running as far as I can
> tell.
>
> - Steve
>
> On Thu, Jun 30, 2022 at 11:07 AM Simon Weller <sw...@ena.com.invalid>
> wrote:
>
> > Steve,
> >
> > Are you running the virtio-block or virtio-scsi drivers?
> >
> > -Si
> >
> >
> > ________________________________
> > From: S.Fuller <st...@gmail.com>>
> > Sent: Thursday, June 30, 2022 8:52 AM
> > To: users@cloudstack.apache.org<ma...@cloudstack.apache.org> <
> users@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> > Subject: Re: Asymmetric traffic issues?
> >
> > EXTERNAL EMAIL: This message originated outside of ENA. Use caution when
> > clicking links, opening attachments, or complying with requests. Click
> the
> > "Phish Alert Report" button above the email, or contact MIS, regarding
> any
> > suspicious message.
> >
> >
> >
> > Well, I checked everything that I could for any QOS settings and there is
> > nothing configured there. What is curious is that Idon't see this
> behavior
> > when the transmitting host is running Linux, or if I'm using the E1000
> > drivers (although with the E1000 driver, the overall throughput is
> lower).
> > It really feels like I'mrunning into some weird issue with the virtio
> > drivers on Windows. My Windows hosts are (to my knowledge) using the
> latest
> > version of the virtio drivers - 100.90.104.21700 dated 2/23/2022.
> >
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
> > On Wed, Jun 29, 2022 at 3:04 PM S.Fuller <stevefuller@gmail.com<mailto:
> stevefuller@gmail.com>> wrote:
> >
> > > Vivek,
> > >
> > > Thanks for the reply. I am using the KVM hypervisor. I'll will review
> the
> > > QoS on the hypervisor for both of the nodes.
> > >
> > > I'm checking throughput between two different VMs that running on two
> > > different hosts within the same cluster. As of right now, I'm receiving
> > > similar results using both iperf3 and nuttcp as the testing tools. We
> are
> > > only seeing this issue when the VM is not on the same host as the
> vrouter
> > > for its isolated network.
> > >
> > > On Tue, Jun 28, 2022 at 6:40 AM Vivek Kumar
> > > <vi...@indiqus.com>.invalid>
> wrote:
> > >
> > >> Hey Fuller,
> > >>
> > >> What hypervisor are you using ? I know you have checked all bandwidth
> > >> limit on templates and global settings, but it’s worth to check the
> QoS
> > on
> > >> the hypervisor level, because at the end it’s the hypervisor which
> > manages
> > >> all. And from where are you trying to check the network throughout,
> > >> between client and server ?
> > >>
> > >>
> > >> Vivek Kumar
> > >> Sr. Manager - Cloud & DevOps
> > >> TechOps | Indiqus Technologies
> > >>
> > >> + 91 7503460090 <tel:++91+7503460090>
> > >> vivek.kumar@indiqus.com<ma...@indiqus.com>
> <ma...@indiqus.com>>
> > >> www.indiqus.com<http://www.indiqus.com><
> http://www.indiqus.com> <
> > https://www.indiqus.com/>
> > >>
> > >>
> > >>
> > >> > On 28-Jun-2022, at 1:58 AM, S.Fuller <stevefuller@gmail.com<mailto:
> stevefuller@gmail.com>> wrote:
> > >> >
> > >> > Environment:
> > >> >
> > >> > Two physical hosts
> > >> > - Cloudstack 4.11.3
> > >> > - Verified that there are no bandwidth limits in place on any of the
> > >> > templates or in global settings.
> > >> >
> > >> > Two isolated networks ("Client" and "Server")
> > >> > - Each has a vrouter with a public and private address
> > >> > - One Windows 2016 VM on each network (running the latest virtio
> > >> drivers)
> > >> > - each node running latest version of Iperf3 to test throughput
> > >> >
> > >> > Testing/Observation:
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> the
> > >> > same physical host, we see symmetrical throughput in the 2 Gbps
> range,
> > >> > whether we run iperf in regular mode or in reverse mode (iperf -R).
> > >> >
> > >> > If the Client VM and the vrouter for the isolated Client network are
> > on
> > >> > different physical hosts, we are seeing 25% of the throughput
> running
> > >> iperf
> > >> > in regular mode vs running it in reverse mode.
> > >> >
> > >> > Has anyone encountered this issue before? If we change the Client VM
> > to
> > >> > Linux (either CentOS 7 or Ubuntu) OR we use the E1000 driver, we see
> > >> > symmetrical throughput in our tests, no matter where the vrouter is
> in
> > >> > relation to the Client VM.
> > >> >
> > >> > --
> > >> > Steve Fuller
> > >> > stevefuller@gmail.com<ma...@gmail.com>
> > >>
> > >>
> > >> --
> > >> This message is intended only for the use of the individual or entity
> to
> > >> which it is addressed and may contain confidential and/or privileged
> > >> information. If you are not the intended recipient, please delete the
> > >> original message and any copy of it from your computer system. You are
> > >> hereby notified that any dissemination, distribution or copying of
> this
> > >> communication is strictly prohibited unless proper authorization has
> > been
> > >> obtained for such action. If you have received this communication in
> > >> error,
> > >> please notify the sender immediately. Although IndiQus attempts to
> sweep
> > >> e-mail and attachments for viruses, it does not guarantee that both
> are
> > >> virus-free and accepts no liability for any damage sustained as a
> result
> > >> of
> > >> viruses.
> > >>
> > >
> > >
> > > --
> > > Steve Fuller
> > > stevefuller@gmail.com<ma...@gmail.com>
> > >
> >
> >
> > --
> > Steve Fuller
> > stevefuller@gmail.com<ma...@gmail.com>
> >
>
>
> --
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
>
> --
> Steve Fuller
> stevefuller@gmail.com<ma...@gmail.com>
>
--
Steve Fuller
stevefuller@gmail.com