You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Toshiaki Hatano <to...@verio.net> on 2013/05/01 22:58:12 UTC
[DISCUSS] Linux native VXLAN support on KVM hypervisor
Hi all,
I’d like to add Linux native VXLAN support on KVM hypervisor.
Currently, advanced zone with VLAN isolation can hold only 4k networks (= accounts) in a zone due to the VLAN ID limitation.
4k accounts per zone is not enough for IaaS provider like us.
Furthermore, VPC will allow single account to consume multiple networks.
Linux kernel 3.7 or later supports VXLAN as part of its ordinal networking function.
VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation and allow 16M isolated networks in the domain.
So, by using linux native VXLAN support, we can extend network limits without introducing unnecessary complexity.
(But in other words, it’s not as flexible as Open vSwitch. Only thing Linux native VXLAN provides is multipoint L2 tunneling.)
Any thoughts about this?
P.S.
I’m currently working on this as my internship project.
As proof of concept, I’ve modified “modifyvlan.sh” script which is actual VLAN create/delete manipulation script called from cloud-agent, to create and to use VXLAN interface instead of VLAN interface.
Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors based on CentOS 6.4 + 3.8.6 kernel.
And it looks working. (But I’m still testing)
P.S.2.
FYI: OpenStack already started process [1] to support Linux native VXLAN.
[1] https://review.openstack.org/#/c/26516/
Best Regards,
--
Toshiaki Hatano
Verio, an NTT Communications company
E-mail: toshiaki.hatano@verio.net
AIM: toshiaki.hatano@verio.net
Phone: (801)437-7482 Office
(801)960-6410 Cellular
This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio Inc. makes no warranty that this email is error or virus free. Thank you.
RE: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Toshiaki Hatano <to...@verio.net>.
Sorry, I forgot to add users list in To.
Thanks,
--
Toshiaki
> -----Original Message-----
> From: Toshiaki Hatano [mailto:toshiaki.hatano@verio.net]
> Sent: Wednesday, May 01, 2013 2:58 PM
> To: dev@cloudstack.apache.org
> Subject: [DISCUSS] Linux native VXLAN support on KVM hypervisor
>
> Hi all,
>
> I’d like to add Linux native VXLAN support on KVM hypervisor.
>
> Currently, advanced zone with VLAN isolation can hold only 4k networks (=
> accounts) in a zone due to the VLAN ID limitation.
> 4k accounts per zone is not enough for IaaS provider like us.
> Furthermore, VPC will allow single account to consume multiple networks.
>
> Linux kernel 3.7 or later supports VXLAN as part of its ordinal networking
> function.
> VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation
> and allow 16M isolated networks in the domain.
> So, by using linux native VXLAN support, we can extend network limits without
> introducing unnecessary complexity.
> (But in other words, it’s not as flexible as Open vSwitch. Only thing Linux
> native VXLAN provides is multipoint L2 tunneling.)
>
> Any thoughts about this?
>
>
> P.S.
>
> I’m currently working on this as my internship project.
> As proof of concept, I’ve modified “modifyvlan.sh” script which is actual
> VLAN create/delete manipulation script called from cloud-agent, to create
> and to use VXLAN interface instead of VLAN interface.
> Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors based
> on CentOS 6.4 + 3.8.6 kernel.
> And it looks working. (But I’m still testing)
>
>
> P.S.2.
>
> FYI: OpenStack already started process [1] to support Linux native VXLAN.
> [1] https://review.openstack.org/#/c/26516/
>
>
> Best Regards,
> --
> Toshiaki Hatano
> Verio, an NTT Communications company
> E-mail: toshiaki.hatano@verio.net
> AIM: toshiaki.hatano@verio.net
> Phone: (801)437-7482 Office
> (801)960-6410 Cellular
>
>
> This email message is intended for the use of the person to whom it has
> been sent, and may contain information that is confidential or legally
> protected. If you are not the intended recipient or have received this
> message in error, you are not authorized to copy, distribute, or otherwise
> use this message or its attachments. Please notify the sender immediately
> by return e-mail and permanently delete this message and any attachments.
> Verio Inc. makes no warranty that this email is error or virus free. Thank
> you.
This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio Inc. makes no warranty that this email is error or virus free. Thank you.
RE: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Toshiaki Hatano <to...@verio.net>.
Sorry, I forgot to add users list in To.
Thanks,
--
Toshiaki
> -----Original Message-----
> From: Toshiaki Hatano [mailto:toshiaki.hatano@verio.net]
> Sent: Wednesday, May 01, 2013 2:58 PM
> To: dev@cloudstack.apache.org
> Subject: [DISCUSS] Linux native VXLAN support on KVM hypervisor
>
> Hi all,
>
> I’d like to add Linux native VXLAN support on KVM hypervisor.
>
> Currently, advanced zone with VLAN isolation can hold only 4k networks (=
> accounts) in a zone due to the VLAN ID limitation.
> 4k accounts per zone is not enough for IaaS provider like us.
> Furthermore, VPC will allow single account to consume multiple networks.
>
> Linux kernel 3.7 or later supports VXLAN as part of its ordinal networking
> function.
> VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation
> and allow 16M isolated networks in the domain.
> So, by using linux native VXLAN support, we can extend network limits without
> introducing unnecessary complexity.
> (But in other words, it’s not as flexible as Open vSwitch. Only thing Linux
> native VXLAN provides is multipoint L2 tunneling.)
>
> Any thoughts about this?
>
>
> P.S.
>
> I’m currently working on this as my internship project.
> As proof of concept, I’ve modified “modifyvlan.sh” script which is actual
> VLAN create/delete manipulation script called from cloud-agent, to create
> and to use VXLAN interface instead of VLAN interface.
> Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors based
> on CentOS 6.4 + 3.8.6 kernel.
> And it looks working. (But I’m still testing)
>
>
> P.S.2.
>
> FYI: OpenStack already started process [1] to support Linux native VXLAN.
> [1] https://review.openstack.org/#/c/26516/
>
>
> Best Regards,
> --
> Toshiaki Hatano
> Verio, an NTT Communications company
> E-mail: toshiaki.hatano@verio.net
> AIM: toshiaki.hatano@verio.net
> Phone: (801)437-7482 Office
> (801)960-6410 Cellular
>
>
> This email message is intended for the use of the person to whom it has
> been sent, and may contain information that is confidential or legally
> protected. If you are not the intended recipient or have received this
> message in error, you are not authorized to copy, distribute, or otherwise
> use this message or its attachments. Please notify the sender immediately
> by return e-mail and permanently delete this message and any attachments.
> Verio Inc. makes no warranty that this email is error or virus free. Thank
> you.
This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio Inc. makes no warranty that this email is error or virus free. Thank you.
RE: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Toshiaki Hatano <to...@verio.net>.
Thank you all for your comment and approval!
I agree with Justin's proposal.
And yesterday, Open vSwitch released v1.10.0 with VXLAN support.
http://openvswitch.org/pipermail/announce/2013-May/000052.html
I've created Jira ticket and start writing design doc.
https://issues.apache.org/jira/browse/CLOUDSTACK-2328
Once I've done design doc, I'll submit it to dev list.
Sincerely,
--
Toshiaki
> -----Original Message-----
> From: Justin Grudzien [mailto:grudzien@gmail.com]
> Sent: Thursday, May 02, 2013 5:35 PM
> To: dev@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
>
> I will +1 this. I spoke with Cisco a few weeks ago and they certainly see
> VXLANS as being the future for cloud infrastructures. In addition to Linux
> support we should look at the Cisco 1000v and open vSwitch support a well.
> Cisco said they already have VMWare support today on the 1000v with KVM
> coming soon.
>
> Justin
>
> Sent from my iPhone
>
> On May 2, 2013, at 3:31 PM, Chip Childers <ch...@sungard.com> wrote:
>
> > On Wed, May 01, 2013 at 04:58:12PM -0400, Toshiaki Hatano wrote:
> >> Hi all,
> >>
> >> I’d like to add Linux native VXLAN support on KVM hypervisor.
> >>
> >> Currently, advanced zone with VLAN isolation can hold only 4k networks
> (= accounts) in a zone due to the VLAN ID limitation.
> >> 4k accounts per zone is not enough for IaaS provider like us.
> >> Furthermore, VPC will allow single account to consume multiple networks.
> >>
> >> Linux kernel 3.7 or later supports VXLAN as part of its ordinal networking
> function.
> >> VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation
> and allow 16M isolated networks in the domain.
> >> So, by using linux native VXLAN support, we can extend network limits
> without introducing unnecessary complexity.
> >> (But in other words, it’s not as flexible as Open vSwitch. Only thing
> >> Linux native VXLAN provides is multipoint L2 tunneling.)
> >>
> >> Any thoughts about this?
> >>
> >>
> >> P.S.
> >>
> >> I’m currently working on this as my internship project.
> >> As proof of concept, I’ve modified “modifyvlan.sh” script which is actual
> VLAN create/delete manipulation script called from cloud-agent, to create
> and to use VXLAN interface instead of VLAN interface.
> >> Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors
> based on CentOS 6.4 + 3.8.6 kernel.
> >> And it looks working. (But I’m still testing)
> >>
> >>
> >> P.S.2.
> >>
> >> FYI: OpenStack already started process [1] to support Linux native VXLAN.
> >> [1] https://review.openstack.org/#/c/26516/
> >>
> >>
> >> Best Regards,
> >> --
> >> Toshiaki Hatano
> >
> > I note that no one has replied to this thread yet, but I'll give you
> > my general +1 on the idea.
> >
> > Can some of the network-centric folks on the dev list please speak up
> > on the proposal?
> >
> > -chip
This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio Inc. makes no warranty that this email is error or virus free. Thank you.
Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by David Nalley <da...@gnsa.us>.
On Thu, May 2, 2013 at 7:34 PM, Justin Grudzien <gr...@gmail.com> wrote:
> I will +1 this. I spoke with Cisco a few weeks ago and they certainly see VXLANS as being the future for cloud infrastructures. In addition to Linux support we should look at the Cisco 1000v and open vSwitch support a well. Cisco said they already have VMWare support today on the 1000v with KVM coming soon.
>
I think we have 1000v support already in master for VMware
--David
RE: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Toshiaki Hatano <to...@verio.net>.
Thank you all for your comment and approval!
I agree with Justin's proposal.
And yesterday, Open vSwitch released v1.10.0 with VXLAN support.
http://openvswitch.org/pipermail/announce/2013-May/000052.html
I've created Jira ticket and start writing design doc.
https://issues.apache.org/jira/browse/CLOUDSTACK-2328
Once I've done design doc, I'll submit it to dev list.
Sincerely,
--
Toshiaki
> -----Original Message-----
> From: Justin Grudzien [mailto:grudzien@gmail.com]
> Sent: Thursday, May 02, 2013 5:35 PM
> To: dev@cloudstack.apache.org
> Cc: dev@cloudstack.apache.org
> Subject: Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
>
> I will +1 this. I spoke with Cisco a few weeks ago and they certainly see
> VXLANS as being the future for cloud infrastructures. In addition to Linux
> support we should look at the Cisco 1000v and open vSwitch support a well.
> Cisco said they already have VMWare support today on the 1000v with KVM
> coming soon.
>
> Justin
>
> Sent from my iPhone
>
> On May 2, 2013, at 3:31 PM, Chip Childers <ch...@sungard.com> wrote:
>
> > On Wed, May 01, 2013 at 04:58:12PM -0400, Toshiaki Hatano wrote:
> >> Hi all,
> >>
> >> I’d like to add Linux native VXLAN support on KVM hypervisor.
> >>
> >> Currently, advanced zone with VLAN isolation can hold only 4k networks
> (= accounts) in a zone due to the VLAN ID limitation.
> >> 4k accounts per zone is not enough for IaaS provider like us.
> >> Furthermore, VPC will allow single account to consume multiple networks.
> >>
> >> Linux kernel 3.7 or later supports VXLAN as part of its ordinal networking
> function.
> >> VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation
> and allow 16M isolated networks in the domain.
> >> So, by using linux native VXLAN support, we can extend network limits
> without introducing unnecessary complexity.
> >> (But in other words, it’s not as flexible as Open vSwitch. Only thing
> >> Linux native VXLAN provides is multipoint L2 tunneling.)
> >>
> >> Any thoughts about this?
> >>
> >>
> >> P.S.
> >>
> >> I’m currently working on this as my internship project.
> >> As proof of concept, I’ve modified “modifyvlan.sh” script which is actual
> VLAN create/delete manipulation script called from cloud-agent, to create
> and to use VXLAN interface instead of VLAN interface.
> >> Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors
> based on CentOS 6.4 + 3.8.6 kernel.
> >> And it looks working. (But I’m still testing)
> >>
> >>
> >> P.S.2.
> >>
> >> FYI: OpenStack already started process [1] to support Linux native VXLAN.
> >> [1] https://review.openstack.org/#/c/26516/
> >>
> >>
> >> Best Regards,
> >> --
> >> Toshiaki Hatano
> >
> > I note that no one has replied to this thread yet, but I'll give you
> > my general +1 on the idea.
> >
> > Can some of the network-centric folks on the dev list please speak up
> > on the proposal?
> >
> > -chip
This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio Inc. makes no warranty that this email is error or virus free. Thank you.
Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Justin Grudzien <gr...@gmail.com>.
I will +1 this. I spoke with Cisco a few weeks ago and they certainly see VXLANS as being the future for cloud infrastructures. In addition to Linux support we should look at the Cisco 1000v and open vSwitch support a well. Cisco said they already have VMWare support today on the 1000v with KVM coming soon.
Justin
Sent from my iPhone
On May 2, 2013, at 3:31 PM, Chip Childers <ch...@sungard.com> wrote:
> On Wed, May 01, 2013 at 04:58:12PM -0400, Toshiaki Hatano wrote:
>> Hi all,
>>
>> I’d like to add Linux native VXLAN support on KVM hypervisor.
>>
>> Currently, advanced zone with VLAN isolation can hold only 4k networks (= accounts) in a zone due to the VLAN ID limitation.
>> 4k accounts per zone is not enough for IaaS provider like us.
>> Furthermore, VPC will allow single account to consume multiple networks.
>>
>> Linux kernel 3.7 or later supports VXLAN as part of its ordinal networking function.
>> VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation and allow 16M isolated networks in the domain.
>> So, by using linux native VXLAN support, we can extend network limits without introducing unnecessary complexity.
>> (But in other words, it’s not as flexible as Open vSwitch. Only thing Linux native VXLAN provides is multipoint L2 tunneling.)
>>
>> Any thoughts about this?
>>
>>
>> P.S.
>>
>> I’m currently working on this as my internship project.
>> As proof of concept, I’ve modified “modifyvlan.sh” script which is actual VLAN create/delete manipulation script called from cloud-agent, to create and to use VXLAN interface instead of VLAN interface.
>> Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors based on CentOS 6.4 + 3.8.6 kernel.
>> And it looks working. (But I’m still testing)
>>
>>
>> P.S.2.
>>
>> FYI: OpenStack already started process [1] to support Linux native VXLAN.
>> [1] https://review.openstack.org/#/c/26516/
>>
>>
>> Best Regards,
>> --
>> Toshiaki Hatano
>
> I note that no one has replied to this thread yet, but I'll give you my
> general +1 on the idea.
>
> Can some of the network-centric folks on the dev list please speak up on
> the proposal?
>
> -chip
Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Marcus Sorensen <sh...@gmail.com>.
Agreed. This will give us four options for >4k isolation.
Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Ahmad Emneina <ae...@gmail.com>.
+1 the more isolation methods, the better.
On Thu, May 2, 2013 at 1:31 PM, Chip Childers <ch...@sungard.com>wrote:
> On Wed, May 01, 2013 at 04:58:12PM -0400, Toshiaki Hatano wrote:
> > Hi all,
> >
> > I’d like to add Linux native VXLAN support on KVM hypervisor.
> >
> > Currently, advanced zone with VLAN isolation can hold only 4k networks
> (= accounts) in a zone due to the VLAN ID limitation.
> > 4k accounts per zone is not enough for IaaS provider like us.
> > Furthermore, VPC will allow single account to consume multiple networks.
> >
> > Linux kernel 3.7 or later supports VXLAN as part of its ordinal
> networking function.
> > VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation
> and allow 16M isolated networks in the domain.
> > So, by using linux native VXLAN support, we can extend network limits
> without introducing unnecessary complexity.
> > (But in other words, it’s not as flexible as Open vSwitch. Only thing
> Linux native VXLAN provides is multipoint L2 tunneling.)
> >
> > Any thoughts about this?
> >
> >
> > P.S.
> >
> > I’m currently working on this as my internship project.
> > As proof of concept, I’ve modified “modifyvlan.sh” script which is
> actual VLAN create/delete manipulation script called from cloud-agent, to
> create and to use VXLAN interface instead of VLAN interface.
> > Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors
> based on CentOS 6.4 + 3.8.6 kernel.
> > And it looks working. (But I’m still testing)
> >
> >
> > P.S.2.
> >
> > FYI: OpenStack already started process [1] to support Linux native VXLAN.
> > [1] https://review.openstack.org/#/c/26516/
> >
> >
> > Best Regards,
> > --
> > Toshiaki Hatano
>
> I note that no one has replied to this thread yet, but I'll give you my
> general +1 on the idea.
>
> Can some of the network-centric folks on the dev list please speak up on
> the proposal?
>
> -chip
>
Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Chip Childers <ch...@sungard.com>.
On Wed, May 01, 2013 at 04:58:12PM -0400, Toshiaki Hatano wrote:
> Hi all,
>
> I’d like to add Linux native VXLAN support on KVM hypervisor.
>
> Currently, advanced zone with VLAN isolation can hold only 4k networks (= accounts) in a zone due to the VLAN ID limitation.
> 4k accounts per zone is not enough for IaaS provider like us.
> Furthermore, VPC will allow single account to consume multiple networks.
>
> Linux kernel 3.7 or later supports VXLAN as part of its ordinal networking function.
> VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation and allow 16M isolated networks in the domain.
> So, by using linux native VXLAN support, we can extend network limits without introducing unnecessary complexity.
> (But in other words, it’s not as flexible as Open vSwitch. Only thing Linux native VXLAN provides is multipoint L2 tunneling.)
>
> Any thoughts about this?
>
>
> P.S.
>
> I’m currently working on this as my internship project.
> As proof of concept, I’ve modified “modifyvlan.sh” script which is actual VLAN create/delete manipulation script called from cloud-agent, to create and to use VXLAN interface instead of VLAN interface.
> Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors based on CentOS 6.4 + 3.8.6 kernel.
> And it looks working. (But I’m still testing)
>
>
> P.S.2.
>
> FYI: OpenStack already started process [1] to support Linux native VXLAN.
> [1] https://review.openstack.org/#/c/26516/
>
>
> Best Regards,
> --
> Toshiaki Hatano
I note that no one has replied to this thread yet, but I'll give you my
general +1 on the idea.
Can some of the network-centric folks on the dev list please speak up on
the proposal?
-chip
Re: [DISCUSS] Linux native VXLAN support on KVM hypervisor
Posted by Chiradeep Vittal <Ch...@citrix.com>.
+1.
In général you'd have to do much more than 'modifyvlan.sh'. You should
take a look at the Nicira (and other) integrations.
On 5/1/13 1:58 PM, "Toshiaki Hatano" <to...@verio.net> wrote:
>Hi all,
>
>I¹d like to add Linux native VXLAN support on KVM hypervisor.
>
>Currently, advanced zone with VLAN isolation can hold only 4k networks (=
>accounts) in a zone due to the VLAN ID limitation.
>4k accounts per zone is not enough for IaaS provider like us.
>Furthermore, VPC will allow single account to consume multiple networks.
>
>Linux kernel 3.7 or later supports VXLAN as part of its ordinal
>networking function.
>VXLAN enable Layer 2 tunneling over UDP/IP with VLAN-like encapsulation
>and allow 16M isolated networks in the domain.
>So, by using linux native VXLAN support, we can extend network limits
>without introducing unnecessary complexity.
>(But in other words, it¹s not as flexible as Open vSwitch. Only thing
>Linux native VXLAN provides is multipoint L2 tunneling.)
>
>Any thoughts about this?
>
>
>P.S.
>
>I¹m currently working on this as my internship project.
>As proof of concept, I¹ve modified ³modifyvlan.sh² script which is actual
>VLAN create/delete manipulation script called from cloud-agent, to create
>and to use VXLAN interface instead of VLAN interface.
>Modified script is tested with CloudStack 4.0.1 and 3 KVM hypervisors
>based on CentOS 6.4 + 3.8.6 kernel.
>And it looks working. (But I¹m still testing)
>
>
>P.S.2.
>
>FYI: OpenStack already started process [1] to support Linux native VXLAN.
>[1] https://review.openstack.org/#/c/26516/
>
>
>Best Regards,
>--
>Toshiaki Hatano
>Verio, an NTT Communications company
>E-mail: toshiaki.hatano@verio.net
>AIM: toshiaki.hatano@verio.net
>Phone: (801)437-7482 Office
> (801)960-6410 Cellular
>
>
>This email message is intended for the use of the person to whom it has
>been sent, and may contain information that is confidential or legally
>protected. If you are not the intended recipient or have received this
>message in error, you are not authorized to copy, distribute, or
>otherwise use this message or its attachments. Please notify the sender
>immediately by return e-mail and permanently delete this message and any
>attachments. Verio Inc. makes no warranty that this email is error or
>virus free. Thank you.