You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Alexandru Stan <al...@lifeincloud.com.INVALID> on 2024/10/01 09:24:46 UTC
Public IP on instances
Hi everyone,
I have a specific scenario with ACS that I'm not sure how to approach, maybe someone here can share a solution/workaround. As far as I know, the only way to have a public ip directly assigned to a vm (I mean on the vm's network adapter) is to use a shared network. But in this case all users would share one router and I wouldn't be able to track network usage individually, correct? Is there any other way to do this AND have traffic usage at vm/user level? Creating multiple shared networks is not an option, it would require constant monitoring of the routers to keep track of ip usage and so on.
Thank you!
AW: Public IP on instances
Posted by me...@swen.io.
Hi Alex,
I guess you are using advanced network in CS, correct? As far as I know in
basic network setup all instances have a public ip attached to the nic
directly. We do traffic monitoring directly at router level via netflow to
get better values. I am not sure if CS is able to track traffic in basic
network, because we only use advanced network.
Regards,
Swen
-----Ursprüngliche Nachricht-----
Von: Alexandru Stan <al...@lifeincloud.com.INVALID>
Gesendet: Dienstag, 1. Oktober 2024 11:25
An: users@cloudstack.apache.org
Betreff: Public IP on instances
Hi everyone,
I have a specific scenario with ACS that I'm not sure how to approach, maybe
someone here can share a solution/workaround. As far as I know, the only way
to have a public ip directly assigned to a vm (I mean on the vm's network
adapter) is to use a shared network. But in this case all users would share
one router and I wouldn't be able to track network usage individually,
correct? Is there any other way to do this AND have traffic usage at vm/user
level? Creating multiple shared networks is not an option, it would require
constant monitoring of the routers to keep track of ip usage and so on.
Thank you!
Re: Public IP on instances
Posted by Nux <nu...@li.nux.ro>.
Forgot to link the pr:
https://github.com/apache/cloudstack/pull/9415/
On 2024-10-01 14:59, Nux wrote:
> That's a good approach, one can also use Prometheus with the libvirt
> exporter and can get per-vm & per-NIC stats that way.
>
> Worth mentioning that 4.20 will bring Security Groups in Shared
> Networks (in regular Advanced Zones!), thus simplifying somewhat
> people's lives in similar situations, as far as IPv4 usage is
> concerned, you can more easily share a subnet with many customers, not
> having to worry they'll spoof addresses etc.
>
> On 2024-10-01 13:39, Stephan Bienek wrote:
>> We are using L2 networks with for example a /29 public subnet for the
>> customer network as well, as Lucian mentioned it.
>>
>> This approach and quite a few other use cases i will share during my
>> talk at CCC 2024 in Madrid.
>>
>> As an alternative to the L2 /29 approach, which is not the most
>> efficient approach if you only need a single IP, you could use a
>> Shared Network with specify VLAN, without VR and tell the customer
>> which single IP to use.
>> In order to make sure no customer is using "wrong" IPs, be sure to use
>> the approach Alex Mattioli mentioned once - creating fixed MAC-IP
>> entries on your routers.
>>
>> We combine the L2 network approach with what Swen mentioned,
>> collecting netflow data from routers via open source "pmacct" for
>> traffic accounting.
>> This could help to get per-customer (or per-IP) accounting data even
>> when using one shared VR.
>>
>> Best regards,
>> Stephan
>>
>>> Wei ZHOU <us...@gmail.com> hat am 01.10.2024 14:11 CEST
>>> geschrieben:
>>>
>>>
>>> +1 with what Lucian said.
>>>
>>> Please update the value of global setting "vm.network.stats.interval"
>>> (by
>>> default 0), and restart mgmt server.
>>> then you can get network statistics of each nic on shared networks.
>>>
>>> -Wei
>>>
>>> On Tue, Oct 1, 2024 at 1:47 PM Nux <nu...@li.nux.ro> wrote:
>>>
>>> > I thought the traffic usage is taken from the hypervisor, for the VM's
>>> > NIC.
>>> > Btw, you can also use L2 networks, may be more flexible and economic
>>> > (with IPv4 usage).
>>> >
>>> > On 2024-10-01 10:24, Alexandru Stan wrote:
>>> > > Hi everyone,
>>> > >
>>> > > I have a specific scenario with ACS that I'm not sure how to approach,
>>> > > maybe someone here can share a solution/workaround. As far as I know,
>>> > > the only way to have a public ip directly assigned to a vm (I mean on
>>> > > the vm's network adapter) is to use a shared network. But in this case
>>> > > all users would share one router and I wouldn't be able to track
>>> > > network usage individually, correct? Is there any other way to do this
>>> > > AND have traffic usage at vm/user level? Creating multiple shared
>>> > > networks is not an option, it would require constant monitoring of the
>>> > > routers to keep track of ip usage and so on.
>>> > >
>>> > > Thank you!
>>> >
Re: Public IP on instances
Posted by Nux <nu...@li.nux.ro>.
That's a good approach, one can also use Prometheus with the libvirt
exporter and can get per-vm & per-NIC stats that way.
Worth mentioning that 4.20 will bring Security Groups in Shared Networks
(in regular Advanced Zones!), thus simplifying somewhat people's lives
in similar situations, as far as IPv4 usage is concerned, you can more
easily share a subnet with many customers, not having to worry they'll
spoof addresses etc.
On 2024-10-01 13:39, Stephan Bienek wrote:
> We are using L2 networks with for example a /29 public subnet for the
> customer network as well, as Lucian mentioned it.
>
> This approach and quite a few other use cases i will share during my
> talk at CCC 2024 in Madrid.
>
> As an alternative to the L2 /29 approach, which is not the most
> efficient approach if you only need a single IP, you could use a Shared
> Network with specify VLAN, without VR and tell the customer which
> single IP to use.
> In order to make sure no customer is using "wrong" IPs, be sure to use
> the approach Alex Mattioli mentioned once - creating fixed MAC-IP
> entries on your routers.
>
> We combine the L2 network approach with what Swen mentioned, collecting
> netflow data from routers via open source "pmacct" for traffic
> accounting.
> This could help to get per-customer (or per-IP) accounting data even
> when using one shared VR.
>
> Best regards,
> Stephan
>
>> Wei ZHOU <us...@gmail.com> hat am 01.10.2024 14:11 CEST
>> geschrieben:
>>
>>
>> +1 with what Lucian said.
>>
>> Please update the value of global setting "vm.network.stats.interval"
>> (by
>> default 0), and restart mgmt server.
>> then you can get network statistics of each nic on shared networks.
>>
>> -Wei
>>
>> On Tue, Oct 1, 2024 at 1:47 PM Nux <nu...@li.nux.ro> wrote:
>>
>> > I thought the traffic usage is taken from the hypervisor, for the VM's
>> > NIC.
>> > Btw, you can also use L2 networks, may be more flexible and economic
>> > (with IPv4 usage).
>> >
>> > On 2024-10-01 10:24, Alexandru Stan wrote:
>> > > Hi everyone,
>> > >
>> > > I have a specific scenario with ACS that I'm not sure how to approach,
>> > > maybe someone here can share a solution/workaround. As far as I know,
>> > > the only way to have a public ip directly assigned to a vm (I mean on
>> > > the vm's network adapter) is to use a shared network. But in this case
>> > > all users would share one router and I wouldn't be able to track
>> > > network usage individually, correct? Is there any other way to do this
>> > > AND have traffic usage at vm/user level? Creating multiple shared
>> > > networks is not an option, it would require constant monitoring of the
>> > > routers to keep track of ip usage and so on.
>> > >
>> > > Thank you!
>> >
Re: Public IP on instances
Posted by Wei ZHOU <us...@gmail.com>.
Hi Marty,
You can get more information of ROUTED mode on
https://docs.cloudstack.apache.org/en/latest/adminguide/networking/dynamic_static_routing.html
-Wei
On Wed, Oct 2, 2024 at 4:49 PM Marty Godsey <ma...@rudio.net> wrote:
> Alex,
>
> So does this mean that the customer will need their own public subnet
> space in the public-public scenario if the VR is just acting as a router?
> How does this work with automatic allocation like it does now?
>
> Regards,
> Marty Godsey
>
>
> From: Alex Mattioli <Al...@shapeblue.com>
> Date: Tuesday, October 1, 2024 at 11:28 AM
> To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
> Subject: RE: Public IP on instances
> WARNING: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
> Building on what Stephan said.
>
> In 4.20 there will be a new feature called "ROUTED mode", in which the ACS
> VR instead of NATting simply routes between two networks, which can be
> private-private, public-private, private-public, or public-public, the last
> two allowing for your end user VMs to have public IPs without the need of
> Operator intervention, also in 4.20 there will be support for BGP in the
> ACS VR, which means that the subnets (Public or otherwise) used by end
> users in Routed mode are automatically advertised upstream.
>
> Either way, be with static or dynamic routing, with Routed mode you'll
> have full control of the public IPs assigned.
>
> There will be a session by Wei Zhoue showcasing this new mode in the next
> CCC on Friday 22nd of November at 11:30am (CEST):
> https://atpscan.global.hornetsecurity.com?d=2_L5ASGiZkQUy3WIwyuwS7ZEKxepbehOc5_5hXyBHbo&f=11FL9O81QOaDecjEcRW18Bc0HDwHHudUjgQm2sZqr9gJbPweH0Shmsafosz_FMYI&i=&k=6QIh&m=xESLIvTl3JcJbFIqHFyvGNC1cUa3dk3JTm5rBb5GnLQJ3vj32DnmC4tQB2MBumH3KUBKIgO_92Wciricec2_QmRLcGT_eCOLZ0_pkooYey0r2M9ujzushgA9RReAifUB&n=-M8F8az2r7mgeJh5YU7MrhqAibcsRyS0M1O0dRutOk5rut1Q8AAkzkeebWTZrnp-&r=EHCXAcWOc9fQZ2opSXDWb5CMVWfKXQzbrDBydD8iFiT8SlLV64-xS7HzpWvCaIqS&s=4abe87433b9f7e516be6a66312b0cfd1832f52c7168732bffd6af7b44d7107b8&u=https%3A%2F%2Fwww.cloudstackcollab.org%2F
>
> Cheers,
> Alex
>
>
>
>
> -----Original Message-----
> From: Stephan Bienek <st...@bienek.org>
> Sent: 01 October 2024 14:39
> To: users@cloudstack.apache.org
> Subject: Re: Public IP on instances
>
> We are using L2 networks with for example a /29 public subnet for the
> customer network as well, as Lucian mentioned it.
>
> This approach and quite a few other use cases i will share during my talk
> at CCC 2024 in Madrid.
>
> As an alternative to the L2 /29 approach, which is not the most efficient
> approach if you only need a single IP, you could use a Shared Network with
> specify VLAN, without VR and tell the customer which single IP to use.
> In order to make sure no customer is using "wrong" IPs, be sure to use the
> approach Alex Mattioli mentioned once - creating fixed MAC-IP entries on
> your routers.
>
> We combine the L2 network approach with what Swen mentioned, collecting
> netflow data from routers via open source "pmacct" for traffic accounting.
> This could help to get per-customer (or per-IP) accounting data even when
> using one shared VR.
>
> Best regards,
> Stephan
>
> > Wei ZHOU <us...@gmail.com> hat am 01.10.2024 14:11 CEST
> geschrieben:
> >
> >
> > +1 with what Lucian said.
> >
> > Please update the value of global setting "vm.network.stats.interval"
> > (by default 0), and restart mgmt server.
> > then you can get network statistics of each nic on shared networks.
> >
> > -Wei
> >
> > On Tue, Oct 1, 2024 at 1:47 PM Nux <nu...@li.nux.ro> wrote:
> >
> > > I thought the traffic usage is taken from the hypervisor, for the
> > > VM's NIC.
> > > Btw, you can also use L2 networks, may be more flexible and economic
> > > (with IPv4 usage).
> > >
> > > On 2024-10-01 10:24, Alexandru Stan wrote:
> > > > Hi everyone,
> > > >
> > > > I have a specific scenario with ACS that I'm not sure how to
> > > > approach, maybe someone here can share a solution/workaround. As
> > > > far as I know, the only way to have a public ip directly assigned
> > > > to a vm (I mean on the vm's network adapter) is to use a shared
> > > > network. But in this case all users would share one router and I
> > > > wouldn't be able to track network usage individually, correct? Is
> > > > there any other way to do this AND have traffic usage at vm/user
> > > > level? Creating multiple shared networks is not an option, it
> > > > would require constant monitoring of the routers to keep track of ip
> usage and so on.
> > > >
> > > > Thank you!
> > >
>
Re: Public IP on instances
Posted by Marty Godsey <ma...@rudio.net>.
Alex,
So does this mean that the customer will need their own public subnet space in the public-public scenario if the VR is just acting as a router? How does this work with automatic allocation like it does now?
Regards,
Marty Godsey
From: Alex Mattioli <Al...@shapeblue.com>
Date: Tuesday, October 1, 2024 at 11:28 AM
To: users@cloudstack.apache.org <us...@cloudstack.apache.org>
Subject: RE: Public IP on instances
WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
Building on what Stephan said.
In 4.20 there will be a new feature called "ROUTED mode", in which the ACS VR instead of NATting simply routes between two networks, which can be private-private, public-private, private-public, or public-public, the last two allowing for your end user VMs to have public IPs without the need of Operator intervention, also in 4.20 there will be support for BGP in the ACS VR, which means that the subnets (Public or otherwise) used by end users in Routed mode are automatically advertised upstream.
Either way, be with static or dynamic routing, with Routed mode you'll have full control of the public IPs assigned.
There will be a session by Wei Zhoue showcasing this new mode in the next CCC on Friday 22nd of November at 11:30am (CEST): https://atpscan.global.hornetsecurity.com?d=2_L5ASGiZkQUy3WIwyuwS7ZEKxepbehOc5_5hXyBHbo&f=11FL9O81QOaDecjEcRW18Bc0HDwHHudUjgQm2sZqr9gJbPweH0Shmsafosz_FMYI&i=&k=6QIh&m=xESLIvTl3JcJbFIqHFyvGNC1cUa3dk3JTm5rBb5GnLQJ3vj32DnmC4tQB2MBumH3KUBKIgO_92Wciricec2_QmRLcGT_eCOLZ0_pkooYey0r2M9ujzushgA9RReAifUB&n=-M8F8az2r7mgeJh5YU7MrhqAibcsRyS0M1O0dRutOk5rut1Q8AAkzkeebWTZrnp-&r=EHCXAcWOc9fQZ2opSXDWb5CMVWfKXQzbrDBydD8iFiT8SlLV64-xS7HzpWvCaIqS&s=4abe87433b9f7e516be6a66312b0cfd1832f52c7168732bffd6af7b44d7107b8&u=https%3A%2F%2Fwww.cloudstackcollab.org%2F
Cheers,
Alex
-----Original Message-----
From: Stephan Bienek <st...@bienek.org>
Sent: 01 October 2024 14:39
To: users@cloudstack.apache.org
Subject: Re: Public IP on instances
We are using L2 networks with for example a /29 public subnet for the customer network as well, as Lucian mentioned it.
This approach and quite a few other use cases i will share during my talk at CCC 2024 in Madrid.
As an alternative to the L2 /29 approach, which is not the most efficient approach if you only need a single IP, you could use a Shared Network with specify VLAN, without VR and tell the customer which single IP to use.
In order to make sure no customer is using "wrong" IPs, be sure to use the approach Alex Mattioli mentioned once - creating fixed MAC-IP entries on your routers.
We combine the L2 network approach with what Swen mentioned, collecting netflow data from routers via open source "pmacct" for traffic accounting.
This could help to get per-customer (or per-IP) accounting data even when using one shared VR.
Best regards,
Stephan
> Wei ZHOU <us...@gmail.com> hat am 01.10.2024 14:11 CEST geschrieben:
>
>
> +1 with what Lucian said.
>
> Please update the value of global setting "vm.network.stats.interval"
> (by default 0), and restart mgmt server.
> then you can get network statistics of each nic on shared networks.
>
> -Wei
>
> On Tue, Oct 1, 2024 at 1:47 PM Nux <nu...@li.nux.ro> wrote:
>
> > I thought the traffic usage is taken from the hypervisor, for the
> > VM's NIC.
> > Btw, you can also use L2 networks, may be more flexible and economic
> > (with IPv4 usage).
> >
> > On 2024-10-01 10:24, Alexandru Stan wrote:
> > > Hi everyone,
> > >
> > > I have a specific scenario with ACS that I'm not sure how to
> > > approach, maybe someone here can share a solution/workaround. As
> > > far as I know, the only way to have a public ip directly assigned
> > > to a vm (I mean on the vm's network adapter) is to use a shared
> > > network. But in this case all users would share one router and I
> > > wouldn't be able to track network usage individually, correct? Is
> > > there any other way to do this AND have traffic usage at vm/user
> > > level? Creating multiple shared networks is not an option, it
> > > would require constant monitoring of the routers to keep track of ip usage and so on.
> > >
> > > Thank you!
> >
RE: Public IP on instances
Posted by Alex Mattioli <Al...@shapeblue.com>.
Building on what Stephan said.
In 4.20 there will be a new feature called "ROUTED mode", in which the ACS VR instead of NATting simply routes between two networks, which can be private-private, public-private, private-public, or public-public, the last two allowing for your end user VMs to have public IPs without the need of Operator intervention, also in 4.20 there will be support for BGP in the ACS VR, which means that the subnets (Public or otherwise) used by end users in Routed mode are automatically advertised upstream.
Either way, be with static or dynamic routing, with Routed mode you'll have full control of the public IPs assigned.
There will be a session by Wei Zhoue showcasing this new mode in the next CCC on Friday 22nd of November at 11:30am (CEST): https://www.cloudstackcollab.org/
Cheers,
Alex
-----Original Message-----
From: Stephan Bienek <st...@bienek.org>
Sent: 01 October 2024 14:39
To: users@cloudstack.apache.org
Subject: Re: Public IP on instances
We are using L2 networks with for example a /29 public subnet for the customer network as well, as Lucian mentioned it.
This approach and quite a few other use cases i will share during my talk at CCC 2024 in Madrid.
As an alternative to the L2 /29 approach, which is not the most efficient approach if you only need a single IP, you could use a Shared Network with specify VLAN, without VR and tell the customer which single IP to use.
In order to make sure no customer is using "wrong" IPs, be sure to use the approach Alex Mattioli mentioned once - creating fixed MAC-IP entries on your routers.
We combine the L2 network approach with what Swen mentioned, collecting netflow data from routers via open source "pmacct" for traffic accounting.
This could help to get per-customer (or per-IP) accounting data even when using one shared VR.
Best regards,
Stephan
> Wei ZHOU <us...@gmail.com> hat am 01.10.2024 14:11 CEST geschrieben:
>
>
> +1 with what Lucian said.
>
> Please update the value of global setting "vm.network.stats.interval"
> (by default 0), and restart mgmt server.
> then you can get network statistics of each nic on shared networks.
>
> -Wei
>
> On Tue, Oct 1, 2024 at 1:47 PM Nux <nu...@li.nux.ro> wrote:
>
> > I thought the traffic usage is taken from the hypervisor, for the
> > VM's NIC.
> > Btw, you can also use L2 networks, may be more flexible and economic
> > (with IPv4 usage).
> >
> > On 2024-10-01 10:24, Alexandru Stan wrote:
> > > Hi everyone,
> > >
> > > I have a specific scenario with ACS that I'm not sure how to
> > > approach, maybe someone here can share a solution/workaround. As
> > > far as I know, the only way to have a public ip directly assigned
> > > to a vm (I mean on the vm's network adapter) is to use a shared
> > > network. But in this case all users would share one router and I
> > > wouldn't be able to track network usage individually, correct? Is
> > > there any other way to do this AND have traffic usage at vm/user
> > > level? Creating multiple shared networks is not an option, it
> > > would require constant monitoring of the routers to keep track of ip usage and so on.
> > >
> > > Thank you!
> >
Re: Public IP on instances
Posted by Stephan Bienek <st...@bienek.org>.
We are using L2 networks with for example a /29 public subnet for the customer network as well, as Lucian mentioned it.
This approach and quite a few other use cases i will share during my talk at CCC 2024 in Madrid.
As an alternative to the L2 /29 approach, which is not the most efficient approach if you only need a single IP, you could use a Shared Network with specify VLAN, without VR and tell the customer which single IP to use.
In order to make sure no customer is using "wrong" IPs, be sure to use the approach Alex Mattioli mentioned once - creating fixed MAC-IP entries on your routers.
We combine the L2 network approach with what Swen mentioned, collecting netflow data from routers via open source "pmacct" for traffic accounting.
This could help to get per-customer (or per-IP) accounting data even when using one shared VR.
Best regards,
Stephan
> Wei ZHOU <us...@gmail.com> hat am 01.10.2024 14:11 CEST geschrieben:
>
>
> +1 with what Lucian said.
>
> Please update the value of global setting "vm.network.stats.interval" (by
> default 0), and restart mgmt server.
> then you can get network statistics of each nic on shared networks.
>
> -Wei
>
> On Tue, Oct 1, 2024 at 1:47 PM Nux <nu...@li.nux.ro> wrote:
>
> > I thought the traffic usage is taken from the hypervisor, for the VM's
> > NIC.
> > Btw, you can also use L2 networks, may be more flexible and economic
> > (with IPv4 usage).
> >
> > On 2024-10-01 10:24, Alexandru Stan wrote:
> > > Hi everyone,
> > >
> > > I have a specific scenario with ACS that I'm not sure how to approach,
> > > maybe someone here can share a solution/workaround. As far as I know,
> > > the only way to have a public ip directly assigned to a vm (I mean on
> > > the vm's network adapter) is to use a shared network. But in this case
> > > all users would share one router and I wouldn't be able to track
> > > network usage individually, correct? Is there any other way to do this
> > > AND have traffic usage at vm/user level? Creating multiple shared
> > > networks is not an option, it would require constant monitoring of the
> > > routers to keep track of ip usage and so on.
> > >
> > > Thank you!
> >
Re: Public IP on instances
Posted by Wei ZHOU <us...@gmail.com>.
+1 with what Lucian said.
Please update the value of global setting "vm.network.stats.interval" (by
default 0), and restart mgmt server.
then you can get network statistics of each nic on shared networks.
-Wei
On Tue, Oct 1, 2024 at 1:47 PM Nux <nu...@li.nux.ro> wrote:
> I thought the traffic usage is taken from the hypervisor, for the VM's
> NIC.
> Btw, you can also use L2 networks, may be more flexible and economic
> (with IPv4 usage).
>
> On 2024-10-01 10:24, Alexandru Stan wrote:
> > Hi everyone,
> >
> > I have a specific scenario with ACS that I'm not sure how to approach,
> > maybe someone here can share a solution/workaround. As far as I know,
> > the only way to have a public ip directly assigned to a vm (I mean on
> > the vm's network adapter) is to use a shared network. But in this case
> > all users would share one router and I wouldn't be able to track
> > network usage individually, correct? Is there any other way to do this
> > AND have traffic usage at vm/user level? Creating multiple shared
> > networks is not an option, it would require constant monitoring of the
> > routers to keep track of ip usage and so on.
> >
> > Thank you!
>
Re: Public IP on instances
Posted by Nux <nu...@li.nux.ro>.
I thought the traffic usage is taken from the hypervisor, for the VM's
NIC.
Btw, you can also use L2 networks, may be more flexible and economic
(with IPv4 usage).
On 2024-10-01 10:24, Alexandru Stan wrote:
> Hi everyone,
>
> I have a specific scenario with ACS that I'm not sure how to approach,
> maybe someone here can share a solution/workaround. As far as I know,
> the only way to have a public ip directly assigned to a vm (I mean on
> the vm's network adapter) is to use a shared network. But in this case
> all users would share one router and I wouldn't be able to track
> network usage individually, correct? Is there any other way to do this
> AND have traffic usage at vm/user level? Creating multiple shared
> networks is not an option, it would require constant monitoring of the
> routers to keep track of ip usage and so on.
>
> Thank you!