You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Thomas Bouron <th...@cloudsoftcorp.com> on 2018/04/12 14:17:36 UTC
[PROPOSAL] Azure ARM shared resource group
Hi Brooklyners
I have recently investigated an issue that occurs when Brooklyn is
deploying a blueprint to Azure ARM. On every deployment, Brooklyn will
start by creating a shared resource group [1] and add all the blueprint VMs
into it. This is nice because all VMs are therefore able to talk to each
other.
But, the main issue here is that this shared resource group is never
deleted. You end up with a load of unused resource groups that you have to
delete manually via the UI (one by one) or the CLI. While this won't cost
you money (resources groups are just logical groups of resources) that
leads to a very annoying issue: after a while, you will run out of resource
groups, which will make subsequent deployments fail. Worse, I think it
particularly bad that we don't clean up after ourselves, leaving resources
that will prevent you from using you account.
As it stands, I can see 2 solutions to fix this:
1. try to remove the shared resource group after a VM is deleted. In case
of a mutli-VM deployment, this will fail for the first attempts (as you
cannot delete a resource group if it is not empty) but should succeed once
the last VM is deleted
2. remove this code from `JcloudsLocation.java` file and use only jClouds
`network` configuration to specify the network the VM should be in. We
could also create an initializer from this code that will create a shared
resource group, but only while setup (i.e. conscious choice from the user)
I'm leaning toward the second option here, as I think it's not Brooklyn
responsibility to sort out the networking. You should just pass the
relevant options to make your deployment successful. I believe it is the
responsibility of the cloud account owner to make sure VMs within the same
network can talk to each other.
Either way, we need to fix this. WDYT?
Best.
[1]
https://github.com/apache/brooklyn-server/blob/master/locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java#L713
--
Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Re: [PROPOSAL] Azure ARM shared resource group
Posted by Duncan Grant <du...@cloudsoft.io>.
The link you posted to creates a "default" network (yes presumably with a
resource group) but it should only do this if the default network doesn't
exist and it should then leave the network there for the next-time you use
the same azure region.
See here for a description of this behaviour
https://github.com/apache/brooklyn-server/pull/739
On Thu, 12 Apr 2018 at 16:04 Thomas Bouron <th...@cloudsoftcorp.com>
wrote:
> Duncan.
>
> jClouds does the right thing and remove the resource group when we release
> the machine, the issue is definitely in Brooklyn (see my previous link):
> Brooklyn always creates a shared resources group, multi-vm deployment or
> not, and we never remove it.
>
> The jClouds issue you referenced is precisely what I'm talking about. It
> shouldn't be assigned to jClouds but Brooklyn.
>
> Best.
>
> On Thu, 12 Apr 2018 at 15:27 Duncan Grant <du...@cloudsoft.io>
> wrote:
>
> > Thomas,
> >
> > Sorry - wrong url - should have been
> > https://issues.apache.org/jira/browse/JCLOUDS-1331
> >
> > Duncan
> >
> > On Thu, 12 Apr 2018 at 15:27 Duncan Grant <du...@cloudsoft.io>
> > wrote:
> >
> > > Thomas,
> > >
> > > I think this problem is in jclouds as is does try to remove the
> resource
> > > group but fails due to caching on the api making it look like the
> > resource
> > > group is not empty. See
> > >
> >
> https://issues.apache.org/jira/browse/JCLOUDS-1331?filter=-2&jql=project%20%3D%20JCLOUDS%20AND%20reporter%20in%20(currentUser())%20order%20by%20created%20DESC
> > >
> > > In the case of a multi-vm deployment I think we create a separate
> > resource
> > > group for each VM so I don't think there should be an issue of the
> > resource
> > > group having more than one VM in it. I think this is the correct
> > behaviour
> > > as resource groups should group things with a shared life-cycle, which
> > > wouldn't apply to all the VMs in your app.
> > >
> > > cheers
> > >
> > > Duncan
> > >
> > > On Thu, 12 Apr 2018 at 15:17 Thomas Bouron <
> > > thomas.bouron@cloudsoftcorp.com> wrote:
> > >
> > >> Hi Brooklyners
> > >>
> > >> I have recently investigated an issue that occurs when Brooklyn is
> > >> deploying a blueprint to Azure ARM. On every deployment, Brooklyn will
> > >> start by creating a shared resource group [1] and add all the
> blueprint
> > >> VMs
> > >> into it. This is nice because all VMs are therefore able to talk to
> each
> > >> other.
> > >>
> > >> But, the main issue here is that this shared resource group is never
> > >> deleted. You end up with a load of unused resource groups that you
> have
> > to
> > >> delete manually via the UI (one by one) or the CLI. While this won't
> > cost
> > >> you money (resources groups are just logical groups of resources) that
> > >> leads to a very annoying issue: after a while, you will run out of
> > >> resource
> > >> groups, which will make subsequent deployments fail. Worse, I think it
> > >> particularly bad that we don't clean up after ourselves, leaving
> > resources
> > >> that will prevent you from using you account.
> > >>
> > >> As it stands, I can see 2 solutions to fix this:
> > >> 1. try to remove the shared resource group after a VM is deleted. In
> > case
> > >> of a mutli-VM deployment, this will fail for the first attempts (as
> you
> > >> cannot delete a resource group if it is not empty) but should succeed
> > once
> > >> the last VM is deleted
> > >> 2. remove this code from `JcloudsLocation.java` file and use only
> > jClouds
> > >> `network` configuration to specify the network the VM should be in. We
> > >> could also create an initializer from this code that will create a
> > shared
> > >> resource group, but only while setup (i.e. conscious choice from the
> > user)
> > >>
> > >> I'm leaning toward the second option here, as I think it's not
> Brooklyn
> > >> responsibility to sort out the networking. You should just pass the
> > >> relevant options to make your deployment successful. I believe it is
> the
> > >> responsibility of the cloud account owner to make sure VMs within the
> > same
> > >> network can talk to each other.
> > >>
> > >> Either way, we need to fix this. WDYT?
> > >>
> > >> Best.
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://github.com/apache/brooklyn-server/blob/master/locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java#L713
> > >> --
> > >>
> > >> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> > >> https://cloudsoft.io/
> > >> Github: https://github.com/tbouron
> > >> Twitter: https://twitter.com/eltibouron
> > >>
> > >
> >
> --
>
> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> https://cloudsoft.io/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>
Re: [PROPOSAL] Azure ARM shared resource group
Posted by Thomas Bouron <th...@cloudsoftcorp.com>.
Duncan.
jClouds does the right thing and remove the resource group when we release
the machine, the issue is definitely in Brooklyn (see my previous link):
Brooklyn always creates a shared resources group, multi-vm deployment or
not, and we never remove it.
The jClouds issue you referenced is precisely what I'm talking about. It
shouldn't be assigned to jClouds but Brooklyn.
Best.
On Thu, 12 Apr 2018 at 15:27 Duncan Grant <du...@cloudsoft.io> wrote:
> Thomas,
>
> Sorry - wrong url - should have been
> https://issues.apache.org/jira/browse/JCLOUDS-1331
>
> Duncan
>
> On Thu, 12 Apr 2018 at 15:27 Duncan Grant <du...@cloudsoft.io>
> wrote:
>
> > Thomas,
> >
> > I think this problem is in jclouds as is does try to remove the resource
> > group but fails due to caching on the api making it look like the
> resource
> > group is not empty. See
> >
> https://issues.apache.org/jira/browse/JCLOUDS-1331?filter=-2&jql=project%20%3D%20JCLOUDS%20AND%20reporter%20in%20(currentUser())%20order%20by%20created%20DESC
> >
> > In the case of a multi-vm deployment I think we create a separate
> resource
> > group for each VM so I don't think there should be an issue of the
> resource
> > group having more than one VM in it. I think this is the correct
> behaviour
> > as resource groups should group things with a shared life-cycle, which
> > wouldn't apply to all the VMs in your app.
> >
> > cheers
> >
> > Duncan
> >
> > On Thu, 12 Apr 2018 at 15:17 Thomas Bouron <
> > thomas.bouron@cloudsoftcorp.com> wrote:
> >
> >> Hi Brooklyners
> >>
> >> I have recently investigated an issue that occurs when Brooklyn is
> >> deploying a blueprint to Azure ARM. On every deployment, Brooklyn will
> >> start by creating a shared resource group [1] and add all the blueprint
> >> VMs
> >> into it. This is nice because all VMs are therefore able to talk to each
> >> other.
> >>
> >> But, the main issue here is that this shared resource group is never
> >> deleted. You end up with a load of unused resource groups that you have
> to
> >> delete manually via the UI (one by one) or the CLI. While this won't
> cost
> >> you money (resources groups are just logical groups of resources) that
> >> leads to a very annoying issue: after a while, you will run out of
> >> resource
> >> groups, which will make subsequent deployments fail. Worse, I think it
> >> particularly bad that we don't clean up after ourselves, leaving
> resources
> >> that will prevent you from using you account.
> >>
> >> As it stands, I can see 2 solutions to fix this:
> >> 1. try to remove the shared resource group after a VM is deleted. In
> case
> >> of a mutli-VM deployment, this will fail for the first attempts (as you
> >> cannot delete a resource group if it is not empty) but should succeed
> once
> >> the last VM is deleted
> >> 2. remove this code from `JcloudsLocation.java` file and use only
> jClouds
> >> `network` configuration to specify the network the VM should be in. We
> >> could also create an initializer from this code that will create a
> shared
> >> resource group, but only while setup (i.e. conscious choice from the
> user)
> >>
> >> I'm leaning toward the second option here, as I think it's not Brooklyn
> >> responsibility to sort out the networking. You should just pass the
> >> relevant options to make your deployment successful. I believe it is the
> >> responsibility of the cloud account owner to make sure VMs within the
> same
> >> network can talk to each other.
> >>
> >> Either way, we need to fix this. WDYT?
> >>
> >> Best.
> >>
> >> [1]
> >>
> >>
> https://github.com/apache/brooklyn-server/blob/master/locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java#L713
> >> --
> >>
> >> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> >> https://cloudsoft.io/
> >> Github: https://github.com/tbouron
> >> Twitter: https://twitter.com/eltibouron
> >>
> >
>
--
Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Re: [PROPOSAL] Azure ARM shared resource group
Posted by Duncan Grant <du...@cloudsoft.io>.
Thomas,
Sorry - wrong url - should have been
https://issues.apache.org/jira/browse/JCLOUDS-1331
Duncan
On Thu, 12 Apr 2018 at 15:27 Duncan Grant <du...@cloudsoft.io> wrote:
> Thomas,
>
> I think this problem is in jclouds as is does try to remove the resource
> group but fails due to caching on the api making it look like the resource
> group is not empty. See
> https://issues.apache.org/jira/browse/JCLOUDS-1331?filter=-2&jql=project%20%3D%20JCLOUDS%20AND%20reporter%20in%20(currentUser())%20order%20by%20created%20DESC
>
> In the case of a multi-vm deployment I think we create a separate resource
> group for each VM so I don't think there should be an issue of the resource
> group having more than one VM in it. I think this is the correct behaviour
> as resource groups should group things with a shared life-cycle, which
> wouldn't apply to all the VMs in your app.
>
> cheers
>
> Duncan
>
> On Thu, 12 Apr 2018 at 15:17 Thomas Bouron <
> thomas.bouron@cloudsoftcorp.com> wrote:
>
>> Hi Brooklyners
>>
>> I have recently investigated an issue that occurs when Brooklyn is
>> deploying a blueprint to Azure ARM. On every deployment, Brooklyn will
>> start by creating a shared resource group [1] and add all the blueprint
>> VMs
>> into it. This is nice because all VMs are therefore able to talk to each
>> other.
>>
>> But, the main issue here is that this shared resource group is never
>> deleted. You end up with a load of unused resource groups that you have to
>> delete manually via the UI (one by one) or the CLI. While this won't cost
>> you money (resources groups are just logical groups of resources) that
>> leads to a very annoying issue: after a while, you will run out of
>> resource
>> groups, which will make subsequent deployments fail. Worse, I think it
>> particularly bad that we don't clean up after ourselves, leaving resources
>> that will prevent you from using you account.
>>
>> As it stands, I can see 2 solutions to fix this:
>> 1. try to remove the shared resource group after a VM is deleted. In case
>> of a mutli-VM deployment, this will fail for the first attempts (as you
>> cannot delete a resource group if it is not empty) but should succeed once
>> the last VM is deleted
>> 2. remove this code from `JcloudsLocation.java` file and use only jClouds
>> `network` configuration to specify the network the VM should be in. We
>> could also create an initializer from this code that will create a shared
>> resource group, but only while setup (i.e. conscious choice from the user)
>>
>> I'm leaning toward the second option here, as I think it's not Brooklyn
>> responsibility to sort out the networking. You should just pass the
>> relevant options to make your deployment successful. I believe it is the
>> responsibility of the cloud account owner to make sure VMs within the same
>> network can talk to each other.
>>
>> Either way, we need to fix this. WDYT?
>>
>> Best.
>>
>> [1]
>>
>> https://github.com/apache/brooklyn-server/blob/master/locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java#L713
>> --
>>
>> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
>> https://cloudsoft.io/
>> Github: https://github.com/tbouron
>> Twitter: https://twitter.com/eltibouron
>>
>
Re: [PROPOSAL] Azure ARM shared resource group
Posted by Duncan Grant <du...@cloudsoft.io>.
Thomas,
I think this problem is in jclouds as is does try to remove the resource
group but fails due to caching on the api making it look like the resource
group is not empty. See
https://issues.apache.org/jira/browse/JCLOUDS-1331?filter=-2&jql=project%20%3D%20JCLOUDS%20AND%20reporter%20in%20(currentUser())%20order%20by%20created%20DESC
In the case of a multi-vm deployment I think we create a separate resource
group for each VM so I don't think there should be an issue of the resource
group having more than one VM in it. I think this is the correct behaviour
as resource groups should group things with a shared life-cycle, which
wouldn't apply to all the VMs in your app.
cheers
Duncan
On Thu, 12 Apr 2018 at 15:17 Thomas Bouron <th...@cloudsoftcorp.com>
wrote:
> Hi Brooklyners
>
> I have recently investigated an issue that occurs when Brooklyn is
> deploying a blueprint to Azure ARM. On every deployment, Brooklyn will
> start by creating a shared resource group [1] and add all the blueprint VMs
> into it. This is nice because all VMs are therefore able to talk to each
> other.
>
> But, the main issue here is that this shared resource group is never
> deleted. You end up with a load of unused resource groups that you have to
> delete manually via the UI (one by one) or the CLI. While this won't cost
> you money (resources groups are just logical groups of resources) that
> leads to a very annoying issue: after a while, you will run out of resource
> groups, which will make subsequent deployments fail. Worse, I think it
> particularly bad that we don't clean up after ourselves, leaving resources
> that will prevent you from using you account.
>
> As it stands, I can see 2 solutions to fix this:
> 1. try to remove the shared resource group after a VM is deleted. In case
> of a mutli-VM deployment, this will fail for the first attempts (as you
> cannot delete a resource group if it is not empty) but should succeed once
> the last VM is deleted
> 2. remove this code from `JcloudsLocation.java` file and use only jClouds
> `network` configuration to specify the network the VM should be in. We
> could also create an initializer from this code that will create a shared
> resource group, but only while setup (i.e. conscious choice from the user)
>
> I'm leaning toward the second option here, as I think it's not Brooklyn
> responsibility to sort out the networking. You should just pass the
> relevant options to make your deployment successful. I believe it is the
> responsibility of the cloud account owner to make sure VMs within the same
> network can talk to each other.
>
> Either way, we need to fix this. WDYT?
>
> Best.
>
> [1]
>
> https://github.com/apache/brooklyn-server/blob/master/locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/JcloudsLocation.java#L713
> --
>
> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> https://cloudsoft.io/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>