You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Richard Downer <ri...@apache.org> on 2016/11/30 13:13:08 UTC
Re: [Proposal] Managing security groups using jclouds
Hi Jose,
This is challenging my memory somewhat, but I recall in the past that
jclouds had trouble with AWS security groups, because the AWS API would
report the security group as being "in use" and therefore undeletable, for
a considerable time after the VM using it was removed. So if there was an
attempt to delete the SG shortly after the VM was deleted, AWS would not
allow it.
I'm not sure if the situation has changed since then - either in the way
jclouds/Brooklyn handles it, or if the AWS API can now handle an SG delete
like this.
So I think there's some mileage in investigating this further, but also in
establishing the facts about the AWS API, and the history of this problem
(my memory is not reliable :-) before writing a lot of code!
And while I'm obviously happy to receive code contributions to Brooklyn,
it's also worth considering if any fixes that may be needed may be better
applied to the Apache jclouds project rather than here - then far more
people than the Brooklyn community would benefit.
Richard.
On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote:
> Hi all,
>
> During the last days I have provisioned about 1500 VM in AWS and Softlayer
> using Brooklyn. (I am researching about large cloud-application
> deployments).
> However, I found a little problem working with AWS. When a VM is created,
> security groups are also created, but when
> the VM is deleted, the related security groups are not deleted. It was a
> problem for me, because the limit of the security group pool is
> 500, and when the pool is full, new VM can not be provisiones because it
> is not possible to add another security group.
>
> Then, I think it could be good idea to allow security groups to be managed
> by Brooklyn. jclouds provides some interesting interfaces
> to manage security groups in a generic way, maintaing the current
> abstraction in JcloudsLocation and JcloudsMachineLocation.
> (Indeed, I already added to my code the capability of managing (deleting)
> security groups (and keyPairs).)
>
> does it make sense for you?
> WDYT?
>
> Thanks a lot,
> Jose
>
>
>
>
Re: [Proposal] Managing security groups using jclouds
Posted by Geoff Macartney <ge...@cloudsoftcorp.com>.
hi Jose
about "I think keyPairs could be also managed easily in spite of the
aforementioned jcloud’s limitations (ec2 and nova 2.0). Probably Geoff
Macartney could say something about that"...
I'm not sure I could say very much! :-) I haven't looked into the
details of the APIs for keys yet, it sounds like you know more about them
than I do. What I guess I would like to see either within Brooklyn or
JClouds is a simple tool for managing keys, as the "SecurityGroupEditor" is
intended to be for security groups (I'm sure it could use a bit of work
yet). I don't know what JClouds provides as an abstraction for keys, or
whether that might be something we could introduce in Brooklyn. Perhaps
you would like to suggest an approach? I'd be glad to review and discuss.
cheers
Geoff
On Thu, 1 Dec 2016 at 20:04 Jose Carrasco <jo...@lcc.uma.es> wrote:
> Hi all,
>
> thanks for your replies.
>
> Richard Downer, Aled Sage, I agree with you about it looks a jclouds
> problem, but it looks it would be solved easily from Brooklyn.
>
> Aled Sage, I see your point about jclouds philosophy, jclouds allows a
> node (VM) to be deleted, but you should wait to the node will be
> terminated in order to delete the related Security Groups. I deleted the
> security groups once Brooklyn had removed applications and VM,
> so I did not found any dependencies between SecurityGroup—VM (I have
> written a little toy code to show my point of view about the
> generic management of security groups [1]).
> I do not agree about incorporate the async cleanup within Brooklyn,
> because it could manage security groups (key pairs, etc…) that were
> not created by Brooklny (jclouds).
>
> In any case, Geoff Macartney has already solved this issue in
> brooklyn-server#245 ([2]). Geoff Macartney, I was using brooklyn 0.9.0 and
> I did not know this new feature, thanks to clarify this point :)
>
> Alex Henevel, Svetoslav Neykov, I agree with you. Stop operation could be
> in charge of managing elements that are
> forgotten by jclouds (security groups, key pairs, etc…).
>
> Unfortunately, jclouds does not offer a generic API to manage KeyPairs.
> KeyPairs can be managed from ec2 domain and nova 2.0 domain.
> I think keyPairs could be also managed easily in spite of the
> aforementioned jcloud’s limitations (ec2 and nova 2.0). Probably Geoff
> Macartney could say
> something about that. In any case (if it is possible) I would like to help
> there.
>
> Best,
> Jose
>
> [1] https://gist.github.com/kiuby88/6052da2ad1bd9499661e09ca44111e06
> [2] https://github.com/apache/brooklyn-server/pull/425
>
>
>
> El 01/12/2016, a las 13:34, Svetoslav Neykov <
> svetoslav.neykov@cloudsoftcorp.com> escribió:
>
> > A "recycle bin" sounds like a nice solution to the problem. We could
> improve jclouds to clean up security groups but I believe having Brooklyn
> manage them is where we should be heading long term. The reason is that
> Brooklyn is starting to create security groups on its own (say through
> SharedLocationSecurityGroupCustomizer) so it can't rely on jclouds to clean
> them up.
> > It's not only security groups. It's the same for any other managed cloud
> resource (like private keys, disks, etc.). One thing that I'd like to have
> in the UI is an exhaustive list of items we've created in the cloud and
> that are expected to be cleaned up at some point. Making locations a first
> class object in Brooklyn sounds like part of the solution here.
> >
> > Svet.
> >
> >
> >> On 1.12.2016 г., at 14:14, Alex Heneveld <
> alex.heneveld@cloudsoftcorp.com> wrote:
> >>
> >> This litter is irritating. On the other hand we don't want application
> >> stops to wait an arbitrarily long time for sec groups to become
> deletable.
> >>
> >> I'd love to switch to a model where location instances become entities.
> >> Security groups could also become part of that, and managed explicitly
> in
> >> the runtime model. "Stop" might then take a long time, or fail, but at
> >> least it's explicit...
> >>
> >> Or if we have some kind of internal "application" with management
> state, it
> >> might have a "recycling bin" and as part of "stop" we move redundant
> but
> >> not-yet-deletable security groups there, and a policy there tries
> >> periodically to remove them. This way it's transparent what we're
> doing,
> >> but doesn't get in the way of normal operation, and also doesn't leave
> >> litter behind.
> >>
> >> (BTW @jose for your use case maybe it's an option to tell brooklyn to
> use a
> >> pre-existing security group?)
> >>
> >> Best
> >> Alex
> >>
> >>
> >>
> >>
> >> On 1 December 2016 at 10:20, Geoff Macartney <
> >> geoff.macartney@cloudsoftcorp.com> wrote:
> >>
> >>> Note the PR "Factor security group code out into separate utility
> class"
> >>> [1] from last week, might be of some use here, but it just provides a
> basic
> >>> toolkit for security groups (not keys yet, but they could be added?).
> >>>
> >>> Factor out the functionality for security groups.
> >>>
> >>> Brings the create/delete and add/remove permissions functionality
> >>> for security groups from JcloudsLocationSecurityGroupCustomizer
> >>> out into its own class. The aim is to make it more convenient
> >>> to write other customizers etc. that might need security groups.
> >>>
> >>> This is intended as nearly as possible as a behaviour-preserving
> >>> change. I have permitted myself one change to the behaviour where
> >>> this seemed sensible; the loop in runOperationWithRetry
> >>> was effectively an infinite loop because of its 100 iteration
> >>> exponential back-off, so I have changed this to a back-off that
> >>> tries for about five minutes.
> >>>
> >>> For the security group deletion, what do you think of the suggestion of
> >>> just changing `removeSecurityGroup` [2] such that
> `runOperationWithRetry`
> >>> will explicitly do an exponential back-off? (i.e. at present it retries
> >>> depending on the `isExceptionRetryable` value but it could override
> that.)
> >>>
> >>> Or is it better to have an explicit long-lived service within Brooklyn
> >>> dedicated to that purpose?
> >>>
> >>> Geoff
> >>>
> >>>
> >>> [1] https://github.com/apache/brooklyn-server/pull/425
> >>> [2]
> >>> https://github.com/apache/brooklyn-server/blob/master/
> >>> locations/jclouds/src/main/java/org/apache/brooklyn/
> >>> location/jclouds/networking/SecurityGroupEditor.java#L114-L123
> >>>
> >>> On Wed, 30 Nov 2016 at 14:20 Aled Sage <al...@gmail.com> wrote:
> >>>
> >>>> Thanks for raising this Jose - sounds like some serious testing of
> >>>> Brooklyn going on as well!
> >>>>
> >>>> Richard's right: the AWS security groups seem to be eventually
> >>>> consistent. Anecdotally, it can take up to 15 minutes for the API to
> >>>> report that the security group is no longer in use. Within jclouds, we
> >>>> retry the check+deletion a few times and then move on (so the security
> >>>> group can be left behind). Given this is still happening, it means
> that
> >>>> the eventual consistency is still there.
> >>>>
> >>>> Improving things in jclouds is appealing, but tricky. The jclouds
> calls
> >>>> would have to block for a long time or leave threads behind to clean
> up
> >>>> after the delete request has returned. That would approach/philosophy
> of
> >>>> the jclouds library, I believe. Perhaps we could tweak the retry times
> >>>> though.
> >>>>
> >>>> ---
> >>>> Andrew Kennedy wrote about this quite a while ago, and said about a
> tool
> >>>> (based on jclouds) for deleting the security groups [1].
> >>>>
> >>>> I know that some customer-specific work has run with that approach to
> >>>> delete unused security groups periodically (that fit some naming
> >>>> convention, and have been unused for at least some configurable length
> >>>> of time).
> >>>>
> >>>> ---
> >>>> We could potentially incorporate the async cleanup within Brooklyn.
> >>>>
> >>>> I'm not sure whether that would just be background cleanup jobs within
> >>>> Brooklyn, or whether we'd somehow expose this on the Brooklyn API.
> >>>>
> >>>> What do your code changes look like for deleting the security groups
> and
> >>>> key-pairs?
> >>>>
> >>>> Aled
> >>>>
> >>>> [1]
> >>>>
> >>>> http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-
> >>> jclouds.html
> >>>>
> >>>>
> >>>> On 30/11/2016 13:13, Richard Downer wrote:
> >>>>> Hi Jose,
> >>>>>
> >>>>> This is challenging my memory somewhat, but I recall in the past that
> >>>>> jclouds had trouble with AWS security groups, because the AWS API
> would
> >>>>> report the security group as being "in use" and therefore
> undeletable,
> >>>> for
> >>>>> a considerable time after the VM using it was removed. So if there
> was
> >>> an
> >>>>> attempt to delete the SG shortly after the VM was deleted, AWS would
> >>> not
> >>>>> allow it.
> >>>>>
> >>>>> I'm not sure if the situation has changed since then - either in the
> >>> way
> >>>>> jclouds/Brooklyn handles it, or if the AWS API can now handle an SG
> >>>> delete
> >>>>> like this.
> >>>>>
> >>>>> So I think there's some mileage in investigating this further, but
> also
> >>>> in
> >>>>> establishing the facts about the AWS API, and the history of this
> >>> problem
> >>>>> (my memory is not reliable :-) before writing a lot of code!
> >>>>>
> >>>>> And while I'm obviously happy to receive code contributions to
> >>> Brooklyn,
> >>>>> it's also worth considering if any fixes that may be needed may be
> >>> better
> >>>>> applied to the Apache jclouds project rather than here - then far
> more
> >>>>> people than the Brooklyn community would benefit.
> >>>>>
> >>>>> Richard.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es>
> wrote:
> >>>>>
> >>>>>> Hi all,
> >>>>>>
> >>>>>> During the last days I have provisioned about 1500 VM in AWS and
> >>>> Softlayer
> >>>>>> using Brooklyn. (I am researching about large cloud-application
> >>>>>> deployments).
> >>>>>> However, I found a little problem working with AWS. When a VM is
> >>>> created,
> >>>>>> security groups are also created, but when
> >>>>>> the VM is deleted, the related security groups are not deleted. It
> >>> was a
> >>>>>> problem for me, because the limit of the security group pool is
> >>>>>> 500, and when the pool is full, new VM can not be provisiones
> because
> >>> it
> >>>>>> is not possible to add another security group.
> >>>>>>
> >>>>>> Then, I think it could be good idea to allow security groups to be
> >>>> managed
> >>>>>> by Brooklyn. jclouds provides some interesting interfaces
> >>>>>> to manage security groups in a generic way, maintaing the current
> >>>>>> abstraction in JcloudsLocation and JcloudsMachineLocation.
> >>>>>> (Indeed, I already added to my code the capability of managing
> >>>> (deleting)
> >>>>>> security groups (and keyPairs).)
> >>>>>>
> >>>>>> does it make sense for you?
> >>>>>> WDYT?
> >>>>>>
> >>>>>> Thanks a lot,
> >>>>>> Jose
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >
>
>
Re: [Proposal] Managing security groups using jclouds
Posted by Jose Carrasco <jo...@lcc.uma.es>.
Hi all,
thanks for your replies.
Richard Downer, Aled Sage, I agree with you about it looks a jclouds problem, but it looks it would be solved easily from Brooklyn.
Aled Sage, I see your point about jclouds philosophy, jclouds allows a node (VM) to be deleted, but you should wait to the node will be
terminated in order to delete the related Security Groups. I deleted the security groups once Brooklyn had removed applications and VM,
so I did not found any dependencies between SecurityGroup—VM (I have written a little toy code to show my point of view about the
generic management of security groups [1]).
I do not agree about incorporate the async cleanup within Brooklyn, because it could manage security groups (key pairs, etc…) that were
not created by Brooklny (jclouds).
In any case, Geoff Macartney has already solved this issue in brooklyn-server#245 ([2]). Geoff Macartney, I was using brooklyn 0.9.0 and
I did not know this new feature, thanks to clarify this point :)
Alex Henevel, Svetoslav Neykov, I agree with you. Stop operation could be in charge of managing elements that are
forgotten by jclouds (security groups, key pairs, etc…).
Unfortunately, jclouds does not offer a generic API to manage KeyPairs. KeyPairs can be managed from ec2 domain and nova 2.0 domain.
I think keyPairs could be also managed easily in spite of the aforementioned jcloud’s limitations (ec2 and nova 2.0). Probably Geoff Macartney could say
something about that. In any case (if it is possible) I would like to help there.
Best,
Jose
[1] https://gist.github.com/kiuby88/6052da2ad1bd9499661e09ca44111e06
[2] https://github.com/apache/brooklyn-server/pull/425
El 01/12/2016, a las 13:34, Svetoslav Neykov <sv...@cloudsoftcorp.com> escribió:
> A "recycle bin" sounds like a nice solution to the problem. We could improve jclouds to clean up security groups but I believe having Brooklyn manage them is where we should be heading long term. The reason is that Brooklyn is starting to create security groups on its own (say through SharedLocationSecurityGroupCustomizer) so it can't rely on jclouds to clean them up.
> It's not only security groups. It's the same for any other managed cloud resource (like private keys, disks, etc.). One thing that I'd like to have in the UI is an exhaustive list of items we've created in the cloud and that are expected to be cleaned up at some point. Making locations a first class object in Brooklyn sounds like part of the solution here.
>
> Svet.
>
>
>> On 1.12.2016 г., at 14:14, Alex Heneveld <al...@cloudsoftcorp.com> wrote:
>>
>> This litter is irritating. On the other hand we don't want application
>> stops to wait an arbitrarily long time for sec groups to become deletable.
>>
>> I'd love to switch to a model where location instances become entities.
>> Security groups could also become part of that, and managed explicitly in
>> the runtime model. "Stop" might then take a long time, or fail, but at
>> least it's explicit...
>>
>> Or if we have some kind of internal "application" with management state, it
>> might have a "recycling bin" and as part of "stop" we move redundant but
>> not-yet-deletable security groups there, and a policy there tries
>> periodically to remove them. This way it's transparent what we're doing,
>> but doesn't get in the way of normal operation, and also doesn't leave
>> litter behind.
>>
>> (BTW @jose for your use case maybe it's an option to tell brooklyn to use a
>> pre-existing security group?)
>>
>> Best
>> Alex
>>
>>
>>
>>
>> On 1 December 2016 at 10:20, Geoff Macartney <
>> geoff.macartney@cloudsoftcorp.com> wrote:
>>
>>> Note the PR "Factor security group code out into separate utility class"
>>> [1] from last week, might be of some use here, but it just provides a basic
>>> toolkit for security groups (not keys yet, but they could be added?).
>>>
>>> Factor out the functionality for security groups.
>>>
>>> Brings the create/delete and add/remove permissions functionality
>>> for security groups from JcloudsLocationSecurityGroupCustomizer
>>> out into its own class. The aim is to make it more convenient
>>> to write other customizers etc. that might need security groups.
>>>
>>> This is intended as nearly as possible as a behaviour-preserving
>>> change. I have permitted myself one change to the behaviour where
>>> this seemed sensible; the loop in runOperationWithRetry
>>> was effectively an infinite loop because of its 100 iteration
>>> exponential back-off, so I have changed this to a back-off that
>>> tries for about five minutes.
>>>
>>> For the security group deletion, what do you think of the suggestion of
>>> just changing `removeSecurityGroup` [2] such that `runOperationWithRetry`
>>> will explicitly do an exponential back-off? (i.e. at present it retries
>>> depending on the `isExceptionRetryable` value but it could override that.)
>>>
>>> Or is it better to have an explicit long-lived service within Brooklyn
>>> dedicated to that purpose?
>>>
>>> Geoff
>>>
>>>
>>> [1] https://github.com/apache/brooklyn-server/pull/425
>>> [2]
>>> https://github.com/apache/brooklyn-server/blob/master/
>>> locations/jclouds/src/main/java/org/apache/brooklyn/
>>> location/jclouds/networking/SecurityGroupEditor.java#L114-L123
>>>
>>> On Wed, 30 Nov 2016 at 14:20 Aled Sage <al...@gmail.com> wrote:
>>>
>>>> Thanks for raising this Jose - sounds like some serious testing of
>>>> Brooklyn going on as well!
>>>>
>>>> Richard's right: the AWS security groups seem to be eventually
>>>> consistent. Anecdotally, it can take up to 15 minutes for the API to
>>>> report that the security group is no longer in use. Within jclouds, we
>>>> retry the check+deletion a few times and then move on (so the security
>>>> group can be left behind). Given this is still happening, it means that
>>>> the eventual consistency is still there.
>>>>
>>>> Improving things in jclouds is appealing, but tricky. The jclouds calls
>>>> would have to block for a long time or leave threads behind to clean up
>>>> after the delete request has returned. That would approach/philosophy of
>>>> the jclouds library, I believe. Perhaps we could tweak the retry times
>>>> though.
>>>>
>>>> ---
>>>> Andrew Kennedy wrote about this quite a while ago, and said about a tool
>>>> (based on jclouds) for deleting the security groups [1].
>>>>
>>>> I know that some customer-specific work has run with that approach to
>>>> delete unused security groups periodically (that fit some naming
>>>> convention, and have been unused for at least some configurable length
>>>> of time).
>>>>
>>>> ---
>>>> We could potentially incorporate the async cleanup within Brooklyn.
>>>>
>>>> I'm not sure whether that would just be background cleanup jobs within
>>>> Brooklyn, or whether we'd somehow expose this on the Brooklyn API.
>>>>
>>>> What do your code changes look like for deleting the security groups and
>>>> key-pairs?
>>>>
>>>> Aled
>>>>
>>>> [1]
>>>>
>>>> http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-
>>> jclouds.html
>>>>
>>>>
>>>> On 30/11/2016 13:13, Richard Downer wrote:
>>>>> Hi Jose,
>>>>>
>>>>> This is challenging my memory somewhat, but I recall in the past that
>>>>> jclouds had trouble with AWS security groups, because the AWS API would
>>>>> report the security group as being "in use" and therefore undeletable,
>>>> for
>>>>> a considerable time after the VM using it was removed. So if there was
>>> an
>>>>> attempt to delete the SG shortly after the VM was deleted, AWS would
>>> not
>>>>> allow it.
>>>>>
>>>>> I'm not sure if the situation has changed since then - either in the
>>> way
>>>>> jclouds/Brooklyn handles it, or if the AWS API can now handle an SG
>>>> delete
>>>>> like this.
>>>>>
>>>>> So I think there's some mileage in investigating this further, but also
>>>> in
>>>>> establishing the facts about the AWS API, and the history of this
>>> problem
>>>>> (my memory is not reliable :-) before writing a lot of code!
>>>>>
>>>>> And while I'm obviously happy to receive code contributions to
>>> Brooklyn,
>>>>> it's also worth considering if any fixes that may be needed may be
>>> better
>>>>> applied to the Apache jclouds project rather than here - then far more
>>>>> people than the Brooklyn community would benefit.
>>>>>
>>>>> Richard.
>>>>>
>>>>>
>>>>>
>>>>> On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> During the last days I have provisioned about 1500 VM in AWS and
>>>> Softlayer
>>>>>> using Brooklyn. (I am researching about large cloud-application
>>>>>> deployments).
>>>>>> However, I found a little problem working with AWS. When a VM is
>>>> created,
>>>>>> security groups are also created, but when
>>>>>> the VM is deleted, the related security groups are not deleted. It
>>> was a
>>>>>> problem for me, because the limit of the security group pool is
>>>>>> 500, and when the pool is full, new VM can not be provisiones because
>>> it
>>>>>> is not possible to add another security group.
>>>>>>
>>>>>> Then, I think it could be good idea to allow security groups to be
>>>> managed
>>>>>> by Brooklyn. jclouds provides some interesting interfaces
>>>>>> to manage security groups in a generic way, maintaing the current
>>>>>> abstraction in JcloudsLocation and JcloudsMachineLocation.
>>>>>> (Indeed, I already added to my code the capability of managing
>>>> (deleting)
>>>>>> security groups (and keyPairs).)
>>>>>>
>>>>>> does it make sense for you?
>>>>>> WDYT?
>>>>>>
>>>>>> Thanks a lot,
>>>>>> Jose
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>
>
Re: [Proposal] Managing security groups using jclouds
Posted by Svetoslav Neykov <sv...@cloudsoftcorp.com>.
A "recycle bin" sounds like a nice solution to the problem. We could improve jclouds to clean up security groups but I believe having Brooklyn manage them is where we should be heading long term. The reason is that Brooklyn is starting to create security groups on its own (say through SharedLocationSecurityGroupCustomizer) so it can't rely on jclouds to clean them up.
It's not only security groups. It's the same for any other managed cloud resource (like private keys, disks, etc.). One thing that I'd like to have in the UI is an exhaustive list of items we've created in the cloud and that are expected to be cleaned up at some point. Making locations a first class object in Brooklyn sounds like part of the solution here.
Svet.
> On 1.12.2016 г., at 14:14, Alex Heneveld <al...@cloudsoftcorp.com> wrote:
>
> This litter is irritating. On the other hand we don't want application
> stops to wait an arbitrarily long time for sec groups to become deletable.
>
> I'd love to switch to a model where location instances become entities.
> Security groups could also become part of that, and managed explicitly in
> the runtime model. "Stop" might then take a long time, or fail, but at
> least it's explicit...
>
> Or if we have some kind of internal "application" with management state, it
> might have a "recycling bin" and as part of "stop" we move redundant but
> not-yet-deletable security groups there, and a policy there tries
> periodically to remove them. This way it's transparent what we're doing,
> but doesn't get in the way of normal operation, and also doesn't leave
> litter behind.
>
> (BTW @jose for your use case maybe it's an option to tell brooklyn to use a
> pre-existing security group?)
>
> Best
> Alex
>
>
>
>
> On 1 December 2016 at 10:20, Geoff Macartney <
> geoff.macartney@cloudsoftcorp.com> wrote:
>
>> Note the PR "Factor security group code out into separate utility class"
>> [1] from last week, might be of some use here, but it just provides a basic
>> toolkit for security groups (not keys yet, but they could be added?).
>>
>> Factor out the functionality for security groups.
>>
>> Brings the create/delete and add/remove permissions functionality
>> for security groups from JcloudsLocationSecurityGroupCustomizer
>> out into its own class. The aim is to make it more convenient
>> to write other customizers etc. that might need security groups.
>>
>> This is intended as nearly as possible as a behaviour-preserving
>> change. I have permitted myself one change to the behaviour where
>> this seemed sensible; the loop in runOperationWithRetry
>> was effectively an infinite loop because of its 100 iteration
>> exponential back-off, so I have changed this to a back-off that
>> tries for about five minutes.
>>
>> For the security group deletion, what do you think of the suggestion of
>> just changing `removeSecurityGroup` [2] such that `runOperationWithRetry`
>> will explicitly do an exponential back-off? (i.e. at present it retries
>> depending on the `isExceptionRetryable` value but it could override that.)
>>
>> Or is it better to have an explicit long-lived service within Brooklyn
>> dedicated to that purpose?
>>
>> Geoff
>>
>>
>> [1] https://github.com/apache/brooklyn-server/pull/425
>> [2]
>> https://github.com/apache/brooklyn-server/blob/master/
>> locations/jclouds/src/main/java/org/apache/brooklyn/
>> location/jclouds/networking/SecurityGroupEditor.java#L114-L123
>>
>> On Wed, 30 Nov 2016 at 14:20 Aled Sage <al...@gmail.com> wrote:
>>
>>> Thanks for raising this Jose - sounds like some serious testing of
>>> Brooklyn going on as well!
>>>
>>> Richard's right: the AWS security groups seem to be eventually
>>> consistent. Anecdotally, it can take up to 15 minutes for the API to
>>> report that the security group is no longer in use. Within jclouds, we
>>> retry the check+deletion a few times and then move on (so the security
>>> group can be left behind). Given this is still happening, it means that
>>> the eventual consistency is still there.
>>>
>>> Improving things in jclouds is appealing, but tricky. The jclouds calls
>>> would have to block for a long time or leave threads behind to clean up
>>> after the delete request has returned. That would approach/philosophy of
>>> the jclouds library, I believe. Perhaps we could tweak the retry times
>>> though.
>>>
>>> ---
>>> Andrew Kennedy wrote about this quite a while ago, and said about a tool
>>> (based on jclouds) for deleting the security groups [1].
>>>
>>> I know that some customer-specific work has run with that approach to
>>> delete unused security groups periodically (that fit some naming
>>> convention, and have been unused for at least some configurable length
>>> of time).
>>>
>>> ---
>>> We could potentially incorporate the async cleanup within Brooklyn.
>>>
>>> I'm not sure whether that would just be background cleanup jobs within
>>> Brooklyn, or whether we'd somehow expose this on the Brooklyn API.
>>>
>>> What do your code changes look like for deleting the security groups and
>>> key-pairs?
>>>
>>> Aled
>>>
>>> [1]
>>>
>>> http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-
>> jclouds.html
>>>
>>>
>>> On 30/11/2016 13:13, Richard Downer wrote:
>>>> Hi Jose,
>>>>
>>>> This is challenging my memory somewhat, but I recall in the past that
>>>> jclouds had trouble with AWS security groups, because the AWS API would
>>>> report the security group as being "in use" and therefore undeletable,
>>> for
>>>> a considerable time after the VM using it was removed. So if there was
>> an
>>>> attempt to delete the SG shortly after the VM was deleted, AWS would
>> not
>>>> allow it.
>>>>
>>>> I'm not sure if the situation has changed since then - either in the
>> way
>>>> jclouds/Brooklyn handles it, or if the AWS API can now handle an SG
>>> delete
>>>> like this.
>>>>
>>>> So I think there's some mileage in investigating this further, but also
>>> in
>>>> establishing the facts about the AWS API, and the history of this
>> problem
>>>> (my memory is not reliable :-) before writing a lot of code!
>>>>
>>>> And while I'm obviously happy to receive code contributions to
>> Brooklyn,
>>>> it's also worth considering if any fixes that may be needed may be
>> better
>>>> applied to the Apache jclouds project rather than here - then far more
>>>> people than the Brooklyn community would benefit.
>>>>
>>>> Richard.
>>>>
>>>>
>>>>
>>>> On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> During the last days I have provisioned about 1500 VM in AWS and
>>> Softlayer
>>>>> using Brooklyn. (I am researching about large cloud-application
>>>>> deployments).
>>>>> However, I found a little problem working with AWS. When a VM is
>>> created,
>>>>> security groups are also created, but when
>>>>> the VM is deleted, the related security groups are not deleted. It
>> was a
>>>>> problem for me, because the limit of the security group pool is
>>>>> 500, and when the pool is full, new VM can not be provisiones because
>> it
>>>>> is not possible to add another security group.
>>>>>
>>>>> Then, I think it could be good idea to allow security groups to be
>>> managed
>>>>> by Brooklyn. jclouds provides some interesting interfaces
>>>>> to manage security groups in a generic way, maintaing the current
>>>>> abstraction in JcloudsLocation and JcloudsMachineLocation.
>>>>> (Indeed, I already added to my code the capability of managing
>>> (deleting)
>>>>> security groups (and keyPairs).)
>>>>>
>>>>> does it make sense for you?
>>>>> WDYT?
>>>>>
>>>>> Thanks a lot,
>>>>> Jose
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>
Re: [Proposal] Managing security groups using jclouds
Posted by Alex Heneveld <al...@cloudsoftcorp.com>.
This litter is irritating. On the other hand we don't want application
stops to wait an arbitrarily long time for sec groups to become deletable.
I'd love to switch to a model where location instances become entities.
Security groups could also become part of that, and managed explicitly in
the runtime model. "Stop" might then take a long time, or fail, but at
least it's explicit...
Or if we have some kind of internal "application" with management state, it
might have a "recycling bin" and as part of "stop" we move redundant but
not-yet-deletable security groups there, and a policy there tries
periodically to remove them. This way it's transparent what we're doing,
but doesn't get in the way of normal operation, and also doesn't leave
litter behind.
(BTW @jose for your use case maybe it's an option to tell brooklyn to use a
pre-existing security group?)
Best
Alex
On 1 December 2016 at 10:20, Geoff Macartney <
geoff.macartney@cloudsoftcorp.com> wrote:
> Note the PR "Factor security group code out into separate utility class"
> [1] from last week, might be of some use here, but it just provides a basic
> toolkit for security groups (not keys yet, but they could be added?).
>
> Factor out the functionality for security groups.
>
> Brings the create/delete and add/remove permissions functionality
> for security groups from JcloudsLocationSecurityGroupCustomizer
> out into its own class. The aim is to make it more convenient
> to write other customizers etc. that might need security groups.
>
> This is intended as nearly as possible as a behaviour-preserving
> change. I have permitted myself one change to the behaviour where
> this seemed sensible; the loop in runOperationWithRetry
> was effectively an infinite loop because of its 100 iteration
> exponential back-off, so I have changed this to a back-off that
> tries for about five minutes.
>
> For the security group deletion, what do you think of the suggestion of
> just changing `removeSecurityGroup` [2] such that `runOperationWithRetry`
> will explicitly do an exponential back-off? (i.e. at present it retries
> depending on the `isExceptionRetryable` value but it could override that.)
>
> Or is it better to have an explicit long-lived service within Brooklyn
> dedicated to that purpose?
>
> Geoff
>
>
> [1] https://github.com/apache/brooklyn-server/pull/425
> [2]
> https://github.com/apache/brooklyn-server/blob/master/
> locations/jclouds/src/main/java/org/apache/brooklyn/
> location/jclouds/networking/SecurityGroupEditor.java#L114-L123
>
> On Wed, 30 Nov 2016 at 14:20 Aled Sage <al...@gmail.com> wrote:
>
> > Thanks for raising this Jose - sounds like some serious testing of
> > Brooklyn going on as well!
> >
> > Richard's right: the AWS security groups seem to be eventually
> > consistent. Anecdotally, it can take up to 15 minutes for the API to
> > report that the security group is no longer in use. Within jclouds, we
> > retry the check+deletion a few times and then move on (so the security
> > group can be left behind). Given this is still happening, it means that
> > the eventual consistency is still there.
> >
> > Improving things in jclouds is appealing, but tricky. The jclouds calls
> > would have to block for a long time or leave threads behind to clean up
> > after the delete request has returned. That would approach/philosophy of
> > the jclouds library, I believe. Perhaps we could tweak the retry times
> > though.
> >
> > ---
> > Andrew Kennedy wrote about this quite a while ago, and said about a tool
> > (based on jclouds) for deleting the security groups [1].
> >
> > I know that some customer-specific work has run with that approach to
> > delete unused security groups periodically (that fit some naming
> > convention, and have been unused for at least some configurable length
> > of time).
> >
> > ---
> > We could potentially incorporate the async cleanup within Brooklyn.
> >
> > I'm not sure whether that would just be background cleanup jobs within
> > Brooklyn, or whether we'd somehow expose this on the Brooklyn API.
> >
> > What do your code changes look like for deleting the security groups and
> > key-pairs?
> >
> > Aled
> >
> > [1]
> >
> > http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-
> jclouds.html
> >
> >
> > On 30/11/2016 13:13, Richard Downer wrote:
> > > Hi Jose,
> > >
> > > This is challenging my memory somewhat, but I recall in the past that
> > > jclouds had trouble with AWS security groups, because the AWS API would
> > > report the security group as being "in use" and therefore undeletable,
> > for
> > > a considerable time after the VM using it was removed. So if there was
> an
> > > attempt to delete the SG shortly after the VM was deleted, AWS would
> not
> > > allow it.
> > >
> > > I'm not sure if the situation has changed since then - either in the
> way
> > > jclouds/Brooklyn handles it, or if the AWS API can now handle an SG
> > delete
> > > like this.
> > >
> > > So I think there's some mileage in investigating this further, but also
> > in
> > > establishing the facts about the AWS API, and the history of this
> problem
> > > (my memory is not reliable :-) before writing a lot of code!
> > >
> > > And while I'm obviously happy to receive code contributions to
> Brooklyn,
> > > it's also worth considering if any fixes that may be needed may be
> better
> > > applied to the Apache jclouds project rather than here - then far more
> > > people than the Brooklyn community would benefit.
> > >
> > > Richard.
> > >
> > >
> > >
> > > On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote:
> > >
> > >> Hi all,
> > >>
> > >> During the last days I have provisioned about 1500 VM in AWS and
> > Softlayer
> > >> using Brooklyn. (I am researching about large cloud-application
> > >> deployments).
> > >> However, I found a little problem working with AWS. When a VM is
> > created,
> > >> security groups are also created, but when
> > >> the VM is deleted, the related security groups are not deleted. It
> was a
> > >> problem for me, because the limit of the security group pool is
> > >> 500, and when the pool is full, new VM can not be provisiones because
> it
> > >> is not possible to add another security group.
> > >>
> > >> Then, I think it could be good idea to allow security groups to be
> > managed
> > >> by Brooklyn. jclouds provides some interesting interfaces
> > >> to manage security groups in a generic way, maintaing the current
> > >> abstraction in JcloudsLocation and JcloudsMachineLocation.
> > >> (Indeed, I already added to my code the capability of managing
> > (deleting)
> > >> security groups (and keyPairs).)
> > >>
> > >> does it make sense for you?
> > >> WDYT?
> > >>
> > >> Thanks a lot,
> > >> Jose
> > >>
> > >>
> > >>
> > >>
> >
> >
> >
>
Re: [Proposal] Managing security groups using jclouds
Posted by Geoff Macartney <ge...@cloudsoftcorp.com>.
Note the PR "Factor security group code out into separate utility class"
[1] from last week, might be of some use here, but it just provides a basic
toolkit for security groups (not keys yet, but they could be added?).
Factor out the functionality for security groups.
Brings the create/delete and add/remove permissions functionality
for security groups from JcloudsLocationSecurityGroupCustomizer
out into its own class. The aim is to make it more convenient
to write other customizers etc. that might need security groups.
This is intended as nearly as possible as a behaviour-preserving
change. I have permitted myself one change to the behaviour where
this seemed sensible; the loop in runOperationWithRetry
was effectively an infinite loop because of its 100 iteration
exponential back-off, so I have changed this to a back-off that
tries for about five minutes.
For the security group deletion, what do you think of the suggestion of
just changing `removeSecurityGroup` [2] such that `runOperationWithRetry`
will explicitly do an exponential back-off? (i.e. at present it retries
depending on the `isExceptionRetryable` value but it could override that.)
Or is it better to have an explicit long-lived service within Brooklyn
dedicated to that purpose?
Geoff
[1] https://github.com/apache/brooklyn-server/pull/425
[2]
https://github.com/apache/brooklyn-server/blob/master/locations/jclouds/src/main/java/org/apache/brooklyn/location/jclouds/networking/SecurityGroupEditor.java#L114-L123
On Wed, 30 Nov 2016 at 14:20 Aled Sage <al...@gmail.com> wrote:
> Thanks for raising this Jose - sounds like some serious testing of
> Brooklyn going on as well!
>
> Richard's right: the AWS security groups seem to be eventually
> consistent. Anecdotally, it can take up to 15 minutes for the API to
> report that the security group is no longer in use. Within jclouds, we
> retry the check+deletion a few times and then move on (so the security
> group can be left behind). Given this is still happening, it means that
> the eventual consistency is still there.
>
> Improving things in jclouds is appealing, but tricky. The jclouds calls
> would have to block for a long time or leave threads behind to clean up
> after the delete request has returned. That would approach/philosophy of
> the jclouds library, I believe. Perhaps we could tweak the retry times
> though.
>
> ---
> Andrew Kennedy wrote about this quite a while ago, and said about a tool
> (based on jclouds) for deleting the security groups [1].
>
> I know that some customer-specific work has run with that approach to
> delete unused security groups periodically (that fit some naming
> convention, and have been unused for at least some configurable length
> of time).
>
> ---
> We could potentially incorporate the async cleanup within Brooklyn.
>
> I'm not sure whether that would just be background cleanup jobs within
> Brooklyn, or whether we'd somehow expose this on the Brooklyn API.
>
> What do your code changes look like for deleting the security groups and
> key-pairs?
>
> Aled
>
> [1]
>
> http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-jclouds.html
>
>
> On 30/11/2016 13:13, Richard Downer wrote:
> > Hi Jose,
> >
> > This is challenging my memory somewhat, but I recall in the past that
> > jclouds had trouble with AWS security groups, because the AWS API would
> > report the security group as being "in use" and therefore undeletable,
> for
> > a considerable time after the VM using it was removed. So if there was an
> > attempt to delete the SG shortly after the VM was deleted, AWS would not
> > allow it.
> >
> > I'm not sure if the situation has changed since then - either in the way
> > jclouds/Brooklyn handles it, or if the AWS API can now handle an SG
> delete
> > like this.
> >
> > So I think there's some mileage in investigating this further, but also
> in
> > establishing the facts about the AWS API, and the history of this problem
> > (my memory is not reliable :-) before writing a lot of code!
> >
> > And while I'm obviously happy to receive code contributions to Brooklyn,
> > it's also worth considering if any fixes that may be needed may be better
> > applied to the Apache jclouds project rather than here - then far more
> > people than the Brooklyn community would benefit.
> >
> > Richard.
> >
> >
> >
> > On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote:
> >
> >> Hi all,
> >>
> >> During the last days I have provisioned about 1500 VM in AWS and
> Softlayer
> >> using Brooklyn. (I am researching about large cloud-application
> >> deployments).
> >> However, I found a little problem working with AWS. When a VM is
> created,
> >> security groups are also created, but when
> >> the VM is deleted, the related security groups are not deleted. It was a
> >> problem for me, because the limit of the security group pool is
> >> 500, and when the pool is full, new VM can not be provisiones because it
> >> is not possible to add another security group.
> >>
> >> Then, I think it could be good idea to allow security groups to be
> managed
> >> by Brooklyn. jclouds provides some interesting interfaces
> >> to manage security groups in a generic way, maintaing the current
> >> abstraction in JcloudsLocation and JcloudsMachineLocation.
> >> (Indeed, I already added to my code the capability of managing
> (deleting)
> >> security groups (and keyPairs).)
> >>
> >> does it make sense for you?
> >> WDYT?
> >>
> >> Thanks a lot,
> >> Jose
> >>
> >>
> >>
> >>
>
>
>
Re: [Proposal] Managing security groups using jclouds
Posted by Aled Sage <al...@gmail.com>.
Thanks for raising this Jose - sounds like some serious testing of
Brooklyn going on as well!
Richard's right: the AWS security groups seem to be eventually
consistent. Anecdotally, it can take up to 15 minutes for the API to
report that the security group is no longer in use. Within jclouds, we
retry the check+deletion a few times and then move on (so the security
group can be left behind). Given this is still happening, it means that
the eventual consistency is still there.
Improving things in jclouds is appealing, but tricky. The jclouds calls
would have to block for a long time or leave threads behind to clean up
after the delete request has returned. That would approach/philosophy of
the jclouds library, I believe. Perhaps we could tweak the retry times
though.
---
Andrew Kennedy wrote about this quite a while ago, and said about a tool
(based on jclouds) for deleting the security groups [1].
I know that some customer-specific work has run with that approach to
delete unused security groups periodically (that fit some naming
convention, and have been unused for at least some configurable length
of time).
---
We could potentially incorporate the async cleanup within Brooklyn.
I'm not sure whether that would just be background cleanup jobs within
Brooklyn, or whether we'd somehow expose this on the Brooklyn API.
What do your code changes look like for deleting the security groups and
key-pairs?
Aled
[1]
http://blog.abstractvisitorpattern.co.uk/2013/03/tidying-up-after-jclouds.html
On 30/11/2016 13:13, Richard Downer wrote:
> Hi Jose,
>
> This is challenging my memory somewhat, but I recall in the past that
> jclouds had trouble with AWS security groups, because the AWS API would
> report the security group as being "in use" and therefore undeletable, for
> a considerable time after the VM using it was removed. So if there was an
> attempt to delete the SG shortly after the VM was deleted, AWS would not
> allow it.
>
> I'm not sure if the situation has changed since then - either in the way
> jclouds/Brooklyn handles it, or if the AWS API can now handle an SG delete
> like this.
>
> So I think there's some mileage in investigating this further, but also in
> establishing the facts about the AWS API, and the history of this problem
> (my memory is not reliable :-) before writing a lot of code!
>
> And while I'm obviously happy to receive code contributions to Brooklyn,
> it's also worth considering if any fixes that may be needed may be better
> applied to the Apache jclouds project rather than here - then far more
> people than the Brooklyn community would benefit.
>
> Richard.
>
>
>
> On 30 November 2016 at 09:40, Jose Carrasco <jo...@lcc.uma.es> wrote:
>
>> Hi all,
>>
>> During the last days I have provisioned about 1500 VM in AWS and Softlayer
>> using Brooklyn. (I am researching about large cloud-application
>> deployments).
>> However, I found a little problem working with AWS. When a VM is created,
>> security groups are also created, but when
>> the VM is deleted, the related security groups are not deleted. It was a
>> problem for me, because the limit of the security group pool is
>> 500, and when the pool is full, new VM can not be provisiones because it
>> is not possible to add another security group.
>>
>> Then, I think it could be good idea to allow security groups to be managed
>> by Brooklyn. jclouds provides some interesting interfaces
>> to manage security groups in a generic way, maintaing the current
>> abstraction in JcloudsLocation and JcloudsMachineLocation.
>> (Indeed, I already added to my code the capability of managing (deleting)
>> security groups (and keyPairs).)
>>
>> does it make sense for you?
>> WDYT?
>>
>> Thanks a lot,
>> Jose
>>
>>
>>
>>