GCE SecurityGroupExtension implementation

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

I'd like to try implement the SecurityGroupExtension interface for GCE. Looking at the documentation it seems that the combination of firewall rules and node tags is flexible enough to allow us implement the functionality.
It's been tried before but the implementation's been removed (see [1]). It's main drawback was that for each security group the code creates a new network.
Currently the biggest mismatch between the jclouds abstraction and the GCE functionality is that firewall rules must be attached to a network. 
Here's my suggested approach:
IpPermission roughly corresponds to a firewall rule
SecurityGroup is just a collection of firewall rules (there's no cloud resource that corresponds to it). The firewall rules of a security group share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They all belong to the same network.
They key bit: createSecurityGroup accepts a Location with a scope of Network, returning a custom implementation of SecurityGroup which keeps a reference to the network, so all IpPermission objects added subsequently will be on it.
While the suggested approach fits into the SecurityGroupExtension interface it's different enough from the other implementations that it might not be worth the trouble of implementing and supporting (even be harmful as users might be surprised by the different behaviour).
Would be interested in hearing other opinions on the approach. I've created a Jira to track the effort at [2].

Svet.


[1] https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c <https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c> 
[2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <https://issues.apache.org/jira/browse/JCLOUDS-1309>

Re: GCE SecurityGroupExtension implementation

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

I think the best thing we can do for the extension is to focus on jclouds managed resources and ignore the rest.
This means that jclouds will know how to map from firewall rules to SecurityGroups and back, and be able to figure out a security group is applied on a node.

This will allow users to continue using jclouds as a cloud abstraction and not care about the underlying implementation. This is also why it's fine to ignore the firewall rules that don't follow the convention (i.e. those created outside of jclouds) - users are interested in SecurityGroups and not GCE firewall rules for which there's the raw API. The only (virtual) SecurityGroups existing in the cloud are those created by jclouds.

Svet.






> On 15.06.2017 г., at 11:18, Ignasi Barrera <na...@apache.org> wrote:
> 
> Thanks for starting this discussion Svet!
> 
> It would be great to come up with a solution we are happy with.
> 
> The main issue I see here is that the primary entity of the extension,
> the SecurityGroup, does not have a mapping in GCE (which was one of
> the reasons we decided to remove the old implementation).
> 
> I am OK applying conventions to implement the extension somehow, but
> conventions are tricky when it comes to deal with existing stuff in
> the provider. With this proposal, what are your plans to implement the
> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
> location)" methods? There is also a method to list the security groups
> from a node. Given that the ComputeService is able to "listNodes" that
> weren't created by jclouds, in case of a returned Node that already
> has firewall rules assigned but that don't fit in our naming
> conventions, how would the "listSecurityGroupsForNode(String id)"
> behave?
> 
> I.
> 
> On 15 June 2017 at 09:03, Svetoslav Neykov
> <sv...@cloudsoftcorp.com> wrote:
>> I'd like to try implement the SecurityGroupExtension interface for GCE. Looking at the documentation it seems that the combination of firewall rules and node tags is flexible enough to allow us implement the functionality.
>> It's been tried before but the implementation's been removed (see [1]). It's main drawback was that for each security group the code creates a new network.
>> Currently the biggest mismatch between the jclouds abstraction and the GCE functionality is that firewall rules must be attached to a network.
>> Here's my suggested approach:
>> IpPermission roughly corresponds to a firewall rule
>> SecurityGroup is just a collection of firewall rules (there's no cloud resource that corresponds to it). The firewall rules of a security group share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They all belong to the same network.
>> They key bit: createSecurityGroup accepts a Location with a scope of Network, returning a custom implementation of SecurityGroup which keeps a reference to the network, so all IpPermission objects added subsequently will be on it.
>> While the suggested approach fits into the SecurityGroupExtension interface it's different enough from the other implementations that it might not be worth the trouble of implementing and supporting (even be harmful as users might be surprised by the different behaviour).
>> Would be interested in hearing other opinions on the approach. I've created a Jira to track the effort at [2].
>> 
>> Svet.
>> 
>> 
>> [1] https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c <https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c>
>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <https://issues.apache.org/jira/browse/JCLOUDS-1309>
>> 

Re: GCE SecurityGroupExtension implementation

[Ignasi Barrera <na...@apache.org>]

Apologies for the late reply.

I think we could go for both approaches. The current compute interface
now would support getting a Network scoped location, as long as the
TempalteBuilder can resolve its region to select the image/hardware.
But let's go for your suggested approach and ask users to be aware
they're using GCE and request them to build the Location object.

One of the main issues here is that we don't have a Network
abstraction yet, and a portable Network/Subnet object in
jclouds-compute. When we have that, it will make sense for GCE (and
probably others) to add a method to the SG extension to create an SG
in a concrete network. Meanwhile, let's go for this approach.

Thanks all for participating in this discussion!




On 23 June 2017 at 16:46, Svetoslav Neykov <sv...@cloudsoft.io> wrote:
> I think the answers to the questions below follow from the answer to another question. Do we support provisioning machines to network scoped location? If yes, then "listAvailableLocations" should follow suit and list the networks as well. When a non-network location is set it's auto-scoped to the "default" network.
> With that, we can do "template.getLocation()" and pass it to the "SecurityGroupExtension".
> If nodes could be attached to just one network it kinda makes sense to do it this way. But in GCE one can attach a machine to multiple networks (even if that's not supported by the jclouds provider atm). So the current approach of passing the region/zone and adding the networks in "templateOptions" fits GCE better. That's why I tend to lean towards not including the network scopes here, instead let the user construct it by himself.
> Adding the network scopes to "listAvailableLocations" brings it's own problems. For example that's much more dynamic, the networks could change during the life of a computeContext, leading to unexpected errors if it's cached.
>
> The above brings us back to Geoff's snippet below. Users need to be aware they are dealing with GCE security groups and extend the node's location with a network.
>
> Svet.
>
>
>
>> On 20.06.2017 г., at 12:39, Ignasi Barrera <na...@apache.org> wrote:
>>
>> Using the Location parameter to create a security group in a Network
>> might do the trick, and perhaps we find the way to document or model
>> it so it is easy to write portable code. We just need to think a bit
>> more about it and explore all the implications and possible
>> alternatives.
>>
>> Here are some additional thoughts to take into account if we go that path:
>>
>> If we use the location parameter we should also think about its
>> relationship with the collection returned by the
>> ComputeService.listAvailableLocations method [1]. It returns the list
>> of locations that can be used to spawn nodes, but I see it as a
>> general method of the locations made available to be managed by the
>> Compute abstraction.
>>
>> * Should we return a location with Network scope for each existing network?
>> * Should we keep that method as-is and ask users to build a proper
>> network-scoped Location object themselves?
>> * How should we refine the semantics of that method so it is crystal
>> clear to users how to use it and what to expect?
>>
>> I.
>>
>> [1] https://github.com/jclouds/jclouds/blob/master/compute/src/main/java/org/jclouds/compute/ComputeService.java#L108-L118
>>
>> P.S. Thanks to you Svet for bringing this discussion to the list!
>>
>>
>> On 19 June 2017 at 10:24, Geoff Macartney
>> <ge...@cloudsoftcorp.com> wrote:
>>> Ok Svet;
>>>
>>> It would certainly be a shame to have to write
>>>
>>> if (provider.equals("GCE")) {
>>>       createSecurityGroupOneWay();
>>> } else {
>>>       createSecurityGroupAnotherWay();
>>> }
>>>
>>> I guess I agree it's probably better, when the provider doesn't support a
>>> concept, to avoid having an implementation that's "nearly but not quite",
>>> otherwise you'd end up with lots of code like the above.
>>>
>>> G
>>>
>>> On Sun, 18 Jun 2017 at 15:23 Svetoslav Neykov <
>>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>>>
>>>>> if that entity does not exist in GCE and
>>>>> users won't be able to use the extension as in other clouds, is there
>>>>> a real use case for it that justifies its implementation?
>>>>
>>>>
>>>> My personal drive for the implementation is to reuse existing code
>>>> managing the ingress access to (freshly spun up) nodes managed by jclouds.
>>>> So it would work nicely for my case :).
>>>>
>>>> What still irks me about the proposal is the different scope of the
>>>> location required to create the security group. It essentially means the
>>>> implementation:
>>>>  * is incompatible with existing code out in the wild - it will break it
>>>> when "securityGroupExtension.isPresent()" returns true,  but the
>>>> "createSecurityGroup" subsequently fails due to incorrect location argument
>>>>  * requires special casing in the code to pass the correct location scope
>>>> - doing template.getLocation() will not do it for GCE which is what I use
>>>> to pass as an argument to "createSecurityGroup"
>>>>
>>>> I took some time to think this through, that's why the delay in response,
>>>> but don't see a nice solution to ^^^. That's why I'm backing off from the
>>>> proposal. Still think it's been a useful discussion to have. Appreciate the
>>>> involvement Ignasi and Geoff.
>>>>
>>>>
>>>> Svet.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On 16.06.2017 г., at 10:01, Ignasi Barrera <na...@apache.org> wrote:
>>>>>
>>>>> I agree it is a grey area where we can only do our best.
>>>>>
>>>>> I'd say it is ok to skip egress rules and to only consider the rules
>>>>> created by jclouds, that is something we could assume given the fact
>>>>> that the concept of security groups does not exist in GCE, which
>>>>> brings me to the question: if that entity does not exist in GCE and
>>>>> users won't be able to use the extension as in other clouds, is there
>>>>> a real use case for it that justifies its implementation?
>>>>>
>>>>> If the answer is yes, then let's move forward with your PoC. But I'd
>>>>> like to make sure that we have identified all conflicting points (I
>>>>> think we do). We can assume we're not going to support existing
>>>>> resources not related to jclouds, but as Geoff mentioned too I think
>>>>> we should try to support onboarding existing resources that were
>>>>> created with previous versions of jclouds with the current
>>>>> "inboundPorts" implementation.
>>>>>
>>>>> On 15 June 2017 at 15:59, Svetoslav Neykov
>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>> The onboarding experience takes a best effort approach and never
>>>> represents the cloud state exactly. It's a gray area of compromises.
>>>>>> For example GCE supports ingress/egress; allow/deny rules. Of those
>>>> only ingress+allow rules can be represented using IpPermission. What do we
>>>> do about the rest? Just ignore, like we do for other clouds? Where do we
>>>> draw the line and say that it doesn't fit the abstraction so it's not
>>>> presented to the user? Taking this to the extreme we could decide that the
>>>> only rules that fit the abstraction are those created by jclouds.
>>>>>>
>>>>>> It would be fairly easy to come up with a solution which transforms the
>>>> firewall rules into a SecurityGroup (read SGs). What I'm worried about is
>>>> updating onboarded SGs. It could result in some unexpected changes.
>>>>>>
>>>>>> Re the security group created for inboundPorts (or anything managed by
>>>> jclouds) - it's something we expect and know what it looks like, how it's
>>>> used, so should be straightforward to accommodate in the design. It's the
>>>> unknown that's scary :).
>>>>>>
>>>>>> If we decide to represent existing firewall rules, here's how it could
>>>> work:
>>>>>> * skip egress; deny rules
>>>>>> * rules having a list of target tags
>>>>>>   - create a security group per tag per network
>>>>>>   - share a rule between security groups - one per tag per network
>>>>>>   - deleting an IpPermission is removing the tag from the firewall
>>>> rule (and deleting it if last one)
>>>>>>   - adding a rule is straightforward - create a permission with the
>>>> same destination tag as the security group ID
>>>>>>   - adding a SG to a node is adding the ID for the SG to the tags of
>>>> the node
>>>>>> * rules having an "All instances in the network" target
>>>>>>   - group into a single SG (per network)
>>>>>>   - consider it a network SG and deny requests to attach it to a node
>>>>>>
>>>>>> I still think the "surprise factor" with the GCE implementation will be
>>>> the requirement to pass a network scope location to the
>>>> "createSecurityGroup" call. It's something that's different from the other
>>>> implementations and means it can't "just work" with existing users. If we
>>>> agree on a solution here, then I'm sure we'll make everything else work.
>>>>>>
>>>>>> Svet.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 15.06.2017 г., at 13:26, Geoff Macartney <
>>>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>>>>>
>>>>>>> hi Ignasi,
>>>>>>>
>>>>>>> that's an interesting issue.  Re. your last paragraph, that could
>>>> certainly
>>>>>>> be a good approach, but it still won't address the question of
>>>> onboarding
>>>>>>> existing infrastructure, right?  i.e. where there are rules that
>>>> weren't
>>>>>>> written according to those conventions - or of upgrading to the new
>>>> jclouds
>>>>>>> version (with GCE security groups) to continue to manage infrastructure
>>>>>>> that _was_ created and provisioned by jclouds.
>>>>>>>
>>>>>>> I'll certainly need to have a think about it. As you say we could
>>>> refine
>>>>>>> and be explicit about how the proposal would work - perhaps come up
>>>> with a
>>>>>>> use-case that illustrates the scenarios, then we could more easily see
>>>> if
>>>>>>> there are gaps to fill.
>>>>>>>
>>>>>>> Geoff
>>>>>>>
>>>>>>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:
>>>>>>>
>>>>>>>> I understand the motivation behind ignoring existing stuff, but I have
>>>>>>>> some concerns.
>>>>>>>>
>>>>>>>> Onboarding existing cloud infrastructure is a valid use case for
>>>>>>>> jclouds users, and that can already be done (limited by the bounds of
>>>>>>>> the Compute abstraction) with the current jclouds implementation. It
>>>>>>>> is fair to expect that if the SecurityGroupExtension is present, and
>>>>>>>> there is a method called "listSecurityGroupsForNode" you should be
>>>>>>>> able to onboard the firewall rules of your existing node. As a user,
>>>>>>>> I'd have these expectations so I don't think we can discard the
>>>>>>>> onboarding use case just because we've found it difficult to map in a
>>>>>>>> consistent way.
>>>>>>>>
>>>>>>>> This said, if we went that path, I think we still need to refine the
>>>>>>>> proposal, or at least be explicit and enumerate all affected points
>>>>>>>> and how we plan to fill the gaps.
>>>>>>>>
>>>>>>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
>>>>>>>> With the actual code, the node would be behind a couple firewall
>>>>>>>> rules, but the security group extension won't be able to return any
>>>>>>>> security group for the node. That is inconsistent, as something you
>>>>>>>> created using the abstraction, cannot be retrieved with the same
>>>>>>>> information using the abstraction extension. It will give users the
>>>>>>>> impression that the node has no firewall rules applied.
>>>>>>>>
>>>>>>>> This case could be as simple as creating the firewall rules for the
>>>>>>>> "inboundPorts" following our SG naming and tag convention. I just want
>>>>>>>> to illustrate that the problem is not exclusive of the resources
>>>>>>>> created "outside" jclouds.
>>>>>>>>
>>>>>>>> On 15 June 2017 at 11:09, Svetoslav Neykov
>>>>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>>>>>>> above:
>>>>>>>>>
>>>>>>>>> That's right, tags are an important part of the convention and I
>>>> didn't
>>>>>>>> expand on that. Your description nicely captures the idea.
>>>>>>>>> I believe the security groups will work across subnets in a single
>>>> vpc
>>>>>>>> network.
>>>>>>>>>
>>>>>>>>> Svet.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
>>>>>>>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>>>>>>>>
>>>>>>>>>> I like the sound of this proposal, I think it's certainly worth
>>>>>>>>>> investigating.
>>>>>>>>>>
>>>>>>>>>> Ignasi those are good questions you've asked.  Here's my thoughts:
>>>>>>>>>>
>>>>>>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and
>>>> be
>>>>>>>>>> explicit about that in documentation. What I mean here is that we
>>>> don't
>>>>>>>>>> need to do a "reverse mapping" of existing stuff back into jclouds
>>>>>>>>>> concepts. So if there are existing firewall rules they simply don't
>>>>>>>> show up
>>>>>>>>>> in the results of 'listSecurityGroups()",
>>>>>>>> "listSecurityGroupsInLocation()"
>>>>>>>>>> or "listSecurityGroupsForNode()".    That is, not all rules
>>>> participate
>>>>>>>> in
>>>>>>>>>> "security groups" from the jclouds point of view, only ones that
>>>> jclouds
>>>>>>>>>> added for security group purposes.
>>>>>>>>>>
>>>>>>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>>>>>>> above:
>>>>>>>>>>
>>>>>>>>>> 1.  What do you think about the idea of using tags to identify
>>>> security
>>>>>>>>>> groups - each security group would correspond to a tag. These tags
>>>> would
>>>>>>>>>> correspond to groupIds in the IPPermission object, i.e. you could
>>>>>>>> create an
>>>>>>>>>> IPPermission using a groupId [1] which would relate to a tag in the
>>>> GCE
>>>>>>>>>> platform. Then any nodes with that tag (= in that security group)
>>>> would
>>>>>>>> be
>>>>>>>>>> permitted access.
>>>>>>>>>> 2. Are there any restrictions/implications relating to subnets in
>>>> your
>>>>>>>>>> proposal, e.g. would you need to use them to achieve what you want,
>>>> or
>>>>>>>> can
>>>>>>>>>> it work irrespective of any subnets on the network?
>>>>>>>>>>
>>>>>>>>>> Geoff
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>>
>>>>>>>>
>>>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org>
>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for starting this discussion Svet!
>>>>>>>>>>>
>>>>>>>>>>> It would be great to come up with a solution we are happy with.
>>>>>>>>>>>
>>>>>>>>>>> The main issue I see here is that the primary entity of the
>>>> extension,
>>>>>>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>>>>>>>>>>> the reasons we decided to remove the old implementation).
>>>>>>>>>>>
>>>>>>>>>>> I am OK applying conventions to implement the extension somehow,
>>>> but
>>>>>>>>>>> conventions are tricky when it comes to deal with existing stuff in
>>>>>>>>>>> the provider. With this proposal, what are your plans to implement
>>>> the
>>>>>>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>>>>>>>>>>> location)" methods? There is also a method to list the security
>>>> groups
>>>>>>>>>>> from a node. Given that the ComputeService is able to "listNodes"
>>>> that
>>>>>>>>>>> weren't created by jclouds, in case of a returned Node that already
>>>>>>>>>>> has firewall rules assigned but that don't fit in our naming
>>>>>>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>>>>>>>>>>> behave?
>>>>>>>>>>>
>>>>>>>>>>> I.
>>>>>>>>>>>
>>>>>>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>>>>>>>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>>>>>>>> I'd like to try implement the SecurityGroupExtension interface for
>>>>>>>> GCE.
>>>>>>>>>>> Looking at the documentation it seems that the combination of
>>>> firewall
>>>>>>>>>>> rules and node tags is flexible enough to allow us implement the
>>>>>>>>>>> functionality.
>>>>>>>>>>>> It's been tried before but the implementation's been removed (see
>>>>>>>> [1]).
>>>>>>>>>>> It's main drawback was that for each security group the code
>>>> creates a
>>>>>>>> new
>>>>>>>>>>> network.
>>>>>>>>>>>> Currently the biggest mismatch between the jclouds abstraction
>>>> and the
>>>>>>>>>>> GCE functionality is that firewall rules must be attached to a
>>>> network.
>>>>>>>>>>>> Here's my suggested approach:
>>>>>>>>>>>> IpPermission roughly corresponds to a firewall rule
>>>>>>>>>>>> SecurityGroup is just a collection of firewall rules (there's no
>>>> cloud
>>>>>>>>>>> resource that corresponds to it). The firewall rules of a security
>>>>>>>> group
>>>>>>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>.
>>>> They
>>>>>>>> all
>>>>>>>>>>> belong to the same network.
>>>>>>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope
>>>> of
>>>>>>>>>>> Network, returning a custom implementation of SecurityGroup which
>>>>>>>> keeps a
>>>>>>>>>>> reference to the network, so all IpPermission objects added
>>>>>>>> subsequently
>>>>>>>>>>> will be on it.
>>>>>>>>>>>> While the suggested approach fits into the SecurityGroupExtension
>>>>>>>>>>> interface it's different enough from the other implementations
>>>> that it
>>>>>>>>>>> might not be worth the trouble of implementing and supporting
>>>> (even be
>>>>>>>>>>> harmful as users might be surprised by the different behaviour).
>>>>>>>>>>>> Would be interested in hearing other opinions on the approach.
>>>> I've
>>>>>>>>>>> created a Jira to track the effort at [2].
>>>>>>>>>>>>
>>>>>>>>>>>> Svet.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>
>>>>>>>>
>>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>>>>>> <
>>>>>>>>>>>
>>>>>>>>
>>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>>>>>>>
>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>>>>>>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>>>>
>

Re: GCE SecurityGroupExtension implementation

[Svetoslav Neykov <sv...@cloudsoft.io>]

I think the answers to the questions below follow from the answer to another question. Do we support provisioning machines to network scoped location? If yes, then "listAvailableLocations" should follow suit and list the networks as well. When a non-network location is set it's auto-scoped to the "default" network.
With that, we can do "template.getLocation()" and pass it to the "SecurityGroupExtension".
If nodes could be attached to just one network it kinda makes sense to do it this way. But in GCE one can attach a machine to multiple networks (even if that's not supported by the jclouds provider atm). So the current approach of passing the region/zone and adding the networks in "templateOptions" fits GCE better. That's why I tend to lean towards not including the network scopes here, instead let the user construct it by himself.
Adding the network scopes to "listAvailableLocations" brings it's own problems. For example that's much more dynamic, the networks could change during the life of a computeContext, leading to unexpected errors if it's cached.

The above brings us back to Geoff's snippet below. Users need to be aware they are dealing with GCE security groups and extend the node's location with a network.

Svet.



> On 20.06.2017 г., at 12:39, Ignasi Barrera <na...@apache.org> wrote:
> 
> Using the Location parameter to create a security group in a Network
> might do the trick, and perhaps we find the way to document or model
> it so it is easy to write portable code. We just need to think a bit
> more about it and explore all the implications and possible
> alternatives.
> 
> Here are some additional thoughts to take into account if we go that path:
> 
> If we use the location parameter we should also think about its
> relationship with the collection returned by the
> ComputeService.listAvailableLocations method [1]. It returns the list
> of locations that can be used to spawn nodes, but I see it as a
> general method of the locations made available to be managed by the
> Compute abstraction.
> 
> * Should we return a location with Network scope for each existing network?
> * Should we keep that method as-is and ask users to build a proper
> network-scoped Location object themselves?
> * How should we refine the semantics of that method so it is crystal
> clear to users how to use it and what to expect?
> 
> I.
> 
> [1] https://github.com/jclouds/jclouds/blob/master/compute/src/main/java/org/jclouds/compute/ComputeService.java#L108-L118
> 
> P.S. Thanks to you Svet for bringing this discussion to the list!
> 
> 
> On 19 June 2017 at 10:24, Geoff Macartney
> <ge...@cloudsoftcorp.com> wrote:
>> Ok Svet;
>> 
>> It would certainly be a shame to have to write
>> 
>> if (provider.equals("GCE")) {
>>       createSecurityGroupOneWay();
>> } else {
>>       createSecurityGroupAnotherWay();
>> }
>> 
>> I guess I agree it's probably better, when the provider doesn't support a
>> concept, to avoid having an implementation that's "nearly but not quite",
>> otherwise you'd end up with lots of code like the above.
>> 
>> G
>> 
>> On Sun, 18 Jun 2017 at 15:23 Svetoslav Neykov <
>> svetoslav.neykov@cloudsoftcorp.com> wrote:
>> 
>>>> if that entity does not exist in GCE and
>>>> users won't be able to use the extension as in other clouds, is there
>>>> a real use case for it that justifies its implementation?
>>> 
>>> 
>>> My personal drive for the implementation is to reuse existing code
>>> managing the ingress access to (freshly spun up) nodes managed by jclouds.
>>> So it would work nicely for my case :).
>>> 
>>> What still irks me about the proposal is the different scope of the
>>> location required to create the security group. It essentially means the
>>> implementation:
>>>  * is incompatible with existing code out in the wild - it will break it
>>> when "securityGroupExtension.isPresent()" returns true,  but the
>>> "createSecurityGroup" subsequently fails due to incorrect location argument
>>>  * requires special casing in the code to pass the correct location scope
>>> - doing template.getLocation() will not do it for GCE which is what I use
>>> to pass as an argument to "createSecurityGroup"
>>> 
>>> I took some time to think this through, that's why the delay in response,
>>> but don't see a nice solution to ^^^. That's why I'm backing off from the
>>> proposal. Still think it's been a useful discussion to have. Appreciate the
>>> involvement Ignasi and Geoff.
>>> 
>>> 
>>> Svet.
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> On 16.06.2017 г., at 10:01, Ignasi Barrera <na...@apache.org> wrote:
>>>> 
>>>> I agree it is a grey area where we can only do our best.
>>>> 
>>>> I'd say it is ok to skip egress rules and to only consider the rules
>>>> created by jclouds, that is something we could assume given the fact
>>>> that the concept of security groups does not exist in GCE, which
>>>> brings me to the question: if that entity does not exist in GCE and
>>>> users won't be able to use the extension as in other clouds, is there
>>>> a real use case for it that justifies its implementation?
>>>> 
>>>> If the answer is yes, then let's move forward with your PoC. But I'd
>>>> like to make sure that we have identified all conflicting points (I
>>>> think we do). We can assume we're not going to support existing
>>>> resources not related to jclouds, but as Geoff mentioned too I think
>>>> we should try to support onboarding existing resources that were
>>>> created with previous versions of jclouds with the current
>>>> "inboundPorts" implementation.
>>>> 
>>>> On 15 June 2017 at 15:59, Svetoslav Neykov
>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>> The onboarding experience takes a best effort approach and never
>>> represents the cloud state exactly. It's a gray area of compromises.
>>>>> For example GCE supports ingress/egress; allow/deny rules. Of those
>>> only ingress+allow rules can be represented using IpPermission. What do we
>>> do about the rest? Just ignore, like we do for other clouds? Where do we
>>> draw the line and say that it doesn't fit the abstraction so it's not
>>> presented to the user? Taking this to the extreme we could decide that the
>>> only rules that fit the abstraction are those created by jclouds.
>>>>> 
>>>>> It would be fairly easy to come up with a solution which transforms the
>>> firewall rules into a SecurityGroup (read SGs). What I'm worried about is
>>> updating onboarded SGs. It could result in some unexpected changes.
>>>>> 
>>>>> Re the security group created for inboundPorts (or anything managed by
>>> jclouds) - it's something we expect and know what it looks like, how it's
>>> used, so should be straightforward to accommodate in the design. It's the
>>> unknown that's scary :).
>>>>> 
>>>>> If we decide to represent existing firewall rules, here's how it could
>>> work:
>>>>> * skip egress; deny rules
>>>>> * rules having a list of target tags
>>>>>   - create a security group per tag per network
>>>>>   - share a rule between security groups - one per tag per network
>>>>>   - deleting an IpPermission is removing the tag from the firewall
>>> rule (and deleting it if last one)
>>>>>   - adding a rule is straightforward - create a permission with the
>>> same destination tag as the security group ID
>>>>>   - adding a SG to a node is adding the ID for the SG to the tags of
>>> the node
>>>>> * rules having an "All instances in the network" target
>>>>>   - group into a single SG (per network)
>>>>>   - consider it a network SG and deny requests to attach it to a node
>>>>> 
>>>>> I still think the "surprise factor" with the GCE implementation will be
>>> the requirement to pass a network scope location to the
>>> "createSecurityGroup" call. It's something that's different from the other
>>> implementations and means it can't "just work" with existing users. If we
>>> agree on a solution here, then I'm sure we'll make everything else work.
>>>>> 
>>>>> Svet.
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> On 15.06.2017 г., at 13:26, Geoff Macartney <
>>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>>>> 
>>>>>> hi Ignasi,
>>>>>> 
>>>>>> that's an interesting issue.  Re. your last paragraph, that could
>>> certainly
>>>>>> be a good approach, but it still won't address the question of
>>> onboarding
>>>>>> existing infrastructure, right?  i.e. where there are rules that
>>> weren't
>>>>>> written according to those conventions - or of upgrading to the new
>>> jclouds
>>>>>> version (with GCE security groups) to continue to manage infrastructure
>>>>>> that _was_ created and provisioned by jclouds.
>>>>>> 
>>>>>> I'll certainly need to have a think about it. As you say we could
>>> refine
>>>>>> and be explicit about how the proposal would work - perhaps come up
>>> with a
>>>>>> use-case that illustrates the scenarios, then we could more easily see
>>> if
>>>>>> there are gaps to fill.
>>>>>> 
>>>>>> Geoff
>>>>>> 
>>>>>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:
>>>>>> 
>>>>>>> I understand the motivation behind ignoring existing stuff, but I have
>>>>>>> some concerns.
>>>>>>> 
>>>>>>> Onboarding existing cloud infrastructure is a valid use case for
>>>>>>> jclouds users, and that can already be done (limited by the bounds of
>>>>>>> the Compute abstraction) with the current jclouds implementation. It
>>>>>>> is fair to expect that if the SecurityGroupExtension is present, and
>>>>>>> there is a method called "listSecurityGroupsForNode" you should be
>>>>>>> able to onboard the firewall rules of your existing node. As a user,
>>>>>>> I'd have these expectations so I don't think we can discard the
>>>>>>> onboarding use case just because we've found it difficult to map in a
>>>>>>> consistent way.
>>>>>>> 
>>>>>>> This said, if we went that path, I think we still need to refine the
>>>>>>> proposal, or at least be explicit and enumerate all affected points
>>>>>>> and how we plan to fill the gaps.
>>>>>>> 
>>>>>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
>>>>>>> With the actual code, the node would be behind a couple firewall
>>>>>>> rules, but the security group extension won't be able to return any
>>>>>>> security group for the node. That is inconsistent, as something you
>>>>>>> created using the abstraction, cannot be retrieved with the same
>>>>>>> information using the abstraction extension. It will give users the
>>>>>>> impression that the node has no firewall rules applied.
>>>>>>> 
>>>>>>> This case could be as simple as creating the firewall rules for the
>>>>>>> "inboundPorts" following our SG naming and tag convention. I just want
>>>>>>> to illustrate that the problem is not exclusive of the resources
>>>>>>> created "outside" jclouds.
>>>>>>> 
>>>>>>> On 15 June 2017 at 11:09, Svetoslav Neykov
>>>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>>>>>> above:
>>>>>>>> 
>>>>>>>> That's right, tags are an important part of the convention and I
>>> didn't
>>>>>>> expand on that. Your description nicely captures the idea.
>>>>>>>> I believe the security groups will work across subnets in a single
>>> vpc
>>>>>>> network.
>>>>>>>> 
>>>>>>>> Svet.
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
>>>>>>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>>>>>>> 
>>>>>>>>> I like the sound of this proposal, I think it's certainly worth
>>>>>>>>> investigating.
>>>>>>>>> 
>>>>>>>>> Ignasi those are good questions you've asked.  Here's my thoughts:
>>>>>>>>> 
>>>>>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and
>>> be
>>>>>>>>> explicit about that in documentation. What I mean here is that we
>>> don't
>>>>>>>>> need to do a "reverse mapping" of existing stuff back into jclouds
>>>>>>>>> concepts. So if there are existing firewall rules they simply don't
>>>>>>> show up
>>>>>>>>> in the results of 'listSecurityGroups()",
>>>>>>> "listSecurityGroupsInLocation()"
>>>>>>>>> or "listSecurityGroupsForNode()".    That is, not all rules
>>> participate
>>>>>>> in
>>>>>>>>> "security groups" from the jclouds point of view, only ones that
>>> jclouds
>>>>>>>>> added for security group purposes.
>>>>>>>>> 
>>>>>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>>>>>> above:
>>>>>>>>> 
>>>>>>>>> 1.  What do you think about the idea of using tags to identify
>>> security
>>>>>>>>> groups - each security group would correspond to a tag. These tags
>>> would
>>>>>>>>> correspond to groupIds in the IPPermission object, i.e. you could
>>>>>>> create an
>>>>>>>>> IPPermission using a groupId [1] which would relate to a tag in the
>>> GCE
>>>>>>>>> platform. Then any nodes with that tag (= in that security group)
>>> would
>>>>>>> be
>>>>>>>>> permitted access.
>>>>>>>>> 2. Are there any restrictions/implications relating to subnets in
>>> your
>>>>>>>>> proposal, e.g. would you need to use them to achieve what you want,
>>> or
>>>>>>> can
>>>>>>>>> it work irrespective of any subnets on the network?
>>>>>>>>> 
>>>>>>>>> Geoff
>>>>>>>>> 
>>>>>>>>> [1]
>>>>>>>>> 
>>>>>>> 
>>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org>
>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Thanks for starting this discussion Svet!
>>>>>>>>>> 
>>>>>>>>>> It would be great to come up with a solution we are happy with.
>>>>>>>>>> 
>>>>>>>>>> The main issue I see here is that the primary entity of the
>>> extension,
>>>>>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>>>>>>>>>> the reasons we decided to remove the old implementation).
>>>>>>>>>> 
>>>>>>>>>> I am OK applying conventions to implement the extension somehow,
>>> but
>>>>>>>>>> conventions are tricky when it comes to deal with existing stuff in
>>>>>>>>>> the provider. With this proposal, what are your plans to implement
>>> the
>>>>>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>>>>>>>>>> location)" methods? There is also a method to list the security
>>> groups
>>>>>>>>>> from a node. Given that the ComputeService is able to "listNodes"
>>> that
>>>>>>>>>> weren't created by jclouds, in case of a returned Node that already
>>>>>>>>>> has firewall rules assigned but that don't fit in our naming
>>>>>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>>>>>>>>>> behave?
>>>>>>>>>> 
>>>>>>>>>> I.
>>>>>>>>>> 
>>>>>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>>>>>>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>>>>>>> I'd like to try implement the SecurityGroupExtension interface for
>>>>>>> GCE.
>>>>>>>>>> Looking at the documentation it seems that the combination of
>>> firewall
>>>>>>>>>> rules and node tags is flexible enough to allow us implement the
>>>>>>>>>> functionality.
>>>>>>>>>>> It's been tried before but the implementation's been removed (see
>>>>>>> [1]).
>>>>>>>>>> It's main drawback was that for each security group the code
>>> creates a
>>>>>>> new
>>>>>>>>>> network.
>>>>>>>>>>> Currently the biggest mismatch between the jclouds abstraction
>>> and the
>>>>>>>>>> GCE functionality is that firewall rules must be attached to a
>>> network.
>>>>>>>>>>> Here's my suggested approach:
>>>>>>>>>>> IpPermission roughly corresponds to a firewall rule
>>>>>>>>>>> SecurityGroup is just a collection of firewall rules (there's no
>>> cloud
>>>>>>>>>> resource that corresponds to it). The firewall rules of a security
>>>>>>> group
>>>>>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>.
>>> They
>>>>>>> all
>>>>>>>>>> belong to the same network.
>>>>>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope
>>> of
>>>>>>>>>> Network, returning a custom implementation of SecurityGroup which
>>>>>>> keeps a
>>>>>>>>>> reference to the network, so all IpPermission objects added
>>>>>>> subsequently
>>>>>>>>>> will be on it.
>>>>>>>>>>> While the suggested approach fits into the SecurityGroupExtension
>>>>>>>>>> interface it's different enough from the other implementations
>>> that it
>>>>>>>>>> might not be worth the trouble of implementing and supporting
>>> (even be
>>>>>>>>>> harmful as users might be surprised by the different behaviour).
>>>>>>>>>>> Would be interested in hearing other opinions on the approach.
>>> I've
>>>>>>>>>> created a Jira to track the effort at [2].
>>>>>>>>>>> 
>>>>>>>>>>> Svet.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> [1]
>>>>>>>>>> 
>>>>>>> 
>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>>>>> <
>>>>>>>>>> 
>>>>>>> 
>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>>>>>> 
>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>>>>>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>>> 

Re: GCE SecurityGroupExtension implementation

[Ignasi Barrera <na...@apache.org>]

Using the Location parameter to create a security group in a Network
might do the trick, and perhaps we find the way to document or model
it so it is easy to write portable code. We just need to think a bit
more about it and explore all the implications and possible
alternatives.

Here are some additional thoughts to take into account if we go that path:

If we use the location parameter we should also think about its
relationship with the collection returned by the
ComputeService.listAvailableLocations method [1]. It returns the list
of locations that can be used to spawn nodes, but I see it as a
general method of the locations made available to be managed by the
Compute abstraction.

* Should we return a location with Network scope for each existing network?
* Should we keep that method as-is and ask users to build a proper
network-scoped Location object themselves?
* How should we refine the semantics of that method so it is crystal
clear to users how to use it and what to expect?

I.

[1] https://github.com/jclouds/jclouds/blob/master/compute/src/main/java/org/jclouds/compute/ComputeService.java#L108-L118

P.S. Thanks to you Svet for bringing this discussion to the list!


On 19 June 2017 at 10:24, Geoff Macartney
<ge...@cloudsoftcorp.com> wrote:
> Ok Svet;
>
> It would certainly be a shame to have to write
>
> if (provider.equals("GCE")) {
>        createSecurityGroupOneWay();
> } else {
>        createSecurityGroupAnotherWay();
> }
>
> I guess I agree it's probably better, when the provider doesn't support a
> concept, to avoid having an implementation that's "nearly but not quite",
> otherwise you'd end up with lots of code like the above.
>
> G
>
> On Sun, 18 Jun 2017 at 15:23 Svetoslav Neykov <
> svetoslav.neykov@cloudsoftcorp.com> wrote:
>
>> > if that entity does not exist in GCE and
>> > users won't be able to use the extension as in other clouds, is there
>> > a real use case for it that justifies its implementation?
>>
>>
>> My personal drive for the implementation is to reuse existing code
>> managing the ingress access to (freshly spun up) nodes managed by jclouds.
>> So it would work nicely for my case :).
>>
>> What still irks me about the proposal is the different scope of the
>> location required to create the security group. It essentially means the
>> implementation:
>>   * is incompatible with existing code out in the wild - it will break it
>> when "securityGroupExtension.isPresent()" returns true,  but the
>> "createSecurityGroup" subsequently fails due to incorrect location argument
>>   * requires special casing in the code to pass the correct location scope
>> - doing template.getLocation() will not do it for GCE which is what I use
>> to pass as an argument to "createSecurityGroup"
>>
>> I took some time to think this through, that's why the delay in response,
>> but don't see a nice solution to ^^^. That's why I'm backing off from the
>> proposal. Still think it's been a useful discussion to have. Appreciate the
>> involvement Ignasi and Geoff.
>>
>>
>> Svet.
>>
>>
>>
>>
>>
>>
>> > On 16.06.2017 г., at 10:01, Ignasi Barrera <na...@apache.org> wrote:
>> >
>> > I agree it is a grey area where we can only do our best.
>> >
>> > I'd say it is ok to skip egress rules and to only consider the rules
>> > created by jclouds, that is something we could assume given the fact
>> > that the concept of security groups does not exist in GCE, which
>> > brings me to the question: if that entity does not exist in GCE and
>> > users won't be able to use the extension as in other clouds, is there
>> > a real use case for it that justifies its implementation?
>> >
>> > If the answer is yes, then let's move forward with your PoC. But I'd
>> > like to make sure that we have identified all conflicting points (I
>> > think we do). We can assume we're not going to support existing
>> > resources not related to jclouds, but as Geoff mentioned too I think
>> > we should try to support onboarding existing resources that were
>> > created with previous versions of jclouds with the current
>> > "inboundPorts" implementation.
>> >
>> > On 15 June 2017 at 15:59, Svetoslav Neykov
>> > <sv...@cloudsoftcorp.com> wrote:
>> >> The onboarding experience takes a best effort approach and never
>> represents the cloud state exactly. It's a gray area of compromises.
>> >> For example GCE supports ingress/egress; allow/deny rules. Of those
>> only ingress+allow rules can be represented using IpPermission. What do we
>> do about the rest? Just ignore, like we do for other clouds? Where do we
>> draw the line and say that it doesn't fit the abstraction so it's not
>> presented to the user? Taking this to the extreme we could decide that the
>> only rules that fit the abstraction are those created by jclouds.
>> >>
>> >> It would be fairly easy to come up with a solution which transforms the
>> firewall rules into a SecurityGroup (read SGs). What I'm worried about is
>> updating onboarded SGs. It could result in some unexpected changes.
>> >>
>> >> Re the security group created for inboundPorts (or anything managed by
>> jclouds) - it's something we expect and know what it looks like, how it's
>> used, so should be straightforward to accommodate in the design. It's the
>> unknown that's scary :).
>> >>
>> >> If we decide to represent existing firewall rules, here's how it could
>> work:
>> >>  * skip egress; deny rules
>> >>  * rules having a list of target tags
>> >>    - create a security group per tag per network
>> >>    - share a rule between security groups - one per tag per network
>> >>    - deleting an IpPermission is removing the tag from the firewall
>> rule (and deleting it if last one)
>> >>    - adding a rule is straightforward - create a permission with the
>> same destination tag as the security group ID
>> >>    - adding a SG to a node is adding the ID for the SG to the tags of
>> the node
>> >>  * rules having an "All instances in the network" target
>> >>    - group into a single SG (per network)
>> >>    - consider it a network SG and deny requests to attach it to a node
>> >>
>> >> I still think the "surprise factor" with the GCE implementation will be
>> the requirement to pass a network scope location to the
>> "createSecurityGroup" call. It's something that's different from the other
>> implementations and means it can't "just work" with existing users. If we
>> agree on a solution here, then I'm sure we'll make everything else work.
>> >>
>> >> Svet.
>> >>
>> >>
>> >>
>> >>
>> >>> On 15.06.2017 г., at 13:26, Geoff Macartney <
>> geoff.macartney@cloudsoftcorp.com> wrote:
>> >>>
>> >>> hi Ignasi,
>> >>>
>> >>> that's an interesting issue.  Re. your last paragraph, that could
>> certainly
>> >>> be a good approach, but it still won't address the question of
>> onboarding
>> >>> existing infrastructure, right?  i.e. where there are rules that
>> weren't
>> >>> written according to those conventions - or of upgrading to the new
>> jclouds
>> >>> version (with GCE security groups) to continue to manage infrastructure
>> >>> that _was_ created and provisioned by jclouds.
>> >>>
>> >>> I'll certainly need to have a think about it. As you say we could
>> refine
>> >>> and be explicit about how the proposal would work - perhaps come up
>> with a
>> >>> use-case that illustrates the scenarios, then we could more easily see
>> if
>> >>> there are gaps to fill.
>> >>>
>> >>> Geoff
>> >>>
>> >>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:
>> >>>
>> >>>> I understand the motivation behind ignoring existing stuff, but I have
>> >>>> some concerns.
>> >>>>
>> >>>> Onboarding existing cloud infrastructure is a valid use case for
>> >>>> jclouds users, and that can already be done (limited by the bounds of
>> >>>> the Compute abstraction) with the current jclouds implementation. It
>> >>>> is fair to expect that if the SecurityGroupExtension is present, and
>> >>>> there is a method called "listSecurityGroupsForNode" you should be
>> >>>> able to onboard the firewall rules of your existing node. As a user,
>> >>>> I'd have these expectations so I don't think we can discard the
>> >>>> onboarding use case just because we've found it difficult to map in a
>> >>>> consistent way.
>> >>>>
>> >>>> This said, if we went that path, I think we still need to refine the
>> >>>> proposal, or at least be explicit and enumerate all affected points
>> >>>> and how we plan to fill the gaps.
>> >>>>
>> >>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
>> >>>> With the actual code, the node would be behind a couple firewall
>> >>>> rules, but the security group extension won't be able to return any
>> >>>> security group for the node. That is inconsistent, as something you
>> >>>> created using the abstraction, cannot be retrieved with the same
>> >>>> information using the abstraction extension. It will give users the
>> >>>> impression that the node has no firewall rules applied.
>> >>>>
>> >>>> This case could be as simple as creating the firewall rules for the
>> >>>> "inboundPorts" following our SG naming and tag convention. I just want
>> >>>> to illustrate that the problem is not exclusive of the resources
>> >>>> created "outside" jclouds.
>> >>>>
>> >>>> On 15 June 2017 at 11:09, Svetoslav Neykov
>> >>>> <sv...@cloudsoftcorp.com> wrote:
>> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>> >>>> above:
>> >>>>>
>> >>>>> That's right, tags are an important part of the convention and I
>> didn't
>> >>>> expand on that. Your description nicely captures the idea.
>> >>>>> I believe the security groups will work across subnets in a single
>> vpc
>> >>>> network.
>> >>>>>
>> >>>>> Svet.
>> >>>>>
>> >>>>>
>> >>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
>> >>>> geoff.macartney@cloudsoftcorp.com> wrote:
>> >>>>>>
>> >>>>>> I like the sound of this proposal, I think it's certainly worth
>> >>>>>> investigating.
>> >>>>>>
>> >>>>>> Ignasi those are good questions you've asked.  Here's my thoughts:
>> >>>>>>
>> >>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and
>> be
>> >>>>>> explicit about that in documentation. What I mean here is that we
>> don't
>> >>>>>> need to do a "reverse mapping" of existing stuff back into jclouds
>> >>>>>> concepts. So if there are existing firewall rules they simply don't
>> >>>> show up
>> >>>>>> in the results of 'listSecurityGroups()",
>> >>>> "listSecurityGroupsInLocation()"
>> >>>>>> or "listSecurityGroupsForNode()".    That is, not all rules
>> participate
>> >>>> in
>> >>>>>> "security groups" from the jclouds point of view, only ones that
>> jclouds
>> >>>>>> added for security group purposes.
>> >>>>>>
>> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>> >>>> above:
>> >>>>>>
>> >>>>>> 1.  What do you think about the idea of using tags to identify
>> security
>> >>>>>> groups - each security group would correspond to a tag. These tags
>> would
>> >>>>>> correspond to groupIds in the IPPermission object, i.e. you could
>> >>>> create an
>> >>>>>> IPPermission using a groupId [1] which would relate to a tag in the
>> GCE
>> >>>>>> platform. Then any nodes with that tag (= in that security group)
>> would
>> >>>> be
>> >>>>>> permitted access.
>> >>>>>> 2. Are there any restrictions/implications relating to subnets in
>> your
>> >>>>>> proposal, e.g. would you need to use them to achieve what you want,
>> or
>> >>>> can
>> >>>>>> it work irrespective of any subnets on the network?
>> >>>>>>
>> >>>>>> Geoff
>> >>>>>>
>> >>>>>> [1]
>> >>>>>>
>> >>>>
>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>> >>>>>>
>> >>>>>>
>> >>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org>
>> wrote:
>> >>>>>>
>> >>>>>>> Thanks for starting this discussion Svet!
>> >>>>>>>
>> >>>>>>> It would be great to come up with a solution we are happy with.
>> >>>>>>>
>> >>>>>>> The main issue I see here is that the primary entity of the
>> extension,
>> >>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>> >>>>>>> the reasons we decided to remove the old implementation).
>> >>>>>>>
>> >>>>>>> I am OK applying conventions to implement the extension somehow,
>> but
>> >>>>>>> conventions are tricky when it comes to deal with existing stuff in
>> >>>>>>> the provider. With this proposal, what are your plans to implement
>> the
>> >>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>> >>>>>>> location)" methods? There is also a method to list the security
>> groups
>> >>>>>>> from a node. Given that the ComputeService is able to "listNodes"
>> that
>> >>>>>>> weren't created by jclouds, in case of a returned Node that already
>> >>>>>>> has firewall rules assigned but that don't fit in our naming
>> >>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>> >>>>>>> behave?
>> >>>>>>>
>> >>>>>>> I.
>> >>>>>>>
>> >>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>> >>>>>>> <sv...@cloudsoftcorp.com> wrote:
>> >>>>>>>> I'd like to try implement the SecurityGroupExtension interface for
>> >>>> GCE.
>> >>>>>>> Looking at the documentation it seems that the combination of
>> firewall
>> >>>>>>> rules and node tags is flexible enough to allow us implement the
>> >>>>>>> functionality.
>> >>>>>>>> It's been tried before but the implementation's been removed (see
>> >>>> [1]).
>> >>>>>>> It's main drawback was that for each security group the code
>> creates a
>> >>>> new
>> >>>>>>> network.
>> >>>>>>>> Currently the biggest mismatch between the jclouds abstraction
>> and the
>> >>>>>>> GCE functionality is that firewall rules must be attached to a
>> network.
>> >>>>>>>> Here's my suggested approach:
>> >>>>>>>> IpPermission roughly corresponds to a firewall rule
>> >>>>>>>> SecurityGroup is just a collection of firewall rules (there's no
>> cloud
>> >>>>>>> resource that corresponds to it). The firewall rules of a security
>> >>>> group
>> >>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>.
>> They
>> >>>> all
>> >>>>>>> belong to the same network.
>> >>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope
>> of
>> >>>>>>> Network, returning a custom implementation of SecurityGroup which
>> >>>> keeps a
>> >>>>>>> reference to the network, so all IpPermission objects added
>> >>>> subsequently
>> >>>>>>> will be on it.
>> >>>>>>>> While the suggested approach fits into the SecurityGroupExtension
>> >>>>>>> interface it's different enough from the other implementations
>> that it
>> >>>>>>> might not be worth the trouble of implementing and supporting
>> (even be
>> >>>>>>> harmful as users might be surprised by the different behaviour).
>> >>>>>>>> Would be interested in hearing other opinions on the approach.
>> I've
>> >>>>>>> created a Jira to track the effort at [2].
>> >>>>>>>>
>> >>>>>>>> Svet.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> [1]
>> >>>>>>>
>> >>>>
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>> >>>>>>> <
>> >>>>>>>
>> >>>>
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>> >>>>>>>>
>> >>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>> >>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>
>>
>>

Re: GCE SecurityGroupExtension implementation

[Geoff Macartney <ge...@cloudsoftcorp.com>]

Ok Svet;

It would certainly be a shame to have to write

if (provider.equals("GCE")) {
       createSecurityGroupOneWay();
} else {
       createSecurityGroupAnotherWay();
}

I guess I agree it's probably better, when the provider doesn't support a
concept, to avoid having an implementation that's "nearly but not quite",
otherwise you'd end up with lots of code like the above.

G

On Sun, 18 Jun 2017 at 15:23 Svetoslav Neykov <
svetoslav.neykov@cloudsoftcorp.com> wrote:

> > if that entity does not exist in GCE and
> > users won't be able to use the extension as in other clouds, is there
> > a real use case for it that justifies its implementation?
>
>
> My personal drive for the implementation is to reuse existing code
> managing the ingress access to (freshly spun up) nodes managed by jclouds.
> So it would work nicely for my case :).
>
> What still irks me about the proposal is the different scope of the
> location required to create the security group. It essentially means the
> implementation:
>   * is incompatible with existing code out in the wild - it will break it
> when "securityGroupExtension.isPresent()" returns true,  but the
> "createSecurityGroup" subsequently fails due to incorrect location argument
>   * requires special casing in the code to pass the correct location scope
> - doing template.getLocation() will not do it for GCE which is what I use
> to pass as an argument to "createSecurityGroup"
>
> I took some time to think this through, that's why the delay in response,
> but don't see a nice solution to ^^^. That's why I'm backing off from the
> proposal. Still think it's been a useful discussion to have. Appreciate the
> involvement Ignasi and Geoff.
>
>
> Svet.
>
>
>
>
>
>
> > On 16.06.2017 г., at 10:01, Ignasi Barrera <na...@apache.org> wrote:
> >
> > I agree it is a grey area where we can only do our best.
> >
> > I'd say it is ok to skip egress rules and to only consider the rules
> > created by jclouds, that is something we could assume given the fact
> > that the concept of security groups does not exist in GCE, which
> > brings me to the question: if that entity does not exist in GCE and
> > users won't be able to use the extension as in other clouds, is there
> > a real use case for it that justifies its implementation?
> >
> > If the answer is yes, then let's move forward with your PoC. But I'd
> > like to make sure that we have identified all conflicting points (I
> > think we do). We can assume we're not going to support existing
> > resources not related to jclouds, but as Geoff mentioned too I think
> > we should try to support onboarding existing resources that were
> > created with previous versions of jclouds with the current
> > "inboundPorts" implementation.
> >
> > On 15 June 2017 at 15:59, Svetoslav Neykov
> > <sv...@cloudsoftcorp.com> wrote:
> >> The onboarding experience takes a best effort approach and never
> represents the cloud state exactly. It's a gray area of compromises.
> >> For example GCE supports ingress/egress; allow/deny rules. Of those
> only ingress+allow rules can be represented using IpPermission. What do we
> do about the rest? Just ignore, like we do for other clouds? Where do we
> draw the line and say that it doesn't fit the abstraction so it's not
> presented to the user? Taking this to the extreme we could decide that the
> only rules that fit the abstraction are those created by jclouds.
> >>
> >> It would be fairly easy to come up with a solution which transforms the
> firewall rules into a SecurityGroup (read SGs). What I'm worried about is
> updating onboarded SGs. It could result in some unexpected changes.
> >>
> >> Re the security group created for inboundPorts (or anything managed by
> jclouds) - it's something we expect and know what it looks like, how it's
> used, so should be straightforward to accommodate in the design. It's the
> unknown that's scary :).
> >>
> >> If we decide to represent existing firewall rules, here's how it could
> work:
> >>  * skip egress; deny rules
> >>  * rules having a list of target tags
> >>    - create a security group per tag per network
> >>    - share a rule between security groups - one per tag per network
> >>    - deleting an IpPermission is removing the tag from the firewall
> rule (and deleting it if last one)
> >>    - adding a rule is straightforward - create a permission with the
> same destination tag as the security group ID
> >>    - adding a SG to a node is adding the ID for the SG to the tags of
> the node
> >>  * rules having an "All instances in the network" target
> >>    - group into a single SG (per network)
> >>    - consider it a network SG and deny requests to attach it to a node
> >>
> >> I still think the "surprise factor" with the GCE implementation will be
> the requirement to pass a network scope location to the
> "createSecurityGroup" call. It's something that's different from the other
> implementations and means it can't "just work" with existing users. If we
> agree on a solution here, then I'm sure we'll make everything else work.
> >>
> >> Svet.
> >>
> >>
> >>
> >>
> >>> On 15.06.2017 г., at 13:26, Geoff Macartney <
> geoff.macartney@cloudsoftcorp.com> wrote:
> >>>
> >>> hi Ignasi,
> >>>
> >>> that's an interesting issue.  Re. your last paragraph, that could
> certainly
> >>> be a good approach, but it still won't address the question of
> onboarding
> >>> existing infrastructure, right?  i.e. where there are rules that
> weren't
> >>> written according to those conventions - or of upgrading to the new
> jclouds
> >>> version (with GCE security groups) to continue to manage infrastructure
> >>> that _was_ created and provisioned by jclouds.
> >>>
> >>> I'll certainly need to have a think about it. As you say we could
> refine
> >>> and be explicit about how the proposal would work - perhaps come up
> with a
> >>> use-case that illustrates the scenarios, then we could more easily see
> if
> >>> there are gaps to fill.
> >>>
> >>> Geoff
> >>>
> >>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:
> >>>
> >>>> I understand the motivation behind ignoring existing stuff, but I have
> >>>> some concerns.
> >>>>
> >>>> Onboarding existing cloud infrastructure is a valid use case for
> >>>> jclouds users, and that can already be done (limited by the bounds of
> >>>> the Compute abstraction) with the current jclouds implementation. It
> >>>> is fair to expect that if the SecurityGroupExtension is present, and
> >>>> there is a method called "listSecurityGroupsForNode" you should be
> >>>> able to onboard the firewall rules of your existing node. As a user,
> >>>> I'd have these expectations so I don't think we can discard the
> >>>> onboarding use case just because we've found it difficult to map in a
> >>>> consistent way.
> >>>>
> >>>> This said, if we went that path, I think we still need to refine the
> >>>> proposal, or at least be explicit and enumerate all affected points
> >>>> and how we plan to fill the gaps.
> >>>>
> >>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
> >>>> With the actual code, the node would be behind a couple firewall
> >>>> rules, but the security group extension won't be able to return any
> >>>> security group for the node. That is inconsistent, as something you
> >>>> created using the abstraction, cannot be retrieved with the same
> >>>> information using the abstraction extension. It will give users the
> >>>> impression that the node has no firewall rules applied.
> >>>>
> >>>> This case could be as simple as creating the firewall rules for the
> >>>> "inboundPorts" following our SG naming and tag convention. I just want
> >>>> to illustrate that the problem is not exclusive of the resources
> >>>> created "outside" jclouds.
> >>>>
> >>>> On 15 June 2017 at 11:09, Svetoslav Neykov
> >>>> <sv...@cloudsoftcorp.com> wrote:
> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
> >>>> above:
> >>>>>
> >>>>> That's right, tags are an important part of the convention and I
> didn't
> >>>> expand on that. Your description nicely captures the idea.
> >>>>> I believe the security groups will work across subnets in a single
> vpc
> >>>> network.
> >>>>>
> >>>>> Svet.
> >>>>>
> >>>>>
> >>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
> >>>> geoff.macartney@cloudsoftcorp.com> wrote:
> >>>>>>
> >>>>>> I like the sound of this proposal, I think it's certainly worth
> >>>>>> investigating.
> >>>>>>
> >>>>>> Ignasi those are good questions you've asked.  Here's my thoughts:
> >>>>>>
> >>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and
> be
> >>>>>> explicit about that in documentation. What I mean here is that we
> don't
> >>>>>> need to do a "reverse mapping" of existing stuff back into jclouds
> >>>>>> concepts. So if there are existing firewall rules they simply don't
> >>>> show up
> >>>>>> in the results of 'listSecurityGroups()",
> >>>> "listSecurityGroupsInLocation()"
> >>>>>> or "listSecurityGroupsForNode()".    That is, not all rules
> participate
> >>>> in
> >>>>>> "security groups" from the jclouds point of view, only ones that
> jclouds
> >>>>>> added for security group purposes.
> >>>>>>
> >>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
> >>>> above:
> >>>>>>
> >>>>>> 1.  What do you think about the idea of using tags to identify
> security
> >>>>>> groups - each security group would correspond to a tag. These tags
> would
> >>>>>> correspond to groupIds in the IPPermission object, i.e. you could
> >>>> create an
> >>>>>> IPPermission using a groupId [1] which would relate to a tag in the
> GCE
> >>>>>> platform. Then any nodes with that tag (= in that security group)
> would
> >>>> be
> >>>>>> permitted access.
> >>>>>> 2. Are there any restrictions/implications relating to subnets in
> your
> >>>>>> proposal, e.g. would you need to use them to achieve what you want,
> or
> >>>> can
> >>>>>> it work irrespective of any subnets on the network?
> >>>>>>
> >>>>>> Geoff
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>
> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
> >>>>>>
> >>>>>>
> >>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org>
> wrote:
> >>>>>>
> >>>>>>> Thanks for starting this discussion Svet!
> >>>>>>>
> >>>>>>> It would be great to come up with a solution we are happy with.
> >>>>>>>
> >>>>>>> The main issue I see here is that the primary entity of the
> extension,
> >>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
> >>>>>>> the reasons we decided to remove the old implementation).
> >>>>>>>
> >>>>>>> I am OK applying conventions to implement the extension somehow,
> but
> >>>>>>> conventions are tricky when it comes to deal with existing stuff in
> >>>>>>> the provider. With this proposal, what are your plans to implement
> the
> >>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
> >>>>>>> location)" methods? There is also a method to list the security
> groups
> >>>>>>> from a node. Given that the ComputeService is able to "listNodes"
> that
> >>>>>>> weren't created by jclouds, in case of a returned Node that already
> >>>>>>> has firewall rules assigned but that don't fit in our naming
> >>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
> >>>>>>> behave?
> >>>>>>>
> >>>>>>> I.
> >>>>>>>
> >>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
> >>>>>>> <sv...@cloudsoftcorp.com> wrote:
> >>>>>>>> I'd like to try implement the SecurityGroupExtension interface for
> >>>> GCE.
> >>>>>>> Looking at the documentation it seems that the combination of
> firewall
> >>>>>>> rules and node tags is flexible enough to allow us implement the
> >>>>>>> functionality.
> >>>>>>>> It's been tried before but the implementation's been removed (see
> >>>> [1]).
> >>>>>>> It's main drawback was that for each security group the code
> creates a
> >>>> new
> >>>>>>> network.
> >>>>>>>> Currently the biggest mismatch between the jclouds abstraction
> and the
> >>>>>>> GCE functionality is that firewall rules must be attached to a
> network.
> >>>>>>>> Here's my suggested approach:
> >>>>>>>> IpPermission roughly corresponds to a firewall rule
> >>>>>>>> SecurityGroup is just a collection of firewall rules (there's no
> cloud
> >>>>>>> resource that corresponds to it). The firewall rules of a security
> >>>> group
> >>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>.
> They
> >>>> all
> >>>>>>> belong to the same network.
> >>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope
> of
> >>>>>>> Network, returning a custom implementation of SecurityGroup which
> >>>> keeps a
> >>>>>>> reference to the network, so all IpPermission objects added
> >>>> subsequently
> >>>>>>> will be on it.
> >>>>>>>> While the suggested approach fits into the SecurityGroupExtension
> >>>>>>> interface it's different enough from the other implementations
> that it
> >>>>>>> might not be worth the trouble of implementing and supporting
> (even be
> >>>>>>> harmful as users might be surprised by the different behaviour).
> >>>>>>>> Would be interested in hearing other opinions on the approach.
> I've
> >>>>>>> created a Jira to track the effort at [2].
> >>>>>>>>
> >>>>>>>> Svet.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>
> >>>>
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> >>>>>>> <
> >>>>>>>
> >>>>
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> >>>>>>>>
> >>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
> >>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>
>
>

Re: GCE SecurityGroupExtension implementation

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

> if that entity does not exist in GCE and
> users won't be able to use the extension as in other clouds, is there
> a real use case for it that justifies its implementation?


My personal drive for the implementation is to reuse existing code managing the ingress access to (freshly spun up) nodes managed by jclouds. So it would work nicely for my case :).

What still irks me about the proposal is the different scope of the location required to create the security group. It essentially means the implementation:
  * is incompatible with existing code out in the wild - it will break it when "securityGroupExtension.isPresent()" returns true,  but the "createSecurityGroup" subsequently fails due to incorrect location argument
  * requires special casing in the code to pass the correct location scope - doing template.getLocation() will not do it for GCE which is what I use to pass as an argument to "createSecurityGroup"

I took some time to think this through, that's why the delay in response, but don't see a nice solution to ^^^. That's why I'm backing off from the proposal. Still think it's been a useful discussion to have. Appreciate the involvement Ignasi and Geoff.


Svet.






> On 16.06.2017 г., at 10:01, Ignasi Barrera <na...@apache.org> wrote:
> 
> I agree it is a grey area where we can only do our best.
> 
> I'd say it is ok to skip egress rules and to only consider the rules
> created by jclouds, that is something we could assume given the fact
> that the concept of security groups does not exist in GCE, which
> brings me to the question: if that entity does not exist in GCE and
> users won't be able to use the extension as in other clouds, is there
> a real use case for it that justifies its implementation?
> 
> If the answer is yes, then let's move forward with your PoC. But I'd
> like to make sure that we have identified all conflicting points (I
> think we do). We can assume we're not going to support existing
> resources not related to jclouds, but as Geoff mentioned too I think
> we should try to support onboarding existing resources that were
> created with previous versions of jclouds with the current
> "inboundPorts" implementation.
> 
> On 15 June 2017 at 15:59, Svetoslav Neykov
> <sv...@cloudsoftcorp.com> wrote:
>> The onboarding experience takes a best effort approach and never represents the cloud state exactly. It's a gray area of compromises.
>> For example GCE supports ingress/egress; allow/deny rules. Of those only ingress+allow rules can be represented using IpPermission. What do we do about the rest? Just ignore, like we do for other clouds? Where do we draw the line and say that it doesn't fit the abstraction so it's not presented to the user? Taking this to the extreme we could decide that the only rules that fit the abstraction are those created by jclouds.
>> 
>> It would be fairly easy to come up with a solution which transforms the firewall rules into a SecurityGroup (read SGs). What I'm worried about is updating onboarded SGs. It could result in some unexpected changes.
>> 
>> Re the security group created for inboundPorts (or anything managed by jclouds) - it's something we expect and know what it looks like, how it's used, so should be straightforward to accommodate in the design. It's the unknown that's scary :).
>> 
>> If we decide to represent existing firewall rules, here's how it could work:
>>  * skip egress; deny rules
>>  * rules having a list of target tags
>>    - create a security group per tag per network
>>    - share a rule between security groups - one per tag per network
>>    - deleting an IpPermission is removing the tag from the firewall rule (and deleting it if last one)
>>    - adding a rule is straightforward - create a permission with the same destination tag as the security group ID
>>    - adding a SG to a node is adding the ID for the SG to the tags of the node
>>  * rules having an "All instances in the network" target
>>    - group into a single SG (per network)
>>    - consider it a network SG and deny requests to attach it to a node
>> 
>> I still think the "surprise factor" with the GCE implementation will be the requirement to pass a network scope location to the "createSecurityGroup" call. It's something that's different from the other implementations and means it can't "just work" with existing users. If we agree on a solution here, then I'm sure we'll make everything else work.
>> 
>> Svet.
>> 
>> 
>> 
>> 
>>> On 15.06.2017 г., at 13:26, Geoff Macartney <ge...@cloudsoftcorp.com> wrote:
>>> 
>>> hi Ignasi,
>>> 
>>> that's an interesting issue.  Re. your last paragraph, that could certainly
>>> be a good approach, but it still won't address the question of onboarding
>>> existing infrastructure, right?  i.e. where there are rules that weren't
>>> written according to those conventions - or of upgrading to the new jclouds
>>> version (with GCE security groups) to continue to manage infrastructure
>>> that _was_ created and provisioned by jclouds.
>>> 
>>> I'll certainly need to have a think about it. As you say we could refine
>>> and be explicit about how the proposal would work - perhaps come up with a
>>> use-case that illustrates the scenarios, then we could more easily see if
>>> there are gaps to fill.
>>> 
>>> Geoff
>>> 
>>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:
>>> 
>>>> I understand the motivation behind ignoring existing stuff, but I have
>>>> some concerns.
>>>> 
>>>> Onboarding existing cloud infrastructure is a valid use case for
>>>> jclouds users, and that can already be done (limited by the bounds of
>>>> the Compute abstraction) with the current jclouds implementation. It
>>>> is fair to expect that if the SecurityGroupExtension is present, and
>>>> there is a method called "listSecurityGroupsForNode" you should be
>>>> able to onboard the firewall rules of your existing node. As a user,
>>>> I'd have these expectations so I don't think we can discard the
>>>> onboarding use case just because we've found it difficult to map in a
>>>> consistent way.
>>>> 
>>>> This said, if we went that path, I think we still need to refine the
>>>> proposal, or at least be explicit and enumerate all affected points
>>>> and how we plan to fill the gaps.
>>>> 
>>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
>>>> With the actual code, the node would be behind a couple firewall
>>>> rules, but the security group extension won't be able to return any
>>>> security group for the node. That is inconsistent, as something you
>>>> created using the abstraction, cannot be retrieved with the same
>>>> information using the abstraction extension. It will give users the
>>>> impression that the node has no firewall rules applied.
>>>> 
>>>> This case could be as simple as creating the firewall rules for the
>>>> "inboundPorts" following our SG naming and tag convention. I just want
>>>> to illustrate that the problem is not exclusive of the resources
>>>> created "outside" jclouds.
>>>> 
>>>> On 15 June 2017 at 11:09, Svetoslav Neykov
>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>>> above:
>>>>> 
>>>>> That's right, tags are an important part of the convention and I didn't
>>>> expand on that. Your description nicely captures the idea.
>>>>> I believe the security groups will work across subnets in a single vpc
>>>> network.
>>>>> 
>>>>> Svet.
>>>>> 
>>>>> 
>>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
>>>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>>>> 
>>>>>> I like the sound of this proposal, I think it's certainly worth
>>>>>> investigating.
>>>>>> 
>>>>>> Ignasi those are good questions you've asked.  Here's my thoughts:
>>>>>> 
>>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and be
>>>>>> explicit about that in documentation. What I mean here is that we don't
>>>>>> need to do a "reverse mapping" of existing stuff back into jclouds
>>>>>> concepts. So if there are existing firewall rules they simply don't
>>>> show up
>>>>>> in the results of 'listSecurityGroups()",
>>>> "listSecurityGroupsInLocation()"
>>>>>> or "listSecurityGroupsForNode()".    That is, not all rules participate
>>>> in
>>>>>> "security groups" from the jclouds point of view, only ones that jclouds
>>>>>> added for security group purposes.
>>>>>> 
>>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>>> above:
>>>>>> 
>>>>>> 1.  What do you think about the idea of using tags to identify security
>>>>>> groups - each security group would correspond to a tag. These tags would
>>>>>> correspond to groupIds in the IPPermission object, i.e. you could
>>>> create an
>>>>>> IPPermission using a groupId [1] which would relate to a tag in the GCE
>>>>>> platform. Then any nodes with that tag (= in that security group) would
>>>> be
>>>>>> permitted access.
>>>>>> 2. Are there any restrictions/implications relating to subnets in your
>>>>>> proposal, e.g. would you need to use them to achieve what you want, or
>>>> can
>>>>>> it work irrespective of any subnets on the network?
>>>>>> 
>>>>>> Geoff
>>>>>> 
>>>>>> [1]
>>>>>> 
>>>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>>>>>> 
>>>>>> 
>>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org> wrote:
>>>>>> 
>>>>>>> Thanks for starting this discussion Svet!
>>>>>>> 
>>>>>>> It would be great to come up with a solution we are happy with.
>>>>>>> 
>>>>>>> The main issue I see here is that the primary entity of the extension,
>>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>>>>>>> the reasons we decided to remove the old implementation).
>>>>>>> 
>>>>>>> I am OK applying conventions to implement the extension somehow, but
>>>>>>> conventions are tricky when it comes to deal with existing stuff in
>>>>>>> the provider. With this proposal, what are your plans to implement the
>>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>>>>>>> location)" methods? There is also a method to list the security groups
>>>>>>> from a node. Given that the ComputeService is able to "listNodes" that
>>>>>>> weren't created by jclouds, in case of a returned Node that already
>>>>>>> has firewall rules assigned but that don't fit in our naming
>>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>>>>>>> behave?
>>>>>>> 
>>>>>>> I.
>>>>>>> 
>>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>>>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>>>> I'd like to try implement the SecurityGroupExtension interface for
>>>> GCE.
>>>>>>> Looking at the documentation it seems that the combination of firewall
>>>>>>> rules and node tags is flexible enough to allow us implement the
>>>>>>> functionality.
>>>>>>>> It's been tried before but the implementation's been removed (see
>>>> [1]).
>>>>>>> It's main drawback was that for each security group the code creates a
>>>> new
>>>>>>> network.
>>>>>>>> Currently the biggest mismatch between the jclouds abstraction and the
>>>>>>> GCE functionality is that firewall rules must be attached to a network.
>>>>>>>> Here's my suggested approach:
>>>>>>>> IpPermission roughly corresponds to a firewall rule
>>>>>>>> SecurityGroup is just a collection of firewall rules (there's no cloud
>>>>>>> resource that corresponds to it). The firewall rules of a security
>>>> group
>>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They
>>>> all
>>>>>>> belong to the same network.
>>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope of
>>>>>>> Network, returning a custom implementation of SecurityGroup which
>>>> keeps a
>>>>>>> reference to the network, so all IpPermission objects added
>>>> subsequently
>>>>>>> will be on it.
>>>>>>>> While the suggested approach fits into the SecurityGroupExtension
>>>>>>> interface it's different enough from the other implementations that it
>>>>>>> might not be worth the trouble of implementing and supporting (even be
>>>>>>> harmful as users might be surprised by the different behaviour).
>>>>>>>> Would be interested in hearing other opinions on the approach. I've
>>>>>>> created a Jira to track the effort at [2].
>>>>>>>> 
>>>>>>>> Svet.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> [1]
>>>>>>> 
>>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>> <
>>>>>>> 
>>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>>> 
>>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>> 

Re: GCE SecurityGroupExtension implementation

[Ignasi Barrera <na...@apache.org>]

I agree it is a grey area where we can only do our best.

I'd say it is ok to skip egress rules and to only consider the rules
created by jclouds, that is something we could assume given the fact
that the concept of security groups does not exist in GCE, which
brings me to the question: if that entity does not exist in GCE and
users won't be able to use the extension as in other clouds, is there
a real use case for it that justifies its implementation?

If the answer is yes, then let's move forward with your PoC. But I'd
like to make sure that we have identified all conflicting points (I
think we do). We can assume we're not going to support existing
resources not related to jclouds, but as Geoff mentioned too I think
we should try to support onboarding existing resources that were
created with previous versions of jclouds with the current
"inboundPorts" implementation.

On 15 June 2017 at 15:59, Svetoslav Neykov
<sv...@cloudsoftcorp.com> wrote:
> The onboarding experience takes a best effort approach and never represents the cloud state exactly. It's a gray area of compromises.
> For example GCE supports ingress/egress; allow/deny rules. Of those only ingress+allow rules can be represented using IpPermission. What do we do about the rest? Just ignore, like we do for other clouds? Where do we draw the line and say that it doesn't fit the abstraction so it's not presented to the user? Taking this to the extreme we could decide that the only rules that fit the abstraction are those created by jclouds.
>
> It would be fairly easy to come up with a solution which transforms the firewall rules into a SecurityGroup (read SGs). What I'm worried about is updating onboarded SGs. It could result in some unexpected changes.
>
> Re the security group created for inboundPorts (or anything managed by jclouds) - it's something we expect and know what it looks like, how it's used, so should be straightforward to accommodate in the design. It's the unknown that's scary :).
>
> If we decide to represent existing firewall rules, here's how it could work:
>   * skip egress; deny rules
>   * rules having a list of target tags
>     - create a security group per tag per network
>     - share a rule between security groups - one per tag per network
>     - deleting an IpPermission is removing the tag from the firewall rule (and deleting it if last one)
>     - adding a rule is straightforward - create a permission with the same destination tag as the security group ID
>     - adding a SG to a node is adding the ID for the SG to the tags of the node
>   * rules having an "All instances in the network" target
>     - group into a single SG (per network)
>     - consider it a network SG and deny requests to attach it to a node
>
> I still think the "surprise factor" with the GCE implementation will be the requirement to pass a network scope location to the "createSecurityGroup" call. It's something that's different from the other implementations and means it can't "just work" with existing users. If we agree on a solution here, then I'm sure we'll make everything else work.
>
> Svet.
>
>
>
>
>> On 15.06.2017 г., at 13:26, Geoff Macartney <ge...@cloudsoftcorp.com> wrote:
>>
>> hi Ignasi,
>>
>> that's an interesting issue.  Re. your last paragraph, that could certainly
>> be a good approach, but it still won't address the question of onboarding
>> existing infrastructure, right?  i.e. where there are rules that weren't
>> written according to those conventions - or of upgrading to the new jclouds
>> version (with GCE security groups) to continue to manage infrastructure
>> that _was_ created and provisioned by jclouds.
>>
>> I'll certainly need to have a think about it. As you say we could refine
>> and be explicit about how the proposal would work - perhaps come up with a
>> use-case that illustrates the scenarios, then we could more easily see if
>> there are gaps to fill.
>>
>> Geoff
>>
>> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:
>>
>>> I understand the motivation behind ignoring existing stuff, but I have
>>> some concerns.
>>>
>>> Onboarding existing cloud infrastructure is a valid use case for
>>> jclouds users, and that can already be done (limited by the bounds of
>>> the Compute abstraction) with the current jclouds implementation. It
>>> is fair to expect that if the SecurityGroupExtension is present, and
>>> there is a method called "listSecurityGroupsForNode" you should be
>>> able to onboard the firewall rules of your existing node. As a user,
>>> I'd have these expectations so I don't think we can discard the
>>> onboarding use case just because we've found it difficult to map in a
>>> consistent way.
>>>
>>> This said, if we went that path, I think we still need to refine the
>>> proposal, or at least be explicit and enumerate all affected points
>>> and how we plan to fill the gaps.
>>>
>>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
>>> With the actual code, the node would be behind a couple firewall
>>> rules, but the security group extension won't be able to return any
>>> security group for the node. That is inconsistent, as something you
>>> created using the abstraction, cannot be retrieved with the same
>>> information using the abstraction extension. It will give users the
>>> impression that the node has no firewall rules applied.
>>>
>>> This case could be as simple as creating the firewall rules for the
>>> "inboundPorts" following our SG naming and tag convention. I just want
>>> to illustrate that the problem is not exclusive of the resources
>>> created "outside" jclouds.
>>>
>>> On 15 June 2017 at 11:09, Svetoslav Neykov
>>> <sv...@cloudsoftcorp.com> wrote:
>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>> above:
>>>>
>>>> That's right, tags are an important part of the convention and I didn't
>>> expand on that. Your description nicely captures the idea.
>>>> I believe the security groups will work across subnets in a single vpc
>>> network.
>>>>
>>>> Svet.
>>>>
>>>>
>>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
>>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>>>
>>>>> I like the sound of this proposal, I think it's certainly worth
>>>>> investigating.
>>>>>
>>>>> Ignasi those are good questions you've asked.  Here's my thoughts:
>>>>>
>>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and be
>>>>> explicit about that in documentation. What I mean here is that we don't
>>>>> need to do a "reverse mapping" of existing stuff back into jclouds
>>>>> concepts. So if there are existing firewall rules they simply don't
>>> show up
>>>>> in the results of 'listSecurityGroups()",
>>> "listSecurityGroupsInLocation()"
>>>>> or "listSecurityGroupsForNode()".    That is, not all rules participate
>>> in
>>>>> "security groups" from the jclouds point of view, only ones that jclouds
>>>>> added for security group purposes.
>>>>>
>>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>>> above:
>>>>>
>>>>> 1.  What do you think about the idea of using tags to identify security
>>>>> groups - each security group would correspond to a tag. These tags would
>>>>> correspond to groupIds in the IPPermission object, i.e. you could
>>> create an
>>>>> IPPermission using a groupId [1] which would relate to a tag in the GCE
>>>>> platform. Then any nodes with that tag (= in that security group) would
>>> be
>>>>> permitted access.
>>>>> 2. Are there any restrictions/implications relating to subnets in your
>>>>> proposal, e.g. would you need to use them to achieve what you want, or
>>> can
>>>>> it work irrespective of any subnets on the network?
>>>>>
>>>>> Geoff
>>>>>
>>>>> [1]
>>>>>
>>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>>>>>
>>>>>
>>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org> wrote:
>>>>>
>>>>>> Thanks for starting this discussion Svet!
>>>>>>
>>>>>> It would be great to come up with a solution we are happy with.
>>>>>>
>>>>>> The main issue I see here is that the primary entity of the extension,
>>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>>>>>> the reasons we decided to remove the old implementation).
>>>>>>
>>>>>> I am OK applying conventions to implement the extension somehow, but
>>>>>> conventions are tricky when it comes to deal with existing stuff in
>>>>>> the provider. With this proposal, what are your plans to implement the
>>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>>>>>> location)" methods? There is also a method to list the security groups
>>>>>> from a node. Given that the ComputeService is able to "listNodes" that
>>>>>> weren't created by jclouds, in case of a returned Node that already
>>>>>> has firewall rules assigned but that don't fit in our naming
>>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>>>>>> behave?
>>>>>>
>>>>>> I.
>>>>>>
>>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>>> I'd like to try implement the SecurityGroupExtension interface for
>>> GCE.
>>>>>> Looking at the documentation it seems that the combination of firewall
>>>>>> rules and node tags is flexible enough to allow us implement the
>>>>>> functionality.
>>>>>>> It's been tried before but the implementation's been removed (see
>>> [1]).
>>>>>> It's main drawback was that for each security group the code creates a
>>> new
>>>>>> network.
>>>>>>> Currently the biggest mismatch between the jclouds abstraction and the
>>>>>> GCE functionality is that firewall rules must be attached to a network.
>>>>>>> Here's my suggested approach:
>>>>>>> IpPermission roughly corresponds to a firewall rule
>>>>>>> SecurityGroup is just a collection of firewall rules (there's no cloud
>>>>>> resource that corresponds to it). The firewall rules of a security
>>> group
>>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They
>>> all
>>>>>> belong to the same network.
>>>>>>> They key bit: createSecurityGroup accepts a Location with a scope of
>>>>>> Network, returning a custom implementation of SecurityGroup which
>>> keeps a
>>>>>> reference to the network, so all IpPermission objects added
>>> subsequently
>>>>>> will be on it.
>>>>>>> While the suggested approach fits into the SecurityGroupExtension
>>>>>> interface it's different enough from the other implementations that it
>>>>>> might not be worth the trouble of implementing and supporting (even be
>>>>>> harmful as users might be surprised by the different behaviour).
>>>>>>> Would be interested in hearing other opinions on the approach. I've
>>>>>> created a Jira to track the effort at [2].
>>>>>>>
>>>>>>> Svet.
>>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>
>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>> <
>>>>>>
>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>>>
>>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>>>>>>
>>>>>>
>>>>
>>>
>

Re: GCE SecurityGroupExtension implementation

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

The onboarding experience takes a best effort approach and never represents the cloud state exactly. It's a gray area of compromises.
For example GCE supports ingress/egress; allow/deny rules. Of those only ingress+allow rules can be represented using IpPermission. What do we do about the rest? Just ignore, like we do for other clouds? Where do we draw the line and say that it doesn't fit the abstraction so it's not presented to the user? Taking this to the extreme we could decide that the only rules that fit the abstraction are those created by jclouds.

It would be fairly easy to come up with a solution which transforms the firewall rules into a SecurityGroup (read SGs). What I'm worried about is updating onboarded SGs. It could result in some unexpected changes.

Re the security group created for inboundPorts (or anything managed by jclouds) - it's something we expect and know what it looks like, how it's used, so should be straightforward to accommodate in the design. It's the unknown that's scary :).

If we decide to represent existing firewall rules, here's how it could work:
  * skip egress; deny rules
  * rules having a list of target tags
    - create a security group per tag per network
    - share a rule between security groups - one per tag per network
    - deleting an IpPermission is removing the tag from the firewall rule (and deleting it if last one)
    - adding a rule is straightforward - create a permission with the same destination tag as the security group ID
    - adding a SG to a node is adding the ID for the SG to the tags of the node
  * rules having an "All instances in the network" target
    - group into a single SG (per network)
    - consider it a network SG and deny requests to attach it to a node
   
I still think the "surprise factor" with the GCE implementation will be the requirement to pass a network scope location to the "createSecurityGroup" call. It's something that's different from the other implementations and means it can't "just work" with existing users. If we agree on a solution here, then I'm sure we'll make everything else work.

Svet.




> On 15.06.2017 г., at 13:26, Geoff Macartney <ge...@cloudsoftcorp.com> wrote:
> 
> hi Ignasi,
> 
> that's an interesting issue.  Re. your last paragraph, that could certainly
> be a good approach, but it still won't address the question of onboarding
> existing infrastructure, right?  i.e. where there are rules that weren't
> written according to those conventions - or of upgrading to the new jclouds
> version (with GCE security groups) to continue to manage infrastructure
> that _was_ created and provisioned by jclouds.
> 
> I'll certainly need to have a think about it. As you say we could refine
> and be explicit about how the proposal would work - perhaps come up with a
> use-case that illustrates the scenarios, then we could more easily see if
> there are gaps to fill.
> 
> Geoff
> 
> On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:
> 
>> I understand the motivation behind ignoring existing stuff, but I have
>> some concerns.
>> 
>> Onboarding existing cloud infrastructure is a valid use case for
>> jclouds users, and that can already be done (limited by the bounds of
>> the Compute abstraction) with the current jclouds implementation. It
>> is fair to expect that if the SecurityGroupExtension is present, and
>> there is a method called "listSecurityGroupsForNode" you should be
>> able to onboard the firewall rules of your existing node. As a user,
>> I'd have these expectations so I don't think we can discard the
>> onboarding use case just because we've found it difficult to map in a
>> consistent way.
>> 
>> This said, if we went that path, I think we still need to refine the
>> proposal, or at least be explicit and enumerate all affected points
>> and how we plan to fill the gaps.
>> 
>> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
>> With the actual code, the node would be behind a couple firewall
>> rules, but the security group extension won't be able to return any
>> security group for the node. That is inconsistent, as something you
>> created using the abstraction, cannot be retrieved with the same
>> information using the abstraction extension. It will give users the
>> impression that the node has no firewall rules applied.
>> 
>> This case could be as simple as creating the firewall rules for the
>> "inboundPorts" following our SG naming and tag convention. I just want
>> to illustrate that the problem is not exclusive of the resources
>> created "outside" jclouds.
>> 
>> On 15 June 2017 at 11:09, Svetoslav Neykov
>> <sv...@cloudsoftcorp.com> wrote:
>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>> above:
>>> 
>>> That's right, tags are an important part of the convention and I didn't
>> expand on that. Your description nicely captures the idea.
>>> I believe the security groups will work across subnets in a single vpc
>> network.
>>> 
>>> Svet.
>>> 
>>> 
>>>> On 15.06.2017 г., at 11:57, Geoff Macartney <
>> geoff.macartney@cloudsoftcorp.com> wrote:
>>>> 
>>>> I like the sound of this proposal, I think it's certainly worth
>>>> investigating.
>>>> 
>>>> Ignasi those are good questions you've asked.  Here's my thoughts:
>>>> 
>>>> We can regard 'security groups' on GCE as a jclouds "overlay", and be
>>>> explicit about that in documentation. What I mean here is that we don't
>>>> need to do a "reverse mapping" of existing stuff back into jclouds
>>>> concepts. So if there are existing firewall rules they simply don't
>> show up
>>>> in the results of 'listSecurityGroups()",
>> "listSecurityGroupsInLocation()"
>>>> or "listSecurityGroupsForNode()".    That is, not all rules participate
>> in
>>>> "security groups" from the jclouds point of view, only ones that jclouds
>>>> added for security group purposes.
>>>> 
>>>> Svet you don't mention GCE "tags" or the question of subnets in the
>> above:
>>>> 
>>>> 1.  What do you think about the idea of using tags to identify security
>>>> groups - each security group would correspond to a tag. These tags would
>>>> correspond to groupIds in the IPPermission object, i.e. you could
>> create an
>>>> IPPermission using a groupId [1] which would relate to a tag in the GCE
>>>> platform. Then any nodes with that tag (= in that security group) would
>> be
>>>> permitted access.
>>>> 2. Are there any restrictions/implications relating to subnets in your
>>>> proposal, e.g. would you need to use them to achieve what you want, or
>> can
>>>> it work irrespective of any subnets on the network?
>>>> 
>>>> Geoff
>>>> 
>>>> [1]
>>>> 
>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>>>> 
>>>> 
>>>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org> wrote:
>>>> 
>>>>> Thanks for starting this discussion Svet!
>>>>> 
>>>>> It would be great to come up with a solution we are happy with.
>>>>> 
>>>>> The main issue I see here is that the primary entity of the extension,
>>>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>>>>> the reasons we decided to remove the old implementation).
>>>>> 
>>>>> I am OK applying conventions to implement the extension somehow, but
>>>>> conventions are tricky when it comes to deal with existing stuff in
>>>>> the provider. With this proposal, what are your plans to implement the
>>>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>>>>> location)" methods? There is also a method to list the security groups
>>>>> from a node. Given that the ComputeService is able to "listNodes" that
>>>>> weren't created by jclouds, in case of a returned Node that already
>>>>> has firewall rules assigned but that don't fit in our naming
>>>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>>>>> behave?
>>>>> 
>>>>> I.
>>>>> 
>>>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>>>>> <sv...@cloudsoftcorp.com> wrote:
>>>>>> I'd like to try implement the SecurityGroupExtension interface for
>> GCE.
>>>>> Looking at the documentation it seems that the combination of firewall
>>>>> rules and node tags is flexible enough to allow us implement the
>>>>> functionality.
>>>>>> It's been tried before but the implementation's been removed (see
>> [1]).
>>>>> It's main drawback was that for each security group the code creates a
>> new
>>>>> network.
>>>>>> Currently the biggest mismatch between the jclouds abstraction and the
>>>>> GCE functionality is that firewall rules must be attached to a network.
>>>>>> Here's my suggested approach:
>>>>>> IpPermission roughly corresponds to a firewall rule
>>>>>> SecurityGroup is just a collection of firewall rules (there's no cloud
>>>>> resource that corresponds to it). The firewall rules of a security
>> group
>>>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They
>> all
>>>>> belong to the same network.
>>>>>> They key bit: createSecurityGroup accepts a Location with a scope of
>>>>> Network, returning a custom implementation of SecurityGroup which
>> keeps a
>>>>> reference to the network, so all IpPermission objects added
>> subsequently
>>>>> will be on it.
>>>>>> While the suggested approach fits into the SecurityGroupExtension
>>>>> interface it's different enough from the other implementations that it
>>>>> might not be worth the trouble of implementing and supporting (even be
>>>>> harmful as users might be surprised by the different behaviour).
>>>>>> Would be interested in hearing other opinions on the approach. I've
>>>>> created a Jira to track the effort at [2].
>>>>>> 
>>>>>> Svet.
>>>>>> 
>>>>>> 
>>>>>> [1]
>>>>> 
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>> <
>>>>> 
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>>> 
>>>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>>>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>>>>> 
>>>>> 
>>> 
>> 

Re: GCE SecurityGroupExtension implementation

[Geoff Macartney <ge...@cloudsoftcorp.com>]

hi Ignasi,

that's an interesting issue.  Re. your last paragraph, that could certainly
be a good approach, but it still won't address the question of onboarding
existing infrastructure, right?  i.e. where there are rules that weren't
written according to those conventions - or of upgrading to the new jclouds
version (with GCE security groups) to continue to manage infrastructure
that _was_ created and provisioned by jclouds.

I'll certainly need to have a think about it. As you say we could refine
and be explicit about how the proposal would work - perhaps come up with a
use-case that illustrates the scenarios, then we could more easily see if
there are gaps to fill.

Geoff

On Thu, 15 Jun 2017 at 11:14 Ignasi Barrera <na...@apache.org> wrote:

> I understand the motivation behind ignoring existing stuff, but I have
> some concerns.
>
> Onboarding existing cloud infrastructure is a valid use case for
> jclouds users, and that can already be done (limited by the bounds of
> the Compute abstraction) with the current jclouds implementation. It
> is fair to expect that if the SecurityGroupExtension is present, and
> there is a method called "listSecurityGroupsForNode" you should be
> able to onboard the firewall rules of your existing node. As a user,
> I'd have these expectations so I don't think we can discard the
> onboarding use case just because we've found it difficult to map in a
> consistent way.
>
> This said, if we went that path, I think we still need to refine the
> proposal, or at least be explicit and enumerate all affected points
> and how we plan to fill the gaps.
>
> Say you create a node in jclouds with "options.inboundPorts(22, 80)".
> With the actual code, the node would be behind a couple firewall
> rules, but the security group extension won't be able to return any
> security group for the node. That is inconsistent, as something you
> created using the abstraction, cannot be retrieved with the same
> information using the abstraction extension. It will give users the
> impression that the node has no firewall rules applied.
>
> This case could be as simple as creating the firewall rules for the
> "inboundPorts" following our SG naming and tag convention. I just want
> to illustrate that the problem is not exclusive of the resources
> created "outside" jclouds.
>
> On 15 June 2017 at 11:09, Svetoslav Neykov
> <sv...@cloudsoftcorp.com> wrote:
> >> Svet you don't mention GCE "tags" or the question of subnets in the
> above:
> >
> > That's right, tags are an important part of the convention and I didn't
> expand on that. Your description nicely captures the idea.
> > I believe the security groups will work across subnets in a single vpc
> network.
> >
> > Svet.
> >
> >
> >> On 15.06.2017 г., at 11:57, Geoff Macartney <
> geoff.macartney@cloudsoftcorp.com> wrote:
> >>
> >> I like the sound of this proposal, I think it's certainly worth
> >> investigating.
> >>
> >> Ignasi those are good questions you've asked.  Here's my thoughts:
> >>
> >> We can regard 'security groups' on GCE as a jclouds "overlay", and be
> >> explicit about that in documentation. What I mean here is that we don't
> >> need to do a "reverse mapping" of existing stuff back into jclouds
> >> concepts. So if there are existing firewall rules they simply don't
> show up
> >> in the results of 'listSecurityGroups()",
> "listSecurityGroupsInLocation()"
> >> or "listSecurityGroupsForNode()".    That is, not all rules participate
> in
> >> "security groups" from the jclouds point of view, only ones that jclouds
> >> added for security group purposes.
> >>
> >> Svet you don't mention GCE "tags" or the question of subnets in the
> above:
> >>
> >> 1.  What do you think about the idea of using tags to identify security
> >> groups - each security group would correspond to a tag. These tags would
> >> correspond to groupIds in the IPPermission object, i.e. you could
> create an
> >> IPPermission using a groupId [1] which would relate to a tag in the GCE
> >> platform. Then any nodes with that tag (= in that security group) would
> be
> >> permitted access.
> >> 2. Are there any restrictions/implications relating to subnets in your
> >> proposal, e.g. would you need to use them to achieve what you want, or
> can
> >> it work irrespective of any subnets on the network?
> >>
> >> Geoff
> >>
> >> [1]
> >>
> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
> >>
> >>
> >> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org> wrote:
> >>
> >>> Thanks for starting this discussion Svet!
> >>>
> >>> It would be great to come up with a solution we are happy with.
> >>>
> >>> The main issue I see here is that the primary entity of the extension,
> >>> the SecurityGroup, does not have a mapping in GCE (which was one of
> >>> the reasons we decided to remove the old implementation).
> >>>
> >>> I am OK applying conventions to implement the extension somehow, but
> >>> conventions are tricky when it comes to deal with existing stuff in
> >>> the provider. With this proposal, what are your plans to implement the
> >>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
> >>> location)" methods? There is also a method to list the security groups
> >>> from a node. Given that the ComputeService is able to "listNodes" that
> >>> weren't created by jclouds, in case of a returned Node that already
> >>> has firewall rules assigned but that don't fit in our naming
> >>> conventions, how would the "listSecurityGroupsForNode(String id)"
> >>> behave?
> >>>
> >>> I.
> >>>
> >>> On 15 June 2017 at 09:03, Svetoslav Neykov
> >>> <sv...@cloudsoftcorp.com> wrote:
> >>>> I'd like to try implement the SecurityGroupExtension interface for
> GCE.
> >>> Looking at the documentation it seems that the combination of firewall
> >>> rules and node tags is flexible enough to allow us implement the
> >>> functionality.
> >>>> It's been tried before but the implementation's been removed (see
> [1]).
> >>> It's main drawback was that for each security group the code creates a
> new
> >>> network.
> >>>> Currently the biggest mismatch between the jclouds abstraction and the
> >>> GCE functionality is that firewall rules must be attached to a network.
> >>>> Here's my suggested approach:
> >>>> IpPermission roughly corresponds to a firewall rule
> >>>> SecurityGroup is just a collection of firewall rules (there's no cloud
> >>> resource that corresponds to it). The firewall rules of a security
> group
> >>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They
> all
> >>> belong to the same network.
> >>>> They key bit: createSecurityGroup accepts a Location with a scope of
> >>> Network, returning a custom implementation of SecurityGroup which
> keeps a
> >>> reference to the network, so all IpPermission objects added
> subsequently
> >>> will be on it.
> >>>> While the suggested approach fits into the SecurityGroupExtension
> >>> interface it's different enough from the other implementations that it
> >>> might not be worth the trouble of implementing and supporting (even be
> >>> harmful as users might be surprised by the different behaviour).
> >>>> Would be interested in hearing other opinions on the approach. I've
> >>> created a Jira to track the effort at [2].
> >>>>
> >>>> Svet.
> >>>>
> >>>>
> >>>> [1]
> >>>
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> >>> <
> >>>
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> >>>>
> >>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
> >>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
> >>>>
> >>>
> >
>

Re: GCE SecurityGroupExtension implementation

[Ignasi Barrera <na...@apache.org>]

I understand the motivation behind ignoring existing stuff, but I have
some concerns.

Onboarding existing cloud infrastructure is a valid use case for
jclouds users, and that can already be done (limited by the bounds of
the Compute abstraction) with the current jclouds implementation. It
is fair to expect that if the SecurityGroupExtension is present, and
there is a method called "listSecurityGroupsForNode" you should be
able to onboard the firewall rules of your existing node. As a user,
I'd have these expectations so I don't think we can discard the
onboarding use case just because we've found it difficult to map in a
consistent way.

This said, if we went that path, I think we still need to refine the
proposal, or at least be explicit and enumerate all affected points
and how we plan to fill the gaps.

Say you create a node in jclouds with "options.inboundPorts(22, 80)".
With the actual code, the node would be behind a couple firewall
rules, but the security group extension won't be able to return any
security group for the node. That is inconsistent, as something you
created using the abstraction, cannot be retrieved with the same
information using the abstraction extension. It will give users the
impression that the node has no firewall rules applied.

This case could be as simple as creating the firewall rules for the
"inboundPorts" following our SG naming and tag convention. I just want
to illustrate that the problem is not exclusive of the resources
created "outside" jclouds.

On 15 June 2017 at 11:09, Svetoslav Neykov
<sv...@cloudsoftcorp.com> wrote:
>> Svet you don't mention GCE "tags" or the question of subnets in the above:
>
> That's right, tags are an important part of the convention and I didn't expand on that. Your description nicely captures the idea.
> I believe the security groups will work across subnets in a single vpc network.
>
> Svet.
>
>
>> On 15.06.2017 г., at 11:57, Geoff Macartney <ge...@cloudsoftcorp.com> wrote:
>>
>> I like the sound of this proposal, I think it's certainly worth
>> investigating.
>>
>> Ignasi those are good questions you've asked.  Here's my thoughts:
>>
>> We can regard 'security groups' on GCE as a jclouds "overlay", and be
>> explicit about that in documentation. What I mean here is that we don't
>> need to do a "reverse mapping" of existing stuff back into jclouds
>> concepts. So if there are existing firewall rules they simply don't show up
>> in the results of 'listSecurityGroups()", "listSecurityGroupsInLocation()"
>> or "listSecurityGroupsForNode()".    That is, not all rules participate in
>> "security groups" from the jclouds point of view, only ones that jclouds
>> added for security group purposes.
>>
>> Svet you don't mention GCE "tags" or the question of subnets in the above:
>>
>> 1.  What do you think about the idea of using tags to identify security
>> groups - each security group would correspond to a tag. These tags would
>> correspond to groupIds in the IPPermission object, i.e. you could create an
>> IPPermission using a groupId [1] which would relate to a tag in the GCE
>> platform. Then any nodes with that tag (= in that security group) would be
>> permitted access.
>> 2. Are there any restrictions/implications relating to subnets in your
>> proposal, e.g. would you need to use them to achieve what you want, or can
>> it work irrespective of any subnets on the network?
>>
>> Geoff
>>
>> [1]
>> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
>>
>>
>> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org> wrote:
>>
>>> Thanks for starting this discussion Svet!
>>>
>>> It would be great to come up with a solution we are happy with.
>>>
>>> The main issue I see here is that the primary entity of the extension,
>>> the SecurityGroup, does not have a mapping in GCE (which was one of
>>> the reasons we decided to remove the old implementation).
>>>
>>> I am OK applying conventions to implement the extension somehow, but
>>> conventions are tricky when it comes to deal with existing stuff in
>>> the provider. With this proposal, what are your plans to implement the
>>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>>> location)" methods? There is also a method to list the security groups
>>> from a node. Given that the ComputeService is able to "listNodes" that
>>> weren't created by jclouds, in case of a returned Node that already
>>> has firewall rules assigned but that don't fit in our naming
>>> conventions, how would the "listSecurityGroupsForNode(String id)"
>>> behave?
>>>
>>> I.
>>>
>>> On 15 June 2017 at 09:03, Svetoslav Neykov
>>> <sv...@cloudsoftcorp.com> wrote:
>>>> I'd like to try implement the SecurityGroupExtension interface for GCE.
>>> Looking at the documentation it seems that the combination of firewall
>>> rules and node tags is flexible enough to allow us implement the
>>> functionality.
>>>> It's been tried before but the implementation's been removed (see [1]).
>>> It's main drawback was that for each security group the code creates a new
>>> network.
>>>> Currently the biggest mismatch between the jclouds abstraction and the
>>> GCE functionality is that firewall rules must be attached to a network.
>>>> Here's my suggested approach:
>>>> IpPermission roughly corresponds to a firewall rule
>>>> SecurityGroup is just a collection of firewall rules (there's no cloud
>>> resource that corresponds to it). The firewall rules of a security group
>>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They all
>>> belong to the same network.
>>>> They key bit: createSecurityGroup accepts a Location with a scope of
>>> Network, returning a custom implementation of SecurityGroup which keeps a
>>> reference to the network, so all IpPermission objects added subsequently
>>> will be on it.
>>>> While the suggested approach fits into the SecurityGroupExtension
>>> interface it's different enough from the other implementations that it
>>> might not be worth the trouble of implementing and supporting (even be
>>> harmful as users might be surprised by the different behaviour).
>>>> Would be interested in hearing other opinions on the approach. I've
>>> created a Jira to track the effort at [2].
>>>>
>>>> Svet.
>>>>
>>>>
>>>> [1]
>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>> <
>>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>>>
>>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>>>
>>>
>

Re: GCE SecurityGroupExtension implementation

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

> Svet you don't mention GCE "tags" or the question of subnets in the above:

That's right, tags are an important part of the convention and I didn't expand on that. Your description nicely captures the idea.
I believe the security groups will work across subnets in a single vpc network.

Svet.


> On 15.06.2017 г., at 11:57, Geoff Macartney <ge...@cloudsoftcorp.com> wrote:
> 
> I like the sound of this proposal, I think it's certainly worth
> investigating.
> 
> Ignasi those are good questions you've asked.  Here's my thoughts:
> 
> We can regard 'security groups' on GCE as a jclouds "overlay", and be
> explicit about that in documentation. What I mean here is that we don't
> need to do a "reverse mapping" of existing stuff back into jclouds
> concepts. So if there are existing firewall rules they simply don't show up
> in the results of 'listSecurityGroups()", "listSecurityGroupsInLocation()"
> or "listSecurityGroupsForNode()".    That is, not all rules participate in
> "security groups" from the jclouds point of view, only ones that jclouds
> added for security group purposes.
> 
> Svet you don't mention GCE "tags" or the question of subnets in the above:
> 
> 1.  What do you think about the idea of using tags to identify security
> groups - each security group would correspond to a tag. These tags would
> correspond to groupIds in the IPPermission object, i.e. you could create an
> IPPermission using a groupId [1] which would relate to a tag in the GCE
> platform. Then any nodes with that tag (= in that security group) would be
> permitted access.
> 2. Are there any restrictions/implications relating to subnets in your
> proposal, e.g. would you need to use them to achieve what you want, or can
> it work irrespective of any subnets on the network?
> 
> Geoff
> 
> [1]
> https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)
> 
> 
> On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org> wrote:
> 
>> Thanks for starting this discussion Svet!
>> 
>> It would be great to come up with a solution we are happy with.
>> 
>> The main issue I see here is that the primary entity of the extension,
>> the SecurityGroup, does not have a mapping in GCE (which was one of
>> the reasons we decided to remove the old implementation).
>> 
>> I am OK applying conventions to implement the extension somehow, but
>> conventions are tricky when it comes to deal with existing stuff in
>> the provider. With this proposal, what are your plans to implement the
>> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
>> location)" methods? There is also a method to list the security groups
>> from a node. Given that the ComputeService is able to "listNodes" that
>> weren't created by jclouds, in case of a returned Node that already
>> has firewall rules assigned but that don't fit in our naming
>> conventions, how would the "listSecurityGroupsForNode(String id)"
>> behave?
>> 
>> I.
>> 
>> On 15 June 2017 at 09:03, Svetoslav Neykov
>> <sv...@cloudsoftcorp.com> wrote:
>>> I'd like to try implement the SecurityGroupExtension interface for GCE.
>> Looking at the documentation it seems that the combination of firewall
>> rules and node tags is flexible enough to allow us implement the
>> functionality.
>>> It's been tried before but the implementation's been removed (see [1]).
>> It's main drawback was that for each security group the code creates a new
>> network.
>>> Currently the biggest mismatch between the jclouds abstraction and the
>> GCE functionality is that firewall rules must be attached to a network.
>>> Here's my suggested approach:
>>> IpPermission roughly corresponds to a firewall rule
>>> SecurityGroup is just a collection of firewall rules (there's no cloud
>> resource that corresponds to it). The firewall rules of a security group
>> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They all
>> belong to the same network.
>>> They key bit: createSecurityGroup accepts a Location with a scope of
>> Network, returning a custom implementation of SecurityGroup which keeps a
>> reference to the network, so all IpPermission objects added subsequently
>> will be on it.
>>> While the suggested approach fits into the SecurityGroupExtension
>> interface it's different enough from the other implementations that it
>> might not be worth the trouble of implementing and supporting (even be
>> harmful as users might be surprised by the different behaviour).
>>> Would be interested in hearing other opinions on the approach. I've
>> created a Jira to track the effort at [2].
>>> 
>>> Svet.
>>> 
>>> 
>>> [1]
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>> <
>> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
>>> 
>>> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
>> https://issues.apache.org/jira/browse/JCLOUDS-1309>
>>> 
>> 

Re: GCE SecurityGroupExtension implementation

[Geoff Macartney <ge...@cloudsoftcorp.com>]

I like the sound of this proposal, I think it's certainly worth
investigating.

Ignasi those are good questions you've asked.  Here's my thoughts:

We can regard 'security groups' on GCE as a jclouds "overlay", and be
explicit about that in documentation. What I mean here is that we don't
need to do a "reverse mapping" of existing stuff back into jclouds
concepts. So if there are existing firewall rules they simply don't show up
in the results of 'listSecurityGroups()", "listSecurityGroupsInLocation()"
or "listSecurityGroupsForNode()".    That is, not all rules participate in
"security groups" from the jclouds point of view, only ones that jclouds
added for security group purposes.

Svet you don't mention GCE "tags" or the question of subnets in the above:

1.  What do you think about the idea of using tags to identify security
groups - each security group would correspond to a tag. These tags would
correspond to groupIds in the IPPermission object, i.e. you could create an
IPPermission using a groupId [1] which would relate to a tag in the GCE
platform. Then any nodes with that tag (= in that security group) would be
permitted access.
2. Are there any restrictions/implications relating to subnets in your
proposal, e.g. would you need to use them to achieve what you want, or can
it work irrespective of any subnets on the network?

Geoff

[1]
https://jclouds.apache.org/reference/javadoc/1.9.x/org/jclouds/net/domain/IpPermission.Builder.html#groupId(java.lang.String)


On Thu, 15 Jun 2017 at 09:18 Ignasi Barrera <na...@apache.org> wrote:

> Thanks for starting this discussion Svet!
>
> It would be great to come up with a solution we are happy with.
>
> The main issue I see here is that the primary entity of the extension,
> the SecurityGroup, does not have a mapping in GCE (which was one of
> the reasons we decided to remove the old implementation).
>
> I am OK applying conventions to implement the extension somehow, but
> conventions are tricky when it comes to deal with existing stuff in
> the provider. With this proposal, what are your plans to implement the
> "listSecurityGroups()" and "listSecurityGroupsInLocation(Location
> location)" methods? There is also a method to list the security groups
> from a node. Given that the ComputeService is able to "listNodes" that
> weren't created by jclouds, in case of a returned Node that already
> has firewall rules assigned but that don't fit in our naming
> conventions, how would the "listSecurityGroupsForNode(String id)"
> behave?
>
> I.
>
> On 15 June 2017 at 09:03, Svetoslav Neykov
> <sv...@cloudsoftcorp.com> wrote:
> > I'd like to try implement the SecurityGroupExtension interface for GCE.
> Looking at the documentation it seems that the combination of firewall
> rules and node tags is flexible enough to allow us implement the
> functionality.
> > It's been tried before but the implementation's been removed (see [1]).
> It's main drawback was that for each security group the code creates a new
> network.
> > Currently the biggest mismatch between the jclouds abstraction and the
> GCE functionality is that firewall rules must be attached to a network.
> > Here's my suggested approach:
> > IpPermission roughly corresponds to a firewall rule
> > SecurityGroup is just a collection of firewall rules (there's no cloud
> resource that corresponds to it). The firewall rules of a security group
> share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They all
> belong to the same network.
> > They key bit: createSecurityGroup accepts a Location with a scope of
> Network, returning a custom implementation of SecurityGroup which keeps a
> reference to the network, so all IpPermission objects added subsequently
> will be on it.
> > While the suggested approach fits into the SecurityGroupExtension
> interface it's different enough from the other implementations that it
> might not be worth the trouble of implementing and supporting (even be
> harmful as users might be surprised by the different behaviour).
> > Would be interested in hearing other opinions on the approach. I've
> created a Jira to track the effort at [2].
> >
> > Svet.
> >
> >
> > [1]
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> <
> https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c
> >
> > [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <
> https://issues.apache.org/jira/browse/JCLOUDS-1309>
> >
>

Re: GCE SecurityGroupExtension implementation

[Ignasi Barrera <na...@apache.org>]

Thanks for starting this discussion Svet!

It would be great to come up with a solution we are happy with.

The main issue I see here is that the primary entity of the extension,
the SecurityGroup, does not have a mapping in GCE (which was one of
the reasons we decided to remove the old implementation).

I am OK applying conventions to implement the extension somehow, but
conventions are tricky when it comes to deal with existing stuff in
the provider. With this proposal, what are your plans to implement the
"listSecurityGroups()" and "listSecurityGroupsInLocation(Location
location)" methods? There is also a method to list the security groups
from a node. Given that the ComputeService is able to "listNodes" that
weren't created by jclouds, in case of a returned Node that already
has firewall rules assigned but that don't fit in our naming
conventions, how would the "listSecurityGroupsForNode(String id)"
behave?

I.

On 15 June 2017 at 09:03, Svetoslav Neykov
<sv...@cloudsoftcorp.com> wrote:
> I'd like to try implement the SecurityGroupExtension interface for GCE. Looking at the documentation it seems that the combination of firewall rules and node tags is flexible enough to allow us implement the functionality.
> It's been tried before but the implementation's been removed (see [1]). It's main drawback was that for each security group the code creates a new network.
> Currently the biggest mismatch between the jclouds abstraction and the GCE functionality is that firewall rules must be attached to a network.
> Here's my suggested approach:
> IpPermission roughly corresponds to a firewall rule
> SecurityGroup is just a collection of firewall rules (there's no cloud resource that corresponds to it). The firewall rules of a security group share the same prefix - jclouds-sg-<sg name>-<permission suffix>. They all belong to the same network.
> They key bit: createSecurityGroup accepts a Location with a scope of Network, returning a custom implementation of SecurityGroup which keeps a reference to the network, so all IpPermission objects added subsequently will be on it.
> While the suggested approach fits into the SecurityGroupExtension interface it's different enough from the other implementations that it might not be worth the trouble of implementing and supporting (even be harmful as users might be surprised by the different behaviour).
> Would be interested in hearing other opinions on the approach. I've created a Jira to track the effort at [2].
>
> Svet.
>
>
> [1] https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c <https://github.com/jclouds/jclouds/commit/2ba48dc9f66416b5d8515bd6a07b27a213d89a7c>
> [2] https://issues.apache.org/jira/browse/JCLOUDS-1309 <https://issues.apache.org/jira/browse/JCLOUDS-1309>
>