[PROPOSAL] Location model refactoring

[Hadrian Zbarcea <hz...@gmail.com>]

Hi,

I was thinking about it for a few months and I still don't have a 
complete proposal, but since discussions already go in that direction 
I'll put it out there and we'll see where it goes.

There are a few motivating factors for this proposal:
1. The docs say that a Location is a "A location that an entity can be 
in". The getParent(), getChildren() methods (if one didn't know the 
interface name) makes the abstraction more of a tree node.
2. In addition to the tree like organization of Locations, there is a 
second tree hierarchy of the kind of location (e.g. 
ProvisioningLocation, SimulatedLocation, AddressableLocation, etc. This 
piece is related to what Andrea mentioned on the other thread (i.e. we 
could use a SimulatedDriver to operate at a Location).
3. In my mind the relation is more of a graph.

After a few conversations with Brooklyn contributors and committers, I 
come to believe that we should refactor a Location to mean something that:

- is addressable (probably in multiple ways)
- provides an API, actually probably more to operate at that location 
(e.g. ssh, http, bash commands, etc). Credentials may be required to use 
the API.

In my model Locations are flat, i.e. no in-memory references (pointers) 
to other locations, just URLs. A location manager (registry, need to 
find a reasonable name) maintains one or more data structures to manage 
the locations.

Example:
List<String> Location.getAddresses() would return something like:
- deployed-at:<id> (equivalent to getParent() today)
- geoip:(lat, long)
- ssh:root@localhost
- karaf:karaf@<ip>
- ssh+yum:root@localhost
- ssh+npm:root@localhost
- ssh+ansible:root@localhost
- winrm:<service address>

(obviously in the above example we have a both Windows and RHEL box, 
which is wrong, but it'll do for this proposal :) ).

An external service would build a graph and would allow for searches on 
a graph given some criteria.

Another thought I will mention is that the way our entities are 
implemented, they give the impression that Brooklyn is more of a CM 
tools with the logic of deploying provided in Java. This is both not 
accurate and doesn't do Brooklyn justice. It is imo next to impossible 
to build an entity that would satisfy a large majority of users. The 
complexity becomes prohibitive. For that reason I see Brooklyn more of 
an enabler for implementers of entities to provide and promote their 
implementation. E.g. it's next to impossible to implement a best, 
complete entity for mysql, and it's more likely to see: hadrian-mysql 
(that has some characteristics, say it's super secure, deployed with 
bash, only works on ubuntu), and andrea-mysql (which is deployed with 
chef and is supper fast) and cipi-mysql (deployed in a cluster topology, 
super resilient). A user would pick the one they prefer and use it in 
their blueprint. This is no different than the kind of ecosystem docker, 
chef, puppet, etc, created.

I'll stop here hoping for some feedback. This proposal has a more 
significant backwards compatibility impact and should be considered 
independently of the other proposal related to locations.

Cheers,
Hadrian

Re: [PROPOSAL] Location model refactoring

[Hadrian Zbarcea <hz...@gmail.com>]

Precisely, I agree with your definition, the "where" means addresses. 
Look at the definition of Location and my proposal and see which one is 
more inline with your statement.

Salt/Ansible absolutely must know the set of machines they manage. If a 
MachineProvisioniningLocation provides (obtain) a new node, that's where 
an event should be issued to update the configuration of salt/ssh. 
Granted for Chef, Puppet (and Salt in master/minion mode) that can be 
done on demand via a blueprint and since the minion will contact the 
master (known via config) you have a second opportunity for issuing such 
an event.

I think Brooklyn absolutely should (if not must) manage the elasticity, 
probably via policies. Otherwise the value of Brooklyn is significantly 
diluted imho.

Cheers,
Hadrian

On 12/08/2015 03:47 AM, Andrea Turli wrote:
> Hadrian,
>
> Let me go back for a moment to the Brooklyn location definition: IMHO it
> answers the question *where* to deploy an entity not *how*.
> If we agree with this naive definition I think I like Alex's proposal of
> adding 'capabilities' to a location, primarily ssh, winrm but also config
> mgmt tool (chef, puppet, salt, etc) and potentially package managers
> (apt-get, yum, npm, ...) so to enrich the metadata model attached to a
> location, avoiding complex locations' hierarchies, I hope.
>
> Also I'm not totally sure Brooklyn needs to manage the elasticity of the
> locations themselves, main focus is to manage the application runtime by
> making use of whichever infra resource is already available (or provison a
> new one!) vs managing the underlying infra scalability per se.
>
> Wdyt?
> Andrea
>
> Il giorno mar 8 dic 2015 04:31 Hadrian Zbarcea <hz...@gmail.com> ha
> scritto:
>
>> Thanks Alex for the comments, pity there aren't more. I either don't fully
>> understand your answer or I failed to properly communicate my proposal. If
>> you like the direction at least there may be some merit to the idea and
>> it's worth exploring more.
>>
>> (1) perfectly fine
>> (2) I don't think the java sub/supertypes are able to properly model
>> brooklyn abstractions.
>>
>> Now back to my proposal ChefLocation was probably of a debatable. But let's
>> consider Salt or Ansible support (I put a first cut on a branch in my fork
>> <https://github.com/hzbarcea/incubator-brooklyn/tree/salt-location>).
>> SaltStack has actually two modes, the master/minion model similar to Puppet
>> and Chef) and a salt-ssh mode similar to ansible. It is this latter
>> mode/style I am referring to. I spent quite a bit of time lately with Salt
>> and I started to like it a lot for many reasons including the license. So,
>> both salt-ssh and ansible need to know the machines they need to manage
>> from a distance (called roster in salt and hosts in ansible). Those
>> machines are necessarily a subset of a location in the general case (e.g.
>> an EC2 based infrastructure), although in some cases, like byon one could
>> make the convention to use all the nodes. Therefore, every time a new node
>> is obtain(ed) the controller node (salt-ssh) needs to add it to its roster,
>> which in my mind becomes a 'specialized' location like a SaltLocation or
>> AnsibleLocation managed by an *Infrastructure entity. The same problem (and
>> model) is used by clocker.
>>
>> I believe there would be a better model to use something like a
>> ManagedLocation with hooks or something, but didn't get far enough to
>> understand what the commonalities are and for now I want to avoid shooting
>> in the dark.
>>
>> Another point, unrelated to the proposal, but related to the Salt thingie
>> and testing in general, I came to believe that we under use some cool
>> features in Brooklyn like the use of the SimulatedLocation. I had to
>> addRule for using a SimulatedDriver and there is more work to really test
>> propertly what's going on there, but I think it's a decent way to bring
>> more unit testing in and rely less on the Live or Integration tests. Btw,
>> this is my 3rd rewrite of the Salt support, and now I start to like the way
>> it looks. I have to port more code from my other local branches and then
>> submit a PR.
>>
>> Best,
>> Hadrian
>>
>>
>> On Thu, Dec 3, 2015 at 10:32 AM, Alex Heneveld <
>> alex.heneveld@cloudsoftcorp.com> wrote:
>>
>>>
>>> Hadrian-
>>>
>>> TL;DR - I like the direction you're going in, although I'm not sure of
>> the
>>> conclustions.
>>>
>>> Firstly there is a lot of synergy with two things I've been working on
>>> lately:
>>>
>>> (1) typed relationships
>>> (2) multiple supertypes/interfaces on entities/brooklyn objects
>>>
>>>
>>> With (1) we're be better able model relationships between things, to say
>> X
>>> is HOSTED_ON Y, where Y is a location.  And I could make queries to find
>>> the unique Y such that X is HOSTED_ON Y and Y is an SshMachineLocation.
>> Or
>>> a ChefLocation.  Or other.  We can express ON_NETWORK to model situations
>>> where we have multiple networks.
>>>
>>> We can also express dependencies better, so rather than say e.g. Tomcat
>>> has a JDBC_URL=attributeWhenReady("db") I could say that MyTomcat takes a
>>> required config D of type <Database>, and it has a HOSTED_ON relationship
>>> with D, and then JDBC_URL=attributeWhenReady(D)... .  The upshot is that
>> a
>>> user won't need to edit the complex "attributeWhenReady" -- they just
>> have
>>> to provide D.  You can imagine a drag-and-drop UI makes this easy!
>>>
>>>
>>> With (2) we can capture capabilities as interfaces -- e.g. a ChefLocation
>>> interface might simply define a `converge(x)` effector.  Any entity which
>>> supports chef then advertises ChefLocation as part of his type.  Same for
>>> Database as discussed in (1).  We kinda support this currently using the
>>> java model, but not for yaml.  The recent commits around the new Type
>>> Registry allows us to track a type hierarchy for arbitrary types, defined
>>> in yaml, in a way which neatly extends the java hierarchy.  (so you can
>>> continue to use java types.)
>>>
>>>
>>> The natural conclusion of this I think is to make Locations -- and
>>> Policies and Enrichers and Feeds -- simple sub-types of Entities. And
>>> possibly nothing more than marker interfaces.  Hopefully the vast
>> majority
>>> of the code which treats those as different could be removed so we treat
>>> them the same.
>>>
>>>
>>> Where I'm not sure I agree with you is when it comes to the value of
>>> building out ChefLocation.  It is a clean underlay, but what problem does
>>> it solve for users?  I think we want to optimize for people to write
>>> blueprints in yaml, like the "my-chef-mysql" example, which I think is
>>> really nice.
>>>
>>> Similar for package managers.  I tend to the view that things like
>>> BashCommands.install are discouraged, except where the installation is
>>> relatively simple.  If you want a real cross-OS blueprint use chef or
>>> ansible or salt.  (And we make that easy.) Or you write blueprints for a
>>> specific OS and hard-code `apt-get`.  Most clouds will give you the
>> choice
>>> of ubuntu or centos.  What benefit would more PackageManagers give us in
>>> Brooklyn?  To me it feels like we'd be trying to do what others do well.
>>>
>>> It's the modelling, grouping, interconnectedness, sensors, policies, and
>>> effectors that I think are our power, and systems managed at a low-level
>>> with chef/salt/etc become truly programmable and autonomic whan you wrap
>>> the ansible/puppet/cf-engine/etc with Brooklyn.  (And I love the idea of
>>> persisting to a queryable, and visualisable, graph store BTW.)
>>>
>>> Best
>>> Alex
>>>
>>>
>>>
>>> On 01/12/2015 13:12, Hadrian Zbarcea wrote:
>>>
>>>> Hi,
>>>>
>>>> I was thinking about it for a few months and I still don't have a
>>>> complete proposal, but since discussions already go in that direction
>> I'll
>>>> put it out there and we'll see where it goes.
>>>>
>>>> There are a few motivating factors for this proposal:
>>>> 1. The docs say that a Location is a "A location that an entity can be
>>>> in". The getParent(), getChildren() methods (if one didn't know the
>>>> interface name) makes the abstraction more of a tree node.
>>>> 2. In addition to the tree like organization of Locations, there is a
>>>> second tree hierarchy of the kind of location (e.g.
>> ProvisioningLocation,
>>>> SimulatedLocation, AddressableLocation, etc. This piece is related to
>> what
>>>> Andrea mentioned on the other thread (i.e. we could use a
>> SimulatedDriver
>>>> to operate at a Location).
>>>> 3. In my mind the relation is more of a graph.
>>>>
>>>> After a few conversations with Brooklyn contributors and committers, I
>>>> come to believe that we should refactor a Location to mean something
>> that:
>>>>
>>>> - is addressable (probably in multiple ways)
>>>> - provides an API, actually probably more to operate at that location
>>>> (e.g. ssh, http, bash commands, etc). Credentials may be required to use
>>>> the API.
>>>>
>>>> In my model Locations are flat, i.e. no in-memory references (pointers)
>>>> to other locations, just URLs. A location manager (registry, need to
>> find a
>>>> reasonable name) maintains one or more data structures to manage the
>>>> locations.
>>>>
>>>> Example:
>>>> List<String> Location.getAddresses() would return something like:
>>>> - deployed-at:<id> (equivalent to getParent() today)
>>>> - geoip:(lat, long)
>>>> - ssh:root@localhost
>>>> - karaf:karaf@<ip>
>>>> - ssh+yum:root@localhost
>>>> - ssh+npm:root@localhost
>>>> - ssh+ansible:root@localhost
>>>> - winrm:<service address>
>>>>
>>>> (obviously in the above example we have a both Windows and RHEL box,
>>>> which is wrong, but it'll do for this proposal :) ).
>>>>
>>>> An external service would build a graph and would allow for searches on
>> a
>>>> graph given some criteria.
>>>>
>>>> Another thought I will mention is that the way our entities are
>>>> implemented, they give the impression that Brooklyn is more of a CM
>> tools
>>>> with the logic of deploying provided in Java. This is both not accurate
>> and
>>>> doesn't do Brooklyn justice. It is imo next to impossible to build an
>>>> entity that would satisfy a large majority of users. The complexity
>> becomes
>>>> prohibitive. For that reason I see Brooklyn more of an enabler for
>>>> implementers of entities to provide and promote their implementation.
>> E.g.
>>>> it's next to impossible to implement a best, complete entity for mysql,
>> and
>>>> it's more likely to see: hadrian-mysql (that has some characteristics,
>> say
>>>> it's super secure, deployed with bash, only works on ubuntu), and
>>>> andrea-mysql (which is deployed with chef and is supper fast) and
>>>> cipi-mysql (deployed in a cluster topology, super resilient). A user
>> would
>>>> pick the one they prefer and use it in their blueprint. This is no
>>>> different than the kind of ecosystem docker, chef, puppet, etc, created.
>>>>
>>>> I'll stop here hoping for some feedback. This proposal has a more
>>>> significant backwards compatibility impact and should be considered
>>>> independently of the other proposal related to locations.
>>>>
>>>> Cheers,
>>>> Hadrian
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

Re: [PROPOSAL] Location model refactoring

[Andrea Turli <an...@cloudsoftcorp.com>]

Hadrian,

Let me go back for a moment to the Brooklyn location definition: IMHO it
answers the question *where* to deploy an entity not *how*.
If we agree with this naive definition I think I like Alex's proposal of
adding 'capabilities' to a location, primarily ssh, winrm but also config
mgmt tool (chef, puppet, salt, etc) and potentially package managers
(apt-get, yum, npm, ...) so to enrich the metadata model attached to a
location, avoiding complex locations' hierarchies, I hope.

Also I'm not totally sure Brooklyn needs to manage the elasticity of the
locations themselves, main focus is to manage the application runtime by
making use of whichever infra resource is already available (or provison a
new one!) vs managing the underlying infra scalability per se.

Wdyt?
Andrea

Il giorno mar 8 dic 2015 04:31 Hadrian Zbarcea <hz...@gmail.com> ha
scritto:

> Thanks Alex for the comments, pity there aren't more. I either don't fully
> understand your answer or I failed to properly communicate my proposal. If
> you like the direction at least there may be some merit to the idea and
> it's worth exploring more.
>
> (1) perfectly fine
> (2) I don't think the java sub/supertypes are able to properly model
> brooklyn abstractions.
>
> Now back to my proposal ChefLocation was probably of a debatable. But let's
> consider Salt or Ansible support (I put a first cut on a branch in my fork
> <https://github.com/hzbarcea/incubator-brooklyn/tree/salt-location>).
> SaltStack has actually two modes, the master/minion model similar to Puppet
> and Chef) and a salt-ssh mode similar to ansible. It is this latter
> mode/style I am referring to. I spent quite a bit of time lately with Salt
> and I started to like it a lot for many reasons including the license. So,
> both salt-ssh and ansible need to know the machines they need to manage
> from a distance (called roster in salt and hosts in ansible). Those
> machines are necessarily a subset of a location in the general case (e.g.
> an EC2 based infrastructure), although in some cases, like byon one could
> make the convention to use all the nodes. Therefore, every time a new node
> is obtain(ed) the controller node (salt-ssh) needs to add it to its roster,
> which in my mind becomes a 'specialized' location like a SaltLocation or
> AnsibleLocation managed by an *Infrastructure entity. The same problem (and
> model) is used by clocker.
>
> I believe there would be a better model to use something like a
> ManagedLocation with hooks or something, but didn't get far enough to
> understand what the commonalities are and for now I want to avoid shooting
> in the dark.
>
> Another point, unrelated to the proposal, but related to the Salt thingie
> and testing in general, I came to believe that we under use some cool
> features in Brooklyn like the use of the SimulatedLocation. I had to
> addRule for using a SimulatedDriver and there is more work to really test
> propertly what's going on there, but I think it's a decent way to bring
> more unit testing in and rely less on the Live or Integration tests. Btw,
> this is my 3rd rewrite of the Salt support, and now I start to like the way
> it looks. I have to port more code from my other local branches and then
> submit a PR.
>
> Best,
> Hadrian
>
>
> On Thu, Dec 3, 2015 at 10:32 AM, Alex Heneveld <
> alex.heneveld@cloudsoftcorp.com> wrote:
>
> >
> > Hadrian-
> >
> > TL;DR - I like the direction you're going in, although I'm not sure of
> the
> > conclustions.
> >
> > Firstly there is a lot of synergy with two things I've been working on
> > lately:
> >
> > (1) typed relationships
> > (2) multiple supertypes/interfaces on entities/brooklyn objects
> >
> >
> > With (1) we're be better able model relationships between things, to say
> X
> > is HOSTED_ON Y, where Y is a location.  And I could make queries to find
> > the unique Y such that X is HOSTED_ON Y and Y is an SshMachineLocation.
> Or
> > a ChefLocation.  Or other.  We can express ON_NETWORK to model situations
> > where we have multiple networks.
> >
> > We can also express dependencies better, so rather than say e.g. Tomcat
> > has a JDBC_URL=attributeWhenReady("db") I could say that MyTomcat takes a
> > required config D of type <Database>, and it has a HOSTED_ON relationship
> > with D, and then JDBC_URL=attributeWhenReady(D)... .  The upshot is that
> a
> > user won't need to edit the complex "attributeWhenReady" -- they just
> have
> > to provide D.  You can imagine a drag-and-drop UI makes this easy!
> >
> >
> > With (2) we can capture capabilities as interfaces -- e.g. a ChefLocation
> > interface might simply define a `converge(x)` effector.  Any entity which
> > supports chef then advertises ChefLocation as part of his type.  Same for
> > Database as discussed in (1).  We kinda support this currently using the
> > java model, but not for yaml.  The recent commits around the new Type
> > Registry allows us to track a type hierarchy for arbitrary types, defined
> > in yaml, in a way which neatly extends the java hierarchy.  (so you can
> > continue to use java types.)
> >
> >
> > The natural conclusion of this I think is to make Locations -- and
> > Policies and Enrichers and Feeds -- simple sub-types of Entities. And
> > possibly nothing more than marker interfaces.  Hopefully the vast
> majority
> > of the code which treats those as different could be removed so we treat
> > them the same.
> >
> >
> > Where I'm not sure I agree with you is when it comes to the value of
> > building out ChefLocation.  It is a clean underlay, but what problem does
> > it solve for users?  I think we want to optimize for people to write
> > blueprints in yaml, like the "my-chef-mysql" example, which I think is
> > really nice.
> >
> > Similar for package managers.  I tend to the view that things like
> > BashCommands.install are discouraged, except where the installation is
> > relatively simple.  If you want a real cross-OS blueprint use chef or
> > ansible or salt.  (And we make that easy.) Or you write blueprints for a
> > specific OS and hard-code `apt-get`.  Most clouds will give you the
> choice
> > of ubuntu or centos.  What benefit would more PackageManagers give us in
> > Brooklyn?  To me it feels like we'd be trying to do what others do well.
> >
> > It's the modelling, grouping, interconnectedness, sensors, policies, and
> > effectors that I think are our power, and systems managed at a low-level
> > with chef/salt/etc become truly programmable and autonomic whan you wrap
> > the ansible/puppet/cf-engine/etc with Brooklyn.  (And I love the idea of
> > persisting to a queryable, and visualisable, graph store BTW.)
> >
> > Best
> > Alex
> >
> >
> >
> > On 01/12/2015 13:12, Hadrian Zbarcea wrote:
> >
> >> Hi,
> >>
> >> I was thinking about it for a few months and I still don't have a
> >> complete proposal, but since discussions already go in that direction
> I'll
> >> put it out there and we'll see where it goes.
> >>
> >> There are a few motivating factors for this proposal:
> >> 1. The docs say that a Location is a "A location that an entity can be
> >> in". The getParent(), getChildren() methods (if one didn't know the
> >> interface name) makes the abstraction more of a tree node.
> >> 2. In addition to the tree like organization of Locations, there is a
> >> second tree hierarchy of the kind of location (e.g.
> ProvisioningLocation,
> >> SimulatedLocation, AddressableLocation, etc. This piece is related to
> what
> >> Andrea mentioned on the other thread (i.e. we could use a
> SimulatedDriver
> >> to operate at a Location).
> >> 3. In my mind the relation is more of a graph.
> >>
> >> After a few conversations with Brooklyn contributors and committers, I
> >> come to believe that we should refactor a Location to mean something
> that:
> >>
> >> - is addressable (probably in multiple ways)
> >> - provides an API, actually probably more to operate at that location
> >> (e.g. ssh, http, bash commands, etc). Credentials may be required to use
> >> the API.
> >>
> >> In my model Locations are flat, i.e. no in-memory references (pointers)
> >> to other locations, just URLs. A location manager (registry, need to
> find a
> >> reasonable name) maintains one or more data structures to manage the
> >> locations.
> >>
> >> Example:
> >> List<String> Location.getAddresses() would return something like:
> >> - deployed-at:<id> (equivalent to getParent() today)
> >> - geoip:(lat, long)
> >> - ssh:root@localhost
> >> - karaf:karaf@<ip>
> >> - ssh+yum:root@localhost
> >> - ssh+npm:root@localhost
> >> - ssh+ansible:root@localhost
> >> - winrm:<service address>
> >>
> >> (obviously in the above example we have a both Windows and RHEL box,
> >> which is wrong, but it'll do for this proposal :) ).
> >>
> >> An external service would build a graph and would allow for searches on
> a
> >> graph given some criteria.
> >>
> >> Another thought I will mention is that the way our entities are
> >> implemented, they give the impression that Brooklyn is more of a CM
> tools
> >> with the logic of deploying provided in Java. This is both not accurate
> and
> >> doesn't do Brooklyn justice. It is imo next to impossible to build an
> >> entity that would satisfy a large majority of users. The complexity
> becomes
> >> prohibitive. For that reason I see Brooklyn more of an enabler for
> >> implementers of entities to provide and promote their implementation.
> E.g.
> >> it's next to impossible to implement a best, complete entity for mysql,
> and
> >> it's more likely to see: hadrian-mysql (that has some characteristics,
> say
> >> it's super secure, deployed with bash, only works on ubuntu), and
> >> andrea-mysql (which is deployed with chef and is supper fast) and
> >> cipi-mysql (deployed in a cluster topology, super resilient). A user
> would
> >> pick the one they prefer and use it in their blueprint. This is no
> >> different than the kind of ecosystem docker, chef, puppet, etc, created.
> >>
> >> I'll stop here hoping for some feedback. This proposal has a more
> >> significant backwards compatibility impact and should be considered
> >> independently of the other proposal related to locations.
> >>
> >> Cheers,
> >> Hadrian
> >>
> >>
> >>
> >>
> >>
> >>
> >
>

Re: [PROPOSAL] Location model refactoring

[Hadrian Zbarcea <hz...@gmail.com>]

Thanks Alex for the comments, pity there aren't more. I either don't fully
understand your answer or I failed to properly communicate my proposal. If
you like the direction at least there may be some merit to the idea and
it's worth exploring more.

(1) perfectly fine
(2) I don't think the java sub/supertypes are able to properly model
brooklyn abstractions.

Now back to my proposal ChefLocation was probably of a debatable. But let's
consider Salt or Ansible support (I put a first cut on a branch in my fork
<https://github.com/hzbarcea/incubator-brooklyn/tree/salt-location>).
SaltStack has actually two modes, the master/minion model similar to Puppet
and Chef) and a salt-ssh mode similar to ansible. It is this latter
mode/style I am referring to. I spent quite a bit of time lately with Salt
and I started to like it a lot for many reasons including the license. So,
both salt-ssh and ansible need to know the machines they need to manage
from a distance (called roster in salt and hosts in ansible). Those
machines are necessarily a subset of a location in the general case (e.g.
an EC2 based infrastructure), although in some cases, like byon one could
make the convention to use all the nodes. Therefore, every time a new node
is obtain(ed) the controller node (salt-ssh) needs to add it to its roster,
which in my mind becomes a 'specialized' location like a SaltLocation or
AnsibleLocation managed by an *Infrastructure entity. The same problem (and
model) is used by clocker.

I believe there would be a better model to use something like a
ManagedLocation with hooks or something, but didn't get far enough to
understand what the commonalities are and for now I want to avoid shooting
in the dark.

Another point, unrelated to the proposal, but related to the Salt thingie
and testing in general, I came to believe that we under use some cool
features in Brooklyn like the use of the SimulatedLocation. I had to
addRule for using a SimulatedDriver and there is more work to really test
propertly what's going on there, but I think it's a decent way to bring
more unit testing in and rely less on the Live or Integration tests. Btw,
this is my 3rd rewrite of the Salt support, and now I start to like the way
it looks. I have to port more code from my other local branches and then
submit a PR.

Best,
Hadrian


On Thu, Dec 3, 2015 at 10:32 AM, Alex Heneveld <
alex.heneveld@cloudsoftcorp.com> wrote:

>
> Hadrian-
>
> TL;DR - I like the direction you're going in, although I'm not sure of the
> conclustions.
>
> Firstly there is a lot of synergy with two things I've been working on
> lately:
>
> (1) typed relationships
> (2) multiple supertypes/interfaces on entities/brooklyn objects
>
>
> With (1) we're be better able model relationships between things, to say X
> is HOSTED_ON Y, where Y is a location.  And I could make queries to find
> the unique Y such that X is HOSTED_ON Y and Y is an SshMachineLocation.  Or
> a ChefLocation.  Or other.  We can express ON_NETWORK to model situations
> where we have multiple networks.
>
> We can also express dependencies better, so rather than say e.g. Tomcat
> has a JDBC_URL=attributeWhenReady("db") I could say that MyTomcat takes a
> required config D of type <Database>, and it has a HOSTED_ON relationship
> with D, and then JDBC_URL=attributeWhenReady(D)... .  The upshot is that a
> user won't need to edit the complex "attributeWhenReady" -- they just have
> to provide D.  You can imagine a drag-and-drop UI makes this easy!
>
>
> With (2) we can capture capabilities as interfaces -- e.g. a ChefLocation
> interface might simply define a `converge(x)` effector.  Any entity which
> supports chef then advertises ChefLocation as part of his type.  Same for
> Database as discussed in (1).  We kinda support this currently using the
> java model, but not for yaml.  The recent commits around the new Type
> Registry allows us to track a type hierarchy for arbitrary types, defined
> in yaml, in a way which neatly extends the java hierarchy.  (so you can
> continue to use java types.)
>
>
> The natural conclusion of this I think is to make Locations -- and
> Policies and Enrichers and Feeds -- simple sub-types of Entities. And
> possibly nothing more than marker interfaces.  Hopefully the vast majority
> of the code which treats those as different could be removed so we treat
> them the same.
>
>
> Where I'm not sure I agree with you is when it comes to the value of
> building out ChefLocation.  It is a clean underlay, but what problem does
> it solve for users?  I think we want to optimize for people to write
> blueprints in yaml, like the "my-chef-mysql" example, which I think is
> really nice.
>
> Similar for package managers.  I tend to the view that things like
> BashCommands.install are discouraged, except where the installation is
> relatively simple.  If you want a real cross-OS blueprint use chef or
> ansible or salt.  (And we make that easy.) Or you write blueprints for a
> specific OS and hard-code `apt-get`.  Most clouds will give you the choice
> of ubuntu or centos.  What benefit would more PackageManagers give us in
> Brooklyn?  To me it feels like we'd be trying to do what others do well.
>
> It's the modelling, grouping, interconnectedness, sensors, policies, and
> effectors that I think are our power, and systems managed at a low-level
> with chef/salt/etc become truly programmable and autonomic whan you wrap
> the ansible/puppet/cf-engine/etc with Brooklyn.  (And I love the idea of
> persisting to a queryable, and visualisable, graph store BTW.)
>
> Best
> Alex
>
>
>
> On 01/12/2015 13:12, Hadrian Zbarcea wrote:
>
>> Hi,
>>
>> I was thinking about it for a few months and I still don't have a
>> complete proposal, but since discussions already go in that direction I'll
>> put it out there and we'll see where it goes.
>>
>> There are a few motivating factors for this proposal:
>> 1. The docs say that a Location is a "A location that an entity can be
>> in". The getParent(), getChildren() methods (if one didn't know the
>> interface name) makes the abstraction more of a tree node.
>> 2. In addition to the tree like organization of Locations, there is a
>> second tree hierarchy of the kind of location (e.g. ProvisioningLocation,
>> SimulatedLocation, AddressableLocation, etc. This piece is related to what
>> Andrea mentioned on the other thread (i.e. we could use a SimulatedDriver
>> to operate at a Location).
>> 3. In my mind the relation is more of a graph.
>>
>> After a few conversations with Brooklyn contributors and committers, I
>> come to believe that we should refactor a Location to mean something that:
>>
>> - is addressable (probably in multiple ways)
>> - provides an API, actually probably more to operate at that location
>> (e.g. ssh, http, bash commands, etc). Credentials may be required to use
>> the API.
>>
>> In my model Locations are flat, i.e. no in-memory references (pointers)
>> to other locations, just URLs. A location manager (registry, need to find a
>> reasonable name) maintains one or more data structures to manage the
>> locations.
>>
>> Example:
>> List<String> Location.getAddresses() would return something like:
>> - deployed-at:<id> (equivalent to getParent() today)
>> - geoip:(lat, long)
>> - ssh:root@localhost
>> - karaf:karaf@<ip>
>> - ssh+yum:root@localhost
>> - ssh+npm:root@localhost
>> - ssh+ansible:root@localhost
>> - winrm:<service address>
>>
>> (obviously in the above example we have a both Windows and RHEL box,
>> which is wrong, but it'll do for this proposal :) ).
>>
>> An external service would build a graph and would allow for searches on a
>> graph given some criteria.
>>
>> Another thought I will mention is that the way our entities are
>> implemented, they give the impression that Brooklyn is more of a CM tools
>> with the logic of deploying provided in Java. This is both not accurate and
>> doesn't do Brooklyn justice. It is imo next to impossible to build an
>> entity that would satisfy a large majority of users. The complexity becomes
>> prohibitive. For that reason I see Brooklyn more of an enabler for
>> implementers of entities to provide and promote their implementation. E.g.
>> it's next to impossible to implement a best, complete entity for mysql, and
>> it's more likely to see: hadrian-mysql (that has some characteristics, say
>> it's super secure, deployed with bash, only works on ubuntu), and
>> andrea-mysql (which is deployed with chef and is supper fast) and
>> cipi-mysql (deployed in a cluster topology, super resilient). A user would
>> pick the one they prefer and use it in their blueprint. This is no
>> different than the kind of ecosystem docker, chef, puppet, etc, created.
>>
>> I'll stop here hoping for some feedback. This proposal has a more
>> significant backwards compatibility impact and should be considered
>> independently of the other proposal related to locations.
>>
>> Cheers,
>> Hadrian
>>
>>
>>
>>
>>
>>
>

Re: [PROPOSAL] Location model refactoring

[Alex Heneveld <al...@cloudsoftcorp.com>]

Hadrian-

TL;DR - I like the direction you're going in, although I'm not sure of 
the conclustions.

Firstly there is a lot of synergy with two things I've been working on 
lately:

(1) typed relationships
(2) multiple supertypes/interfaces on entities/brooklyn objects


With (1) we're be better able model relationships between things, to say 
X is HOSTED_ON Y, where Y is a location.  And I could make queries to 
find the unique Y such that X is HOSTED_ON Y and Y is an 
SshMachineLocation.  Or a ChefLocation.  Or other.  We can express 
ON_NETWORK to model situations where we have multiple networks.

We can also express dependencies better, so rather than say e.g. Tomcat 
has a JDBC_URL=attributeWhenReady("db") I could say that MyTomcat takes 
a required config D of type <Database>, and it has a HOSTED_ON 
relationship with D, and then JDBC_URL=attributeWhenReady(D)... .  The 
upshot is that a user won't need to edit the complex 
"attributeWhenReady" -- they just have to provide D.  You can imagine a 
drag-and-drop UI makes this easy!


With (2) we can capture capabilities as interfaces -- e.g. a 
ChefLocation interface might simply define a `converge(x)` effector.  
Any entity which supports chef then advertises ChefLocation as part of 
his type.  Same for Database as discussed in (1).  We kinda support this 
currently using the java model, but not for yaml.  The recent commits 
around the new Type Registry allows us to track a type hierarchy for 
arbitrary types, defined in yaml, in a way which neatly extends the java 
hierarchy.  (so you can continue to use java types.)


The natural conclusion of this I think is to make Locations -- and 
Policies and Enrichers and Feeds -- simple sub-types of Entities. And 
possibly nothing more than marker interfaces.  Hopefully the vast 
majority of the code which treats those as different could be removed so 
we treat them the same.


Where I'm not sure I agree with you is when it comes to the value of 
building out ChefLocation.  It is a clean underlay, but what problem 
does it solve for users?  I think we want to optimize for people to 
write blueprints in yaml, like the "my-chef-mysql" example, which I 
think is really nice.

Similar for package managers.  I tend to the view that things like 
BashCommands.install are discouraged, except where the installation is 
relatively simple.  If you want a real cross-OS blueprint use chef or 
ansible or salt.  (And we make that easy.) Or you write blueprints for a 
specific OS and hard-code `apt-get`.  Most clouds will give you the 
choice of ubuntu or centos.  What benefit would more PackageManagers 
give us in Brooklyn?  To me it feels like we'd be trying to do what 
others do well.

It's the modelling, grouping, interconnectedness, sensors, policies, and 
effectors that I think are our power, and systems managed at a low-level 
with chef/salt/etc become truly programmable and autonomic whan you wrap 
the ansible/puppet/cf-engine/etc with Brooklyn.  (And I love the idea of 
persisting to a queryable, and visualisable, graph store BTW.)

Best
Alex


On 01/12/2015 13:12, Hadrian Zbarcea wrote:
> Hi,
>
> I was thinking about it for a few months and I still don't have a 
> complete proposal, but since discussions already go in that direction 
> I'll put it out there and we'll see where it goes.
>
> There are a few motivating factors for this proposal:
> 1. The docs say that a Location is a "A location that an entity can be 
> in". The getParent(), getChildren() methods (if one didn't know the 
> interface name) makes the abstraction more of a tree node.
> 2. In addition to the tree like organization of Locations, there is a 
> second tree hierarchy of the kind of location (e.g. 
> ProvisioningLocation, SimulatedLocation, AddressableLocation, etc. 
> This piece is related to what Andrea mentioned on the other thread 
> (i.e. we could use a SimulatedDriver to operate at a Location).
> 3. In my mind the relation is more of a graph.
>
> After a few conversations with Brooklyn contributors and committers, I 
> come to believe that we should refactor a Location to mean something 
> that:
>
> - is addressable (probably in multiple ways)
> - provides an API, actually probably more to operate at that location 
> (e.g. ssh, http, bash commands, etc). Credentials may be required to 
> use the API.
>
> In my model Locations are flat, i.e. no in-memory references 
> (pointers) to other locations, just URLs. A location manager 
> (registry, need to find a reasonable name) maintains one or more data 
> structures to manage the locations.
>
> Example:
> List<String> Location.getAddresses() would return something like:
> - deployed-at:<id> (equivalent to getParent() today)
> - geoip:(lat, long)
> - ssh:root@localhost
> - karaf:karaf@<ip>
> - ssh+yum:root@localhost
> - ssh+npm:root@localhost
> - ssh+ansible:root@localhost
> - winrm:<service address>
>
> (obviously in the above example we have a both Windows and RHEL box, 
> which is wrong, but it'll do for this proposal :) ).
>
> An external service would build a graph and would allow for searches 
> on a graph given some criteria.
>
> Another thought I will mention is that the way our entities are 
> implemented, they give the impression that Brooklyn is more of a CM 
> tools with the logic of deploying provided in Java. This is both not 
> accurate and doesn't do Brooklyn justice. It is imo next to impossible 
> to build an entity that would satisfy a large majority of users. The 
> complexity becomes prohibitive. For that reason I see Brooklyn more of 
> an enabler for implementers of entities to provide and promote their 
> implementation. E.g. it's next to impossible to implement a best, 
> complete entity for mysql, and it's more likely to see: hadrian-mysql 
> (that has some characteristics, say it's super secure, deployed with 
> bash, only works on ubuntu), and andrea-mysql (which is deployed with 
> chef and is supper fast) and cipi-mysql (deployed in a cluster 
> topology, super resilient). A user would pick the one they prefer and 
> use it in their blueprint. This is no different than the kind of 
> ecosystem docker, chef, puppet, etc, created.
>
> I'll stop here hoping for some feedback. This proposal has a more 
> significant backwards compatibility impact and should be considered 
> independently of the other proposal related to locations.
>
> Cheers,
> Hadrian
>
>
>
>
>

Re: [PROPOSAL] Location model refactoring

[Hadrian Zbarcea <hz...@gmail.com>]

No opinion? Wasn't explained properly?

Hadrian


On 12/01/2015 08:12 AM, Hadrian Zbarcea wrote:
> Hi,
>
> I was thinking about it for a few months and I still don't have a
> complete proposal, but since discussions already go in that direction
> I'll put it out there and we'll see where it goes.
>
> There are a few motivating factors for this proposal:
> 1. The docs say that a Location is a "A location that an entity can be
> in". The getParent(), getChildren() methods (if one didn't know the
> interface name) makes the abstraction more of a tree node.
> 2. In addition to the tree like organization of Locations, there is a
> second tree hierarchy of the kind of location (e.g.
> ProvisioningLocation, SimulatedLocation, AddressableLocation, etc. This
> piece is related to what Andrea mentioned on the other thread (i.e. we
> could use a SimulatedDriver to operate at a Location).
> 3. In my mind the relation is more of a graph.
>
> After a few conversations with Brooklyn contributors and committers, I
> come to believe that we should refactor a Location to mean something that:
>
> - is addressable (probably in multiple ways)
> - provides an API, actually probably more to operate at that location
> (e.g. ssh, http, bash commands, etc). Credentials may be required to use
> the API.
>
> In my model Locations are flat, i.e. no in-memory references (pointers)
> to other locations, just URLs. A location manager (registry, need to
> find a reasonable name) maintains one or more data structures to manage
> the locations.
>
> Example:
> List<String> Location.getAddresses() would return something like:
> - deployed-at:<id> (equivalent to getParent() today)
> - geoip:(lat, long)
> - ssh:root@localhost
> - karaf:karaf@<ip>
> - ssh+yum:root@localhost
> - ssh+npm:root@localhost
> - ssh+ansible:root@localhost
> - winrm:<service address>
>
> (obviously in the above example we have a both Windows and RHEL box,
> which is wrong, but it'll do for this proposal :) ).
>
> An external service would build a graph and would allow for searches on
> a graph given some criteria.
>
> Another thought I will mention is that the way our entities are
> implemented, they give the impression that Brooklyn is more of a CM
> tools with the logic of deploying provided in Java. This is both not
> accurate and doesn't do Brooklyn justice. It is imo next to impossible
> to build an entity that would satisfy a large majority of users. The
> complexity becomes prohibitive. For that reason I see Brooklyn more of
> an enabler for implementers of entities to provide and promote their
> implementation. E.g. it's next to impossible to implement a best,
> complete entity for mysql, and it's more likely to see: hadrian-mysql
> (that has some characteristics, say it's super secure, deployed with
> bash, only works on ubuntu), and andrea-mysql (which is deployed with
> chef and is supper fast) and cipi-mysql (deployed in a cluster topology,
> super resilient). A user would pick the one they prefer and use it in
> their blueprint. This is no different than the kind of ecosystem docker,
> chef, puppet, etc, created.
>
> I'll stop here hoping for some feedback. This proposal has a more
> significant backwards compatibility impact and should be considered
> independently of the other proposal related to locations.
>
> Cheers,
> Hadrian
>
>
>
>
>