You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Darren Shepherd <da...@gmail.com> on 2013/09/18 00:10:31 UTC
[PROPOSAL] Modularize Spring
Currently ACS code is fairly modular in that you can add plug-ins to ACS to
extend most functionality. Unfortunately ACS is not packaged in a modular
way. It is still delivered essentially as one large unit. There are many
reason for this but one large barrier to modularizing ACS is that the
Spring configuration is managed as one large unit.
I propose that we break apart the Spring XML configuration such that each
component contributes its own configuration. Additionally each component
will be loaded into its own Spring ApplicationContext such that its beans
will not conflict with the wiring of other beans in ACS. This change will
lay the foundation for a richer plugin model and additionally a more
distributed architecture.
The technical details for this proposal can be found on the wiki [1].
Darren
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Spring
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Are you referring to what I have on review board? Any requests I have on
review board at the moment have no interdependencies. They are just a
bunch of random things I found while doing analysis and work for this
proposal.
Darren
On Wed, Sep 18, 2013 at 4:41 AM, Daan Hoogland <da...@gmail.com>wrote:
> Darren,
>
> I had a quick look at your four submissions, Is there no dependency
> between them? if so, can you add those to the review requests?
>
> thanks,
> Daan
>
> On Wed, Sep 18, 2013 at 1:25 PM, Daan Hoogland <da...@gmail.com>
> wrote:
> > sounds great Darren,
> >
> > By component, you mean maven project or some larger chunk like
> > distribution package? (did i miss this definition somewhere or do we
> > define the components now?)
> >
> > regards,
> > Daan
> >
> > On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> > <da...@gmail.com> wrote:
> >> Currently ACS code is fairly modular in that you can add plug-ins to
> ACS to
> >> extend most functionality. Unfortunately ACS is not packaged in a
> modular
> >> way. It is still delivered essentially as one large unit. There are
> many
> >> reason for this but one large barrier to modularizing ACS is that the
> >> Spring configuration is managed as one large unit.
> >>
> >> I propose that we break apart the Spring XML configuration such that
> each
> >> component contributes its own configuration. Additionally each
> component
> >> will be loaded into its own Spring ApplicationContext such that its
> beans
> >> will not conflict with the wiring of other beans in ACS. This change
> will
> >> lay the foundation for a richer plugin model and additionally a more
> >> distributed architecture.
> >>
> >> The technical details for this proposal can be found on the wiki [1].
> >>
> >> Darren
> >>
> >> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Spring
>
Re: [PROPOSAL] Modularize Spring
Posted by Daan Hoogland <da...@gmail.com>.
Darren,
I had a quick look at your four submissions, Is there no dependency
between them? if so, can you add those to the review requests?
thanks,
Daan
On Wed, Sep 18, 2013 at 1:25 PM, Daan Hoogland <da...@gmail.com> wrote:
> sounds great Darren,
>
> By component, you mean maven project or some larger chunk like
> distribution package? (did i miss this definition somewhere or do we
> define the components now?)
>
> regards,
> Daan
>
> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> <da...@gmail.com> wrote:
>> Currently ACS code is fairly modular in that you can add plug-ins to ACS to
>> extend most functionality. Unfortunately ACS is not packaged in a modular
>> way. It is still delivered essentially as one large unit. There are many
>> reason for this but one large barrier to modularizing ACS is that the
>> Spring configuration is managed as one large unit.
>>
>> I propose that we break apart the Spring XML configuration such that each
>> component contributes its own configuration. Additionally each component
>> will be loaded into its own Spring ApplicationContext such that its beans
>> will not conflict with the wiring of other beans in ACS. This change will
>> lay the foundation for a richer plugin model and additionally a more
>> distributed architecture.
>>
>> The technical details for this proposal can be found on the wiki [1].
>>
>> Darren
>>
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Spring
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Sounds great. What's on the spring modularization will support this
(obviously the storage framework needs to handle the canHandle logic),
but it seems a lot of the storage code is being refactored, so we'll
probably have to do a nasty merge.
Darren
On Wed, Oct 9, 2013 at 6:03 PM, SuichII, Christopher
<Ch...@netapp.com> wrote:
> Yep, that's the idea. In fact, we really need it to be dynamic so that the strategy or provider can respond depending on the action. For example, snapshotting may be supported while reverting may not be.
>
> --
> Chris Suich
> chris.suich@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 9, 2013, at 8:01 PM, Darren Shepherd <da...@gmail.com>
> wrote:
>
>> I think I'm fine with that. Is the enum type return dynamically at
>> runtime. So the API would be something like "PlugInPriority
>> canHandle(...)"?
>>
>> Darren
>>
>> On Wed, Oct 9, 2013 at 1:13 PM, SuichII, Christopher
>> <Ch...@netapp.com> wrote:
>>> I think I'll look into a version of (2). The difference being that I think using an int is too large of a range and provides unnecessary granularity. If two strategies or providers both have snapshot strategies, they are both simply going to return the max int. However, if we use an enum with values like:
>>>
>>> HIGHEST, PLUGIN, HYPERVISOR, DEFAULT and NO, (HIGHEST would be reserved for unforeseen future use, testing, simulators, etc.)
>>>
>>> then we allow strategies and providers to fall in the same bucket. All strategies and providers would be sorted and asked to handle operations in that order. Ultimately, this requires that plugins do their best to determine whether they can actually handle an operation, because if two say they can, there is no way for the MS to intelligently choose between the two.
>>>
>>> --
>>> Chris Suich
>>> chris.suich@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>> On Oct 4, 2013, at 6:10 PM, Darren Shepherd <da...@gmail.com> wrote:
>>>
>>>> Sure, I'm open to suggestions. Basically I think we've discussed
>>>>
>>>> 1) Global Setting
>>>> 2) canHandle() returns an int
>>>> 3) Strategy has an enum type assigned
>>>>
>>>> I'm open to all three, I don't have much vested interest in this.
>>>>
>>>> Darren
>>>>
>>>> On Fri, Oct 4, 2013 at 3:00 PM, SuichII, Christopher
>>>> <Ch...@netapp.com> wrote:
>>>>> Well, it seems OK, but I think we should keep on discussing our options. One concern I have with the global config approach is that it adds manual steps for 'installing' extensions. Each extension must have installation instructions to indicate which global configurations it must be included in and where in that list it should be put (and of course, many extension are going to say that they should be at the front of the list).
>>>>>
>>>>> -Chris
>>>>> --
>>>>> Chris Suich
>>>>> chris.suich@netapp.com
>>>>> NetApp Software Engineer
>>>>> Data Center Platforms – Cloud Solutions
>>>>> Citrix, Cisco & Red Hat
>>>>>
>>>>> On Oct 4, 2013, at 12:12 PM, Darren Shepherd <da...@gmail.com> wrote:
>>>>>
>>>>>> On 10/04/2013 11:58 AM, SuichII, Christopher wrote:
>>>>>>> Darren,
>>>>>>>
>>>>>>> I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
>>>>>>>
>>>>>>> -Chris
>>>>>>>
>>>>>>
>>>>>> I would *prefer* extensions to be order independent, but if we determine they are order dependant, then that is fine too. So if we conclude that the simplest way to address this is to order the Strategies based on configuration, then I will add an ordering "global configuration" as described at https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions.
>>>>>>
>>>>>> Does the order configuration setting approach seem fine?
>>>>>>
>>>>>> Darren
>>>>>
>>>
>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
Yep, that's the idea. In fact, we really need it to be dynamic so that the strategy or provider can respond depending on the action. For example, snapshotting may be supported while reverting may not be.
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Oct 9, 2013, at 8:01 PM, Darren Shepherd <da...@gmail.com>
wrote:
> I think I'm fine with that. Is the enum type return dynamically at
> runtime. So the API would be something like "PlugInPriority
> canHandle(...)"?
>
> Darren
>
> On Wed, Oct 9, 2013 at 1:13 PM, SuichII, Christopher
> <Ch...@netapp.com> wrote:
>> I think I'll look into a version of (2). The difference being that I think using an int is too large of a range and provides unnecessary granularity. If two strategies or providers both have snapshot strategies, they are both simply going to return the max int. However, if we use an enum with values like:
>>
>> HIGHEST, PLUGIN, HYPERVISOR, DEFAULT and NO, (HIGHEST would be reserved for unforeseen future use, testing, simulators, etc.)
>>
>> then we allow strategies and providers to fall in the same bucket. All strategies and providers would be sorted and asked to handle operations in that order. Ultimately, this requires that plugins do their best to determine whether they can actually handle an operation, because if two say they can, there is no way for the MS to intelligently choose between the two.
>>
>> --
>> Chris Suich
>> chris.suich@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Oct 4, 2013, at 6:10 PM, Darren Shepherd <da...@gmail.com> wrote:
>>
>>> Sure, I'm open to suggestions. Basically I think we've discussed
>>>
>>> 1) Global Setting
>>> 2) canHandle() returns an int
>>> 3) Strategy has an enum type assigned
>>>
>>> I'm open to all three, I don't have much vested interest in this.
>>>
>>> Darren
>>>
>>> On Fri, Oct 4, 2013 at 3:00 PM, SuichII, Christopher
>>> <Ch...@netapp.com> wrote:
>>>> Well, it seems OK, but I think we should keep on discussing our options. One concern I have with the global config approach is that it adds manual steps for 'installing' extensions. Each extension must have installation instructions to indicate which global configurations it must be included in and where in that list it should be put (and of course, many extension are going to say that they should be at the front of the list).
>>>>
>>>> -Chris
>>>> --
>>>> Chris Suich
>>>> chris.suich@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Oct 4, 2013, at 12:12 PM, Darren Shepherd <da...@gmail.com> wrote:
>>>>
>>>>> On 10/04/2013 11:58 AM, SuichII, Christopher wrote:
>>>>>> Darren,
>>>>>>
>>>>>> I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
>>>>>>
>>>>>> -Chris
>>>>>>
>>>>>
>>>>> I would *prefer* extensions to be order independent, but if we determine they are order dependant, then that is fine too. So if we conclude that the simplest way to address this is to order the Strategies based on configuration, then I will add an ordering "global configuration" as described at https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions.
>>>>>
>>>>> Does the order configuration setting approach seem fine?
>>>>>
>>>>> Darren
>>>>
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
I think I'm fine with that. Is the enum type return dynamically at
runtime. So the API would be something like "PlugInPriority
canHandle(...)"?
Darren
On Wed, Oct 9, 2013 at 1:13 PM, SuichII, Christopher
<Ch...@netapp.com> wrote:
> I think I'll look into a version of (2). The difference being that I think using an int is too large of a range and provides unnecessary granularity. If two strategies or providers both have snapshot strategies, they are both simply going to return the max int. However, if we use an enum with values like:
>
> HIGHEST, PLUGIN, HYPERVISOR, DEFAULT and NO, (HIGHEST would be reserved for unforeseen future use, testing, simulators, etc.)
>
> then we allow strategies and providers to fall in the same bucket. All strategies and providers would be sorted and asked to handle operations in that order. Ultimately, this requires that plugins do their best to determine whether they can actually handle an operation, because if two say they can, there is no way for the MS to intelligently choose between the two.
>
> --
> Chris Suich
> chris.suich@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 4, 2013, at 6:10 PM, Darren Shepherd <da...@gmail.com> wrote:
>
>> Sure, I'm open to suggestions. Basically I think we've discussed
>>
>> 1) Global Setting
>> 2) canHandle() returns an int
>> 3) Strategy has an enum type assigned
>>
>> I'm open to all three, I don't have much vested interest in this.
>>
>> Darren
>>
>> On Fri, Oct 4, 2013 at 3:00 PM, SuichII, Christopher
>> <Ch...@netapp.com> wrote:
>>> Well, it seems OK, but I think we should keep on discussing our options. One concern I have with the global config approach is that it adds manual steps for 'installing' extensions. Each extension must have installation instructions to indicate which global configurations it must be included in and where in that list it should be put (and of course, many extension are going to say that they should be at the front of the list).
>>>
>>> -Chris
>>> --
>>> Chris Suich
>>> chris.suich@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>> On Oct 4, 2013, at 12:12 PM, Darren Shepherd <da...@gmail.com> wrote:
>>>
>>>> On 10/04/2013 11:58 AM, SuichII, Christopher wrote:
>>>>> Darren,
>>>>>
>>>>> I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
>>>>>
>>>>> -Chris
>>>>>
>>>>
>>>> I would *prefer* extensions to be order independent, but if we determine they are order dependant, then that is fine too. So if we conclude that the simplest way to address this is to order the Strategies based on configuration, then I will add an ordering "global configuration" as described at https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions.
>>>>
>>>> Does the order configuration setting approach seem fine?
>>>>
>>>> Darren
>>>
>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
I think I'll look into a version of (2). The difference being that I think using an int is too large of a range and provides unnecessary granularity. If two strategies or providers both have snapshot strategies, they are both simply going to return the max int. However, if we use an enum with values like:
HIGHEST, PLUGIN, HYPERVISOR, DEFAULT and NO, (HIGHEST would be reserved for unforeseen future use, testing, simulators, etc.)
then we allow strategies and providers to fall in the same bucket. All strategies and providers would be sorted and asked to handle operations in that order. Ultimately, this requires that plugins do their best to determine whether they can actually handle an operation, because if two say they can, there is no way for the MS to intelligently choose between the two.
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Oct 4, 2013, at 6:10 PM, Darren Shepherd <da...@gmail.com> wrote:
> Sure, I'm open to suggestions. Basically I think we've discussed
>
> 1) Global Setting
> 2) canHandle() returns an int
> 3) Strategy has an enum type assigned
>
> I'm open to all three, I don't have much vested interest in this.
>
> Darren
>
> On Fri, Oct 4, 2013 at 3:00 PM, SuichII, Christopher
> <Ch...@netapp.com> wrote:
>> Well, it seems OK, but I think we should keep on discussing our options. One concern I have with the global config approach is that it adds manual steps for 'installing' extensions. Each extension must have installation instructions to indicate which global configurations it must be included in and where in that list it should be put (and of course, many extension are going to say that they should be at the front of the list).
>>
>> -Chris
>> --
>> Chris Suich
>> chris.suich@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Oct 4, 2013, at 12:12 PM, Darren Shepherd <da...@gmail.com> wrote:
>>
>>> On 10/04/2013 11:58 AM, SuichII, Christopher wrote:
>>>> Darren,
>>>>
>>>> I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
>>>>
>>>> -Chris
>>>>
>>>
>>> I would *prefer* extensions to be order independent, but if we determine they are order dependant, then that is fine too. So if we conclude that the simplest way to address this is to order the Strategies based on configuration, then I will add an ordering "global configuration" as described at https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions.
>>>
>>> Does the order configuration setting approach seem fine?
>>>
>>> Darren
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Sure, I'm open to suggestions. Basically I think we've discussed
1) Global Setting
2) canHandle() returns an int
3) Strategy has an enum type assigned
I'm open to all three, I don't have much vested interest in this.
Darren
On Fri, Oct 4, 2013 at 3:00 PM, SuichII, Christopher
<Ch...@netapp.com> wrote:
> Well, it seems OK, but I think we should keep on discussing our options. One concern I have with the global config approach is that it adds manual steps for 'installing' extensions. Each extension must have installation instructions to indicate which global configurations it must be included in and where in that list it should be put (and of course, many extension are going to say that they should be at the front of the list).
>
> -Chris
> --
> Chris Suich
> chris.suich@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 4, 2013, at 12:12 PM, Darren Shepherd <da...@gmail.com> wrote:
>
>> On 10/04/2013 11:58 AM, SuichII, Christopher wrote:
>>> Darren,
>>>
>>> I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
>>>
>>> -Chris
>>>
>>
>> I would *prefer* extensions to be order independent, but if we determine they are order dependant, then that is fine too. So if we conclude that the simplest way to address this is to order the Strategies based on configuration, then I will add an ordering "global configuration" as described at https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions.
>>
>> Does the order configuration setting approach seem fine?
>>
>> Darren
>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
Well, it seems OK, but I think we should keep on discussing our options. One concern I have with the global config approach is that it adds manual steps for 'installing' extensions. Each extension must have installation instructions to indicate which global configurations it must be included in and where in that list it should be put (and of course, many extension are going to say that they should be at the front of the list).
-Chris
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Oct 4, 2013, at 12:12 PM, Darren Shepherd <da...@gmail.com> wrote:
> On 10/04/2013 11:58 AM, SuichII, Christopher wrote:
>> Darren,
>>
>> I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
>>
>> -Chris
>>
>
> I would *prefer* extensions to be order independent, but if we determine they are order dependant, then that is fine too. So if we conclude that the simplest way to address this is to order the Strategies based on configuration, then I will add an ordering "global configuration" as described at https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions.
>
> Does the order configuration setting approach seem fine?
>
> Darren
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
On 10/04/2013 11:58 AM, SuichII, Christopher wrote:
> Darren,
>
> I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
>
> -Chris
>
I would *prefer* extensions to be order independent, but if we determine
they are order dependant, then that is fine too. So if we conclude that
the simplest way to address this is to order the Strategies based on
configuration, then I will add an ordering "global configuration" as
described at
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions.
Does the order configuration setting approach seem fine?
Darren
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
Darren,
I think one of the benefits of allowing the priority to be specified in the xml is that it can be configured after deployment. If for some reason two strategies or providers conflict, then their priorities can be changed in XML to resolve the conflict. I believe the Spring @Order annotation an be specified in XML, not just as an annotation.
-Chris
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Oct 4, 2013, at 10:36 AM, Edison Su <Ed...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: SuichII, Christopher [mailto:Chris.Suich@netapp.com]
>> Sent: Thursday, October 03, 2013 9:02 PM
>> To: <de...@cloudstack.apache.org>
>> Subject: Re: [PROPOSAL] Modularize Spring
>>
>> Sure - I could see that working. Anyone have thoughts whether an enum
>> could be used instead of an integer? That way we can provide categories or a
>> well defined scale (like 0-5)? If we give them free range of 1-100 (or any
>> integer range), I imagine people will likely go to the extremes and just use 0
>> for can't handle, 1 for low priority (default) and 100 for high priority (storage
>> providers). We still have the problem of handling conflicts, or implementers
>> who return the same value. However, I'm not sure there is much we can do
>> beyond selecting the first implementation that we come across with the
>> highest priority. We should document and suggest that implementations
>> ensure their canHandle() method is as intelligent as possible and only takes
>> control of operations they are truly authorities on.
>
> How to use the way spring 4.0 did:
> https://jira.springsource.org/browse/SPR-5574?
> http://stackoverflow.com/questions/16967971/spring-ordered-list-of-beans
> Add order in bean.
>
>>
>> -Chris
>> --
>> Chris Suich
>> chris.suich@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms - Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Oct 3, 2013, at 7:44 PM, Darren Shepherd <da...@gmail.com>
>> wrote:
>>
>>> Could it be just as simple as the canhandle() returns an int and not Boolean.
>> So the ancient would return 1 but if the netapp matches it returns 100. If it
>> does not handle at all you return -1. This effectively gives you prioritization.
>> So the calling code would still loop through all strategies each time looking for
>> the best match. I don't want the priority to be determined at load time as
>> that is less flexible.
>>>
>>> Darren
>>>
>>>> On Oct 3, 2013, at 5:42 PM, "SuichII, Christopher"
>> <Ch...@netapp.com> wrote:
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> After talking with Edison, I think what we could do is allow the strategies
>> to either specify a priority or 'type' and then order them by that when they
>> are loaded. For example, we could have an enum of types like 'PLUGIN,
>> HYPERVISOR and DEFAULT' so that we can make sure plugins are asked
>> canHandle() first, then hypervisor implementations, then finally resort to the
>> default/ancient implementation. This 'type' or 'category' could be specified
>> as a bean parameter or as part of the strategy/provider interface. Any
>> thoughts on which are better?
>>>>
>>>> The problem with just making canHandle() more intelligent is that we do
>> need some kind of ordering. Ideally, the non-default implementations
>> should be asked first, then fall back onto the stock implementations. You saw
>> the problem yourself - the XenServerSnapshotStrategy just says it will handle
>> all requests, so if a non-standard strategy wants to be given a chance, it must
>> be asked before the hypervisor or ancient implementation.
>>>>
>>>> Alternatively if this matches the same usage of the global configuration
>> ordering system, we could add in the storage subsystem strategies and
>> providers.
>>>>
>>>> The reason for log4j changes is that we may want to enable a different
>> level of verbosity by default. Our support organization likes to have very
>> verbose logs to assist them in triaging support issues. The lowest level log4j
>> categories are 'com.cloud', 'org' and 'net' and those are set to DEBUG and
>> INFO. Even if we add a line for 'com', the default value should not be 'TRACE'
>> like we would like ours to be. I'm not all that great with log4j, though, so
>> maybe I'm missing a simpler solution.
>>>>
>>>> I'll try to keep an eye on the commands.properties/rbac stuff - that is
>> good to know.
>>>>
>>>> Thanks,
>>>> Chris
>>>> --
>>>> Chris Suich
>>>> chris.suich@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat
>>>>
>>>>> On Oct 3, 2013, at 4:30 PM, Darren Shepherd
>> <da...@gmail.com> wrote:
>>>>>
>>>>> I forgot to mention about canHandle() ordering. For extensions, it
>>>>> should be preferred that we do not rely on loaded or configured order.
>>>>> The canHandle() implementation should assume that they're may be any
>>>>> order. So having said that, I looked at XenServerSnapshotStrategy
>>>>> and its canHandle() is brilliantly implemented as "return true;"
>>>>>
>>>>> Can we look at making the strategies order independent and not
>>>>> having another order configuration parameter?
>>>>>
>>>>> Darren
>>>>>
>>>>> On Thu, Oct 3, 2013 at 4:26 PM, Darren Shepherd
>>>>> <da...@gmail.com> wrote:
>>>>>> Chris,
>>>>>>
>>>>>> I've updated the wiki [1]. Copying from the wiki
>>>>>>
>>>>>> "Extensions are automatically discovered based on the interfaces
>>>>>> they implement and which module is their parent. For example, if
>>>>>> you place a storage extension in a child module of the network
>>>>>> module, it will not be discovered. Additionally, depending on the
>>>>>> extension, the ordering may be important in how they extension is
>>>>>> used. For the extensions that ordering is important there is also a
>>>>>> "Global Configuration" setting for the ordering. The value of the
>>>>>> setting is a comma seperated list like
>>>>>> SHA256SALT,MD5,LDAP,PLAINTEXT. The values are the names of the
>>>>>> extension. The name will be the result of getName(), if the bean
>>>>>> implements the Named interface, or the sort class name (ie
>> getClass().getSimpleName())."
>>>>>>
>>>>>> I setup all the extensible types based on what was in
>>>>>> componentContext.xml and also trial an error. The types you have
>>>>>> mentioned were not in componetContext.xml so I missed them. It
>>>>>> works today, but is not properly extensible. I will update the
>>>>>> configuration and wiki today.
>>>>>>
>>>>>> I've haven't made progress on the commands.properties as I realized
>>>>>> it will overlap with the rbac work being done. I will start a
>>>>>> thread regarding that so maybe it will be address in the rbac work.
>>>>>>
>>>>>> Regarding log4j. First, why do you have to change that?
>>>>>> Regardless, at the end of the day you are limited to whatever log4j can
>> provide.
>>>>>> So log4j AFAIK does not support multiple log4j config files so the
>>>>>> best we can do is allow somebody to supply a different log4j config
>>>>>> file and configure cloudstack to use that. One of the things I
>>>>>> don't particularily like is that ACS forces log4j to its own
>>>>>> configuration file and will not allow the regular log4j config
>>>>>> discovery. So what I could do is add a JVM property like
>>>>>> -Dcloudstack.log4jstandardconfig that will disable the ACS behavior
>>>>>> of configuring log4j and then you can use the log4j standard approach
>> [2].
>>>>>>
>>>>>> Darren
>>>>>>
>>>>>> [1]
>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions
>>>>>> [2] http://logging.apache.org/log4j/1.2/manual.html
>>>>>>
>>>>>> On Thu, Oct 3, 2013 at 3:29 PM, SuichII, Christopher
>>>>>> <Ch...@netapp.com> wrote:
>>>>>>> Darren,
>>>>>>>
>>>>>>> I read through this thread and your docs on the wiki, but I'd
>> appreciate it if you could answer a couple questions for me:
>>>>>>>
>>>>>>> -When creating an extension, such as a DataStoreProvider, those
>> items are currently added into the list of providers for the appropriate bean,
>> such as:
>>>>>>> <bean id="dataStoreProviderManager"
>>>>>>>
>>>>>>> class="org.apache.cloudstack.storage.datastore.provider.DataStoreP
>>>>>>> roviderManagerImpl">
>>>>>>> <property name="providers">
>>>>>>> <list merge="true">
>>>>>>> <ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
>>>>>>> <ref local="cloudStackImageStoreProviderImpl"/>
>>>>>>> <ref local="s3ImageStoreProviderImpl"/>
>>>>>>> <ref local="swiftImageStoreProviderImpl"/>
>>>>>>> <ref local="solidFireDataStoreProvider"/>
>>>>>>> </list>
>>>>>>> </property>
>>>>>>> </bean>
>>>>>>>
>>>>>>> So, how do we add our bean to that list?
>>>>>>>
>>>>>>> -There are a number of extensions that are not currently listed, such
>> as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are
>> omitted from
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
>>>>>>>
>>>>>>> -I know somewhere in this thread you talked about the order of
>> beans, but can you document how the ordering or precedence works in the
>> wiki? For example, if I create a DataMotionStrategy, how do I ensure that my
>> strategy's canHandle() method is invoked before the
>> AncientDataStoreStrategy?
>>>>>>>
>>>>>>> -Is there any progress on modularizing commands.properties and the
>> log4j configuration?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Chris
>>>>>>> --
>>>>>>> Chris Suich
>>>>>>> chris.suich@netapp.com<ma...@netapp.com>
>>>>>>> NetApp Software Engineer
>>>>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat
>>>>
>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
On 10/04/2013 10:36 AM, Edison Su wrote:
>
> How to use the way spring 4.0 did:
> https://jira.springsource.org/browse/SPR-5574?
> http://stackoverflow.com/questions/16967971/spring-ordered-list-of-beans
> Add order in bean.
>
That is a spring specific API/interface. In general we should never
using Spring APIs. Only bootstrap, initialization, or naturally Spring
specific code should use Spring APIs. Currently ACS does not use Spring
APIs with the exception of @Component which is only there until I get
around to cleaning up AWSAPI and Usage Server which use component
scanning still.
Regardless, that is Spring 4.0, which we don't use, and the extension
discovery now is not based on Injection in a way that the code in Spring
4.0 would actually sort the beans.
Darren
RE: [PROPOSAL] Modularize Spring
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: SuichII, Christopher [mailto:Chris.Suich@netapp.com]
> Sent: Thursday, October 03, 2013 9:02 PM
> To: <de...@cloudstack.apache.org>
> Subject: Re: [PROPOSAL] Modularize Spring
>
> Sure - I could see that working. Anyone have thoughts whether an enum
> could be used instead of an integer? That way we can provide categories or a
> well defined scale (like 0-5)? If we give them free range of 1-100 (or any
> integer range), I imagine people will likely go to the extremes and just use 0
> for can't handle, 1 for low priority (default) and 100 for high priority (storage
> providers). We still have the problem of handling conflicts, or implementers
> who return the same value. However, I'm not sure there is much we can do
> beyond selecting the first implementation that we come across with the
> highest priority. We should document and suggest that implementations
> ensure their canHandle() method is as intelligent as possible and only takes
> control of operations they are truly authorities on.
How to use the way spring 4.0 did:
https://jira.springsource.org/browse/SPR-5574?
http://stackoverflow.com/questions/16967971/spring-ordered-list-of-beans
Add order in bean.
>
> -Chris
> --
> Chris Suich
> chris.suich@netapp.com
> NetApp Software Engineer
> Data Center Platforms - Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Oct 3, 2013, at 7:44 PM, Darren Shepherd <da...@gmail.com>
> wrote:
>
> > Could it be just as simple as the canhandle() returns an int and not Boolean.
> So the ancient would return 1 but if the netapp matches it returns 100. If it
> does not handle at all you return -1. This effectively gives you prioritization.
> So the calling code would still loop through all strategies each time looking for
> the best match. I don't want the priority to be determined at load time as
> that is less flexible.
> >
> > Darren
> >
> >> On Oct 3, 2013, at 5:42 PM, "SuichII, Christopher"
> <Ch...@netapp.com> wrote:
> >>
> >> Thanks for the quick reply.
> >>
> >> After talking with Edison, I think what we could do is allow the strategies
> to either specify a priority or 'type' and then order them by that when they
> are loaded. For example, we could have an enum of types like 'PLUGIN,
> HYPERVISOR and DEFAULT' so that we can make sure plugins are asked
> canHandle() first, then hypervisor implementations, then finally resort to the
> default/ancient implementation. This 'type' or 'category' could be specified
> as a bean parameter or as part of the strategy/provider interface. Any
> thoughts on which are better?
> >>
> >> The problem with just making canHandle() more intelligent is that we do
> need some kind of ordering. Ideally, the non-default implementations
> should be asked first, then fall back onto the stock implementations. You saw
> the problem yourself - the XenServerSnapshotStrategy just says it will handle
> all requests, so if a non-standard strategy wants to be given a chance, it must
> be asked before the hypervisor or ancient implementation.
> >>
> >> Alternatively if this matches the same usage of the global configuration
> ordering system, we could add in the storage subsystem strategies and
> providers.
> >>
> >> The reason for log4j changes is that we may want to enable a different
> level of verbosity by default. Our support organization likes to have very
> verbose logs to assist them in triaging support issues. The lowest level log4j
> categories are 'com.cloud', 'org' and 'net' and those are set to DEBUG and
> INFO. Even if we add a line for 'com', the default value should not be 'TRACE'
> like we would like ours to be. I'm not all that great with log4j, though, so
> maybe I'm missing a simpler solution.
> >>
> >> I'll try to keep an eye on the commands.properties/rbac stuff - that is
> good to know.
> >>
> >> Thanks,
> >> Chris
> >> --
> >> Chris Suich
> >> chris.suich@netapp.com
> >> NetApp Software Engineer
> >> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat
> >>
> >>> On Oct 3, 2013, at 4:30 PM, Darren Shepherd
> <da...@gmail.com> wrote:
> >>>
> >>> I forgot to mention about canHandle() ordering. For extensions, it
> >>> should be preferred that we do not rely on loaded or configured order.
> >>> The canHandle() implementation should assume that they're may be any
> >>> order. So having said that, I looked at XenServerSnapshotStrategy
> >>> and its canHandle() is brilliantly implemented as "return true;"
> >>>
> >>> Can we look at making the strategies order independent and not
> >>> having another order configuration parameter?
> >>>
> >>> Darren
> >>>
> >>> On Thu, Oct 3, 2013 at 4:26 PM, Darren Shepherd
> >>> <da...@gmail.com> wrote:
> >>>> Chris,
> >>>>
> >>>> I've updated the wiki [1]. Copying from the wiki
> >>>>
> >>>> "Extensions are automatically discovered based on the interfaces
> >>>> they implement and which module is their parent. For example, if
> >>>> you place a storage extension in a child module of the network
> >>>> module, it will not be discovered. Additionally, depending on the
> >>>> extension, the ordering may be important in how they extension is
> >>>> used. For the extensions that ordering is important there is also a
> >>>> "Global Configuration" setting for the ordering. The value of the
> >>>> setting is a comma seperated list like
> >>>> SHA256SALT,MD5,LDAP,PLAINTEXT. The values are the names of the
> >>>> extension. The name will be the result of getName(), if the bean
> >>>> implements the Named interface, or the sort class name (ie
> getClass().getSimpleName())."
> >>>>
> >>>> I setup all the extensible types based on what was in
> >>>> componentContext.xml and also trial an error. The types you have
> >>>> mentioned were not in componetContext.xml so I missed them. It
> >>>> works today, but is not properly extensible. I will update the
> >>>> configuration and wiki today.
> >>>>
> >>>> I've haven't made progress on the commands.properties as I realized
> >>>> it will overlap with the rbac work being done. I will start a
> >>>> thread regarding that so maybe it will be address in the rbac work.
> >>>>
> >>>> Regarding log4j. First, why do you have to change that?
> >>>> Regardless, at the end of the day you are limited to whatever log4j can
> provide.
> >>>> So log4j AFAIK does not support multiple log4j config files so the
> >>>> best we can do is allow somebody to supply a different log4j config
> >>>> file and configure cloudstack to use that. One of the things I
> >>>> don't particularily like is that ACS forces log4j to its own
> >>>> configuration file and will not allow the regular log4j config
> >>>> discovery. So what I could do is add a JVM property like
> >>>> -Dcloudstack.log4jstandardconfig that will disable the ACS behavior
> >>>> of configuring log4j and then you can use the log4j standard approach
> [2].
> >>>>
> >>>> Darren
> >>>>
> >>>> [1]
> >>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions
> >>>> [2] http://logging.apache.org/log4j/1.2/manual.html
> >>>>
> >>>> On Thu, Oct 3, 2013 at 3:29 PM, SuichII, Christopher
> >>>> <Ch...@netapp.com> wrote:
> >>>>> Darren,
> >>>>>
> >>>>> I read through this thread and your docs on the wiki, but I'd
> appreciate it if you could answer a couple questions for me:
> >>>>>
> >>>>> -When creating an extension, such as a DataStoreProvider, those
> items are currently added into the list of providers for the appropriate bean,
> such as:
> >>>>> <bean id="dataStoreProviderManager"
> >>>>>
> >>>>> class="org.apache.cloudstack.storage.datastore.provider.DataStoreP
> >>>>> roviderManagerImpl">
> >>>>> <property name="providers">
> >>>>> <list merge="true">
> >>>>> <ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
> >>>>> <ref local="cloudStackImageStoreProviderImpl"/>
> >>>>> <ref local="s3ImageStoreProviderImpl"/>
> >>>>> <ref local="swiftImageStoreProviderImpl"/>
> >>>>> <ref local="solidFireDataStoreProvider"/>
> >>>>> </list>
> >>>>> </property>
> >>>>> </bean>
> >>>>>
> >>>>> So, how do we add our bean to that list?
> >>>>>
> >>>>> -There are a number of extensions that are not currently listed, such
> as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are
> omitted from
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
> >>>>>
> >>>>> -I know somewhere in this thread you talked about the order of
> beans, but can you document how the ordering or precedence works in the
> wiki? For example, if I create a DataMotionStrategy, how do I ensure that my
> strategy's canHandle() method is invoked before the
> AncientDataStoreStrategy?
> >>>>>
> >>>>> -Is there any progress on modularizing commands.properties and the
> log4j configuration?
> >>>>>
> >>>>> Thanks,
> >>>>> Chris
> >>>>> --
> >>>>> Chris Suich
> >>>>> chris.suich@netapp.com<ma...@netapp.com>
> >>>>> NetApp Software Engineer
> >>>>> Data Center Platforms - Cloud Solutions Citrix, Cisco & Red Hat
> >>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
Sure - I could see that working. Anyone have thoughts whether an enum could be used instead of an integer? That way we can provide categories or a well defined scale (like 0-5)? If we give them free range of 1-100 (or any integer range), I imagine people will likely go to the extremes and just use 0 for can't handle, 1 for low priority (default) and 100 for high priority (storage providers). We still have the problem of handling conflicts, or implementers who return the same value. However, I'm not sure there is much we can do beyond selecting the first implementation that we come across with the highest priority. We should document and suggest that implementations ensure their canHandle() method is as intelligent as possible and only takes control of operations they are truly authorities on.
-Chris
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Oct 3, 2013, at 7:44 PM, Darren Shepherd <da...@gmail.com> wrote:
> Could it be just as simple as the canhandle() returns an int and not Boolean. So the ancient would return 1 but if the netapp matches it returns 100. If it does not handle at all you return -1. This effectively gives you prioritization. So the calling code would still loop through all strategies each time looking for the best match. I don't want the priority to be determined at load time as that is less flexible.
>
> Darren
>
>> On Oct 3, 2013, at 5:42 PM, "SuichII, Christopher" <Ch...@netapp.com> wrote:
>>
>> Thanks for the quick reply.
>>
>> After talking with Edison, I think what we could do is allow the strategies to either specify a priority or 'type' and then order them by that when they are loaded. For example, we could have an enum of types like 'PLUGIN, HYPERVISOR and DEFAULT' so that we can make sure plugins are asked canHandle() first, then hypervisor implementations, then finally resort to the default/ancient implementation. This 'type' or 'category' could be specified as a bean parameter or as part of the strategy/provider interface. Any thoughts on which are better?
>>
>> The problem with just making canHandle() more intelligent is that we do need some kind of ordering. Ideally, the non-default implementations should be asked first, then fall back onto the stock implementations. You saw the problem yourself - the XenServerSnapshotStrategy just says it will handle all requests, so if a non-standard strategy wants to be given a chance, it must be asked before the hypervisor or ancient implementation.
>>
>> Alternatively if this matches the same usage of the global configuration ordering system, we could add in the storage subsystem strategies and providers.
>>
>> The reason for log4j changes is that we may want to enable a different level of verbosity by default. Our support organization likes to have very verbose logs to assist them in triaging support issues. The lowest level log4j categories are 'com.cloud', 'org' and 'net' and those are set to DEBUG and INFO. Even if we add a line for 'com', the default value should not be 'TRACE' like we would like ours to be. I'm not all that great with log4j, though, so maybe I'm missing a simpler solution.
>>
>> I'll try to keep an eye on the commands.properties/rbac stuff - that is good to know.
>>
>> Thanks,
>> Chris
>> --
>> Chris Suich
>> chris.suich@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>>> On Oct 3, 2013, at 4:30 PM, Darren Shepherd <da...@gmail.com> wrote:
>>>
>>> I forgot to mention about canHandle() ordering. For extensions, it
>>> should be preferred that we do not rely on loaded or configured order.
>>> The canHandle() implementation should assume that they're may be any
>>> order. So having said that, I looked at XenServerSnapshotStrategy and
>>> its canHandle() is brilliantly implemented as "return true;"
>>>
>>> Can we look at making the strategies order independent and not having
>>> another order configuration parameter?
>>>
>>> Darren
>>>
>>> On Thu, Oct 3, 2013 at 4:26 PM, Darren Shepherd
>>> <da...@gmail.com> wrote:
>>>> Chris,
>>>>
>>>> I've updated the wiki [1]. Copying from the wiki
>>>>
>>>> "Extensions are automatically discovered based on the interfaces they
>>>> implement and which module is their parent. For example, if you place
>>>> a storage extension in a child module of the network module, it will
>>>> not be discovered. Additionally, depending on the extension, the
>>>> ordering may be important in how they extension is used. For the
>>>> extensions that ordering is important there is also a "Global
>>>> Configuration" setting for the ordering. The value of the setting is a
>>>> comma seperated list like SHA256SALT,MD5,LDAP,PLAINTEXT. The values
>>>> are the names of the extension. The name will be the result of
>>>> getName(), if the bean implements the Named interface, or the sort
>>>> class name (ie getClass().getSimpleName())."
>>>>
>>>> I setup all the extensible types based on what was in
>>>> componentContext.xml and also trial an error. The types you have
>>>> mentioned were not in componetContext.xml so I missed them. It works
>>>> today, but is not properly extensible. I will update the
>>>> configuration and wiki today.
>>>>
>>>> I've haven't made progress on the commands.properties as I realized it
>>>> will overlap with the rbac work being done. I will start a thread
>>>> regarding that so maybe it will be address in the rbac work.
>>>>
>>>> Regarding log4j. First, why do you have to change that? Regardless,
>>>> at the end of the day you are limited to whatever log4j can provide.
>>>> So log4j AFAIK does not support multiple log4j config files so the
>>>> best we can do is allow somebody to supply a different log4j config
>>>> file and configure cloudstack to use that. One of the things I don't
>>>> particularily like is that ACS forces log4j to its own configuration
>>>> file and will not allow the regular log4j config discovery. So what I
>>>> could do is add a JVM property like -Dcloudstack.log4jstandardconfig
>>>> that will disable the ACS behavior of configuring log4j and then you
>>>> can use the log4j standard approach [2].
>>>>
>>>> Darren
>>>>
>>>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions
>>>> [2] http://logging.apache.org/log4j/1.2/manual.html
>>>>
>>>> On Thu, Oct 3, 2013 at 3:29 PM, SuichII, Christopher
>>>> <Ch...@netapp.com> wrote:
>>>>> Darren,
>>>>>
>>>>> I read through this thread and your docs on the wiki, but I'd appreciate it if you could answer a couple questions for me:
>>>>>
>>>>> -When creating an extension, such as a DataStoreProvider, those items are currently added into the list of providers for the appropriate bean, such as:
>>>>> <bean id="dataStoreProviderManager"
>>>>> class="org.apache.cloudstack.storage.datastore.provider.DataStoreProviderManagerImpl">
>>>>> <property name="providers">
>>>>> <list merge="true">
>>>>> <ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
>>>>> <ref local="cloudStackImageStoreProviderImpl"/>
>>>>> <ref local="s3ImageStoreProviderImpl"/>
>>>>> <ref local="swiftImageStoreProviderImpl"/>
>>>>> <ref local="solidFireDataStoreProvider"/>
>>>>> </list>
>>>>> </property>
>>>>> </bean>
>>>>>
>>>>> So, how do we add our bean to that list?
>>>>>
>>>>> -There are a number of extensions that are not currently listed, such as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are omitted from https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
>>>>>
>>>>> -I know somewhere in this thread you talked about the order of beans, but can you document how the ordering or precedence works in the wiki? For example, if I create a DataMotionStrategy, how do I ensure that my strategy's canHandle() method is invoked before the AncientDataStoreStrategy?
>>>>>
>>>>> -Is there any progress on modularizing commands.properties and the log4j configuration?
>>>>>
>>>>> Thanks,
>>>>> Chris
>>>>> --
>>>>> Chris Suich
>>>>> chris.suich@netapp.com<ma...@netapp.com>
>>>>> NetApp Software Engineer
>>>>> Data Center Platforms – Cloud Solutions
>>>>> Citrix, Cisco & Red Hat
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Could it be just as simple as the canhandle() returns an int and not Boolean. So the ancient would return 1 but if the netapp matches it returns 100. If it does not handle at all you return -1. This effectively gives you prioritization. So the calling code would still loop through all strategies each time looking for the best match. I don't want the priority to be determined at load time as that is less flexible.
Darren
> On Oct 3, 2013, at 5:42 PM, "SuichII, Christopher" <Ch...@netapp.com> wrote:
>
> Thanks for the quick reply.
>
> After talking with Edison, I think what we could do is allow the strategies to either specify a priority or 'type' and then order them by that when they are loaded. For example, we could have an enum of types like 'PLUGIN, HYPERVISOR and DEFAULT' so that we can make sure plugins are asked canHandle() first, then hypervisor implementations, then finally resort to the default/ancient implementation. This 'type' or 'category' could be specified as a bean parameter or as part of the strategy/provider interface. Any thoughts on which are better?
>
> The problem with just making canHandle() more intelligent is that we do need some kind of ordering. Ideally, the non-default implementations should be asked first, then fall back onto the stock implementations. You saw the problem yourself - the XenServerSnapshotStrategy just says it will handle all requests, so if a non-standard strategy wants to be given a chance, it must be asked before the hypervisor or ancient implementation.
>
> Alternatively if this matches the same usage of the global configuration ordering system, we could add in the storage subsystem strategies and providers.
>
> The reason for log4j changes is that we may want to enable a different level of verbosity by default. Our support organization likes to have very verbose logs to assist them in triaging support issues. The lowest level log4j categories are 'com.cloud', 'org' and 'net' and those are set to DEBUG and INFO. Even if we add a line for 'com', the default value should not be 'TRACE' like we would like ours to be. I'm not all that great with log4j, though, so maybe I'm missing a simpler solution.
>
> I'll try to keep an eye on the commands.properties/rbac stuff - that is good to know.
>
> Thanks,
> Chris
> --
> Chris Suich
> chris.suich@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
>> On Oct 3, 2013, at 4:30 PM, Darren Shepherd <da...@gmail.com> wrote:
>>
>> I forgot to mention about canHandle() ordering. For extensions, it
>> should be preferred that we do not rely on loaded or configured order.
>> The canHandle() implementation should assume that they're may be any
>> order. So having said that, I looked at XenServerSnapshotStrategy and
>> its canHandle() is brilliantly implemented as "return true;"
>>
>> Can we look at making the strategies order independent and not having
>> another order configuration parameter?
>>
>> Darren
>>
>> On Thu, Oct 3, 2013 at 4:26 PM, Darren Shepherd
>> <da...@gmail.com> wrote:
>>> Chris,
>>>
>>> I've updated the wiki [1]. Copying from the wiki
>>>
>>> "Extensions are automatically discovered based on the interfaces they
>>> implement and which module is their parent. For example, if you place
>>> a storage extension in a child module of the network module, it will
>>> not be discovered. Additionally, depending on the extension, the
>>> ordering may be important in how they extension is used. For the
>>> extensions that ordering is important there is also a "Global
>>> Configuration" setting for the ordering. The value of the setting is a
>>> comma seperated list like SHA256SALT,MD5,LDAP,PLAINTEXT. The values
>>> are the names of the extension. The name will be the result of
>>> getName(), if the bean implements the Named interface, or the sort
>>> class name (ie getClass().getSimpleName())."
>>>
>>> I setup all the extensible types based on what was in
>>> componentContext.xml and also trial an error. The types you have
>>> mentioned were not in componetContext.xml so I missed them. It works
>>> today, but is not properly extensible. I will update the
>>> configuration and wiki today.
>>>
>>> I've haven't made progress on the commands.properties as I realized it
>>> will overlap with the rbac work being done. I will start a thread
>>> regarding that so maybe it will be address in the rbac work.
>>>
>>> Regarding log4j. First, why do you have to change that? Regardless,
>>> at the end of the day you are limited to whatever log4j can provide.
>>> So log4j AFAIK does not support multiple log4j config files so the
>>> best we can do is allow somebody to supply a different log4j config
>>> file and configure cloudstack to use that. One of the things I don't
>>> particularily like is that ACS forces log4j to its own configuration
>>> file and will not allow the regular log4j config discovery. So what I
>>> could do is add a JVM property like -Dcloudstack.log4jstandardconfig
>>> that will disable the ACS behavior of configuring log4j and then you
>>> can use the log4j standard approach [2].
>>>
>>> Darren
>>>
>>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions
>>> [2] http://logging.apache.org/log4j/1.2/manual.html
>>>
>>> On Thu, Oct 3, 2013 at 3:29 PM, SuichII, Christopher
>>> <Ch...@netapp.com> wrote:
>>>> Darren,
>>>>
>>>> I read through this thread and your docs on the wiki, but I'd appreciate it if you could answer a couple questions for me:
>>>>
>>>> -When creating an extension, such as a DataStoreProvider, those items are currently added into the list of providers for the appropriate bean, such as:
>>>> <bean id="dataStoreProviderManager"
>>>> class="org.apache.cloudstack.storage.datastore.provider.DataStoreProviderManagerImpl">
>>>> <property name="providers">
>>>> <list merge="true">
>>>> <ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
>>>> <ref local="cloudStackImageStoreProviderImpl"/>
>>>> <ref local="s3ImageStoreProviderImpl"/>
>>>> <ref local="swiftImageStoreProviderImpl"/>
>>>> <ref local="solidFireDataStoreProvider"/>
>>>> </list>
>>>> </property>
>>>> </bean>
>>>>
>>>> So, how do we add our bean to that list?
>>>>
>>>> -There are a number of extensions that are not currently listed, such as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are omitted from https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
>>>>
>>>> -I know somewhere in this thread you talked about the order of beans, but can you document how the ordering or precedence works in the wiki? For example, if I create a DataMotionStrategy, how do I ensure that my strategy's canHandle() method is invoked before the AncientDataStoreStrategy?
>>>>
>>>> -Is there any progress on modularizing commands.properties and the log4j configuration?
>>>>
>>>> Thanks,
>>>> Chris
>>>> --
>>>> Chris Suich
>>>> chris.suich@netapp.com<ma...@netapp.com>
>>>> NetApp Software Engineer
>>>> Data Center Platforms – Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
Thanks for the quick reply.
After talking with Edison, I think what we could do is allow the strategies to either specify a priority or 'type' and then order them by that when they are loaded. For example, we could have an enum of types like 'PLUGIN, HYPERVISOR and DEFAULT' so that we can make sure plugins are asked canHandle() first, then hypervisor implementations, then finally resort to the default/ancient implementation. This 'type' or 'category' could be specified as a bean parameter or as part of the strategy/provider interface. Any thoughts on which are better?
The problem with just making canHandle() more intelligent is that we do need some kind of ordering. Ideally, the non-default implementations should be asked first, then fall back onto the stock implementations. You saw the problem yourself - the XenServerSnapshotStrategy just says it will handle all requests, so if a non-standard strategy wants to be given a chance, it must be asked before the hypervisor or ancient implementation.
Alternatively if this matches the same usage of the global configuration ordering system, we could add in the storage subsystem strategies and providers.
The reason for log4j changes is that we may want to enable a different level of verbosity by default. Our support organization likes to have very verbose logs to assist them in triaging support issues. The lowest level log4j categories are 'com.cloud', 'org' and 'net' and those are set to DEBUG and INFO. Even if we add a line for 'com', the default value should not be 'TRACE' like we would like ours to be. I'm not all that great with log4j, though, so maybe I'm missing a simpler solution.
I'll try to keep an eye on the commands.properties/rbac stuff - that is good to know.
Thanks,
Chris
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Oct 3, 2013, at 4:30 PM, Darren Shepherd <da...@gmail.com> wrote:
> I forgot to mention about canHandle() ordering. For extensions, it
> should be preferred that we do not rely on loaded or configured order.
> The canHandle() implementation should assume that they're may be any
> order. So having said that, I looked at XenServerSnapshotStrategy and
> its canHandle() is brilliantly implemented as "return true;"
>
> Can we look at making the strategies order independent and not having
> another order configuration parameter?
>
> Darren
>
> On Thu, Oct 3, 2013 at 4:26 PM, Darren Shepherd
> <da...@gmail.com> wrote:
>> Chris,
>>
>> I've updated the wiki [1]. Copying from the wiki
>>
>> "Extensions are automatically discovered based on the interfaces they
>> implement and which module is their parent. For example, if you place
>> a storage extension in a child module of the network module, it will
>> not be discovered. Additionally, depending on the extension, the
>> ordering may be important in how they extension is used. For the
>> extensions that ordering is important there is also a "Global
>> Configuration" setting for the ordering. The value of the setting is a
>> comma seperated list like SHA256SALT,MD5,LDAP,PLAINTEXT. The values
>> are the names of the extension. The name will be the result of
>> getName(), if the bean implements the Named interface, or the sort
>> class name (ie getClass().getSimpleName())."
>>
>> I setup all the extensible types based on what was in
>> componentContext.xml and also trial an error. The types you have
>> mentioned were not in componetContext.xml so I missed them. It works
>> today, but is not properly extensible. I will update the
>> configuration and wiki today.
>>
>> I've haven't made progress on the commands.properties as I realized it
>> will overlap with the rbac work being done. I will start a thread
>> regarding that so maybe it will be address in the rbac work.
>>
>> Regarding log4j. First, why do you have to change that? Regardless,
>> at the end of the day you are limited to whatever log4j can provide.
>> So log4j AFAIK does not support multiple log4j config files so the
>> best we can do is allow somebody to supply a different log4j config
>> file and configure cloudstack to use that. One of the things I don't
>> particularily like is that ACS forces log4j to its own configuration
>> file and will not allow the regular log4j config discovery. So what I
>> could do is add a JVM property like -Dcloudstack.log4jstandardconfig
>> that will disable the ACS behavior of configuring log4j and then you
>> can use the log4j standard approach [2].
>>
>> Darren
>>
>> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions
>> [2] http://logging.apache.org/log4j/1.2/manual.html
>>
>> On Thu, Oct 3, 2013 at 3:29 PM, SuichII, Christopher
>> <Ch...@netapp.com> wrote:
>>> Darren,
>>>
>>> I read through this thread and your docs on the wiki, but I'd appreciate it if you could answer a couple questions for me:
>>>
>>> -When creating an extension, such as a DataStoreProvider, those items are currently added into the list of providers for the appropriate bean, such as:
>>> <bean id="dataStoreProviderManager"
>>> class="org.apache.cloudstack.storage.datastore.provider.DataStoreProviderManagerImpl">
>>> <property name="providers">
>>> <list merge="true">
>>> <ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
>>> <ref local="cloudStackImageStoreProviderImpl"/>
>>> <ref local="s3ImageStoreProviderImpl"/>
>>> <ref local="swiftImageStoreProviderImpl"/>
>>> <ref local="solidFireDataStoreProvider"/>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> So, how do we add our bean to that list?
>>>
>>> -There are a number of extensions that are not currently listed, such as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are omitted from https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
>>>
>>> -I know somewhere in this thread you talked about the order of beans, but can you document how the ordering or precedence works in the wiki? For example, if I create a DataMotionStrategy, how do I ensure that my strategy's canHandle() method is invoked before the AncientDataStoreStrategy?
>>>
>>> -Is there any progress on modularizing commands.properties and the log4j configuration?
>>>
>>> Thanks,
>>> Chris
>>> --
>>> Chris Suich
>>> chris.suich@netapp.com<ma...@netapp.com>
>>> NetApp Software Engineer
>>> Data Center Platforms – Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
I forgot to mention about canHandle() ordering. For extensions, it
should be preferred that we do not rely on loaded or configured order.
The canHandle() implementation should assume that they're may be any
order. So having said that, I looked at XenServerSnapshotStrategy and
its canHandle() is brilliantly implemented as "return true;"
Can we look at making the strategies order independent and not having
another order configuration parameter?
Darren
On Thu, Oct 3, 2013 at 4:26 PM, Darren Shepherd
<da...@gmail.com> wrote:
> Chris,
>
> I've updated the wiki [1]. Copying from the wiki
>
> "Extensions are automatically discovered based on the interfaces they
> implement and which module is their parent. For example, if you place
> a storage extension in a child module of the network module, it will
> not be discovered. Additionally, depending on the extension, the
> ordering may be important in how they extension is used. For the
> extensions that ordering is important there is also a "Global
> Configuration" setting for the ordering. The value of the setting is a
> comma seperated list like SHA256SALT,MD5,LDAP,PLAINTEXT. The values
> are the names of the extension. The name will be the result of
> getName(), if the bean implements the Named interface, or the sort
> class name (ie getClass().getSimpleName())."
>
> I setup all the extensible types based on what was in
> componentContext.xml and also trial an error. The types you have
> mentioned were not in componetContext.xml so I missed them. It works
> today, but is not properly extensible. I will update the
> configuration and wiki today.
>
> I've haven't made progress on the commands.properties as I realized it
> will overlap with the rbac work being done. I will start a thread
> regarding that so maybe it will be address in the rbac work.
>
> Regarding log4j. First, why do you have to change that? Regardless,
> at the end of the day you are limited to whatever log4j can provide.
> So log4j AFAIK does not support multiple log4j config files so the
> best we can do is allow somebody to supply a different log4j config
> file and configure cloudstack to use that. One of the things I don't
> particularily like is that ACS forces log4j to its own configuration
> file and will not allow the regular log4j config discovery. So what I
> could do is add a JVM property like -Dcloudstack.log4jstandardconfig
> that will disable the ACS behavior of configuring log4j and then you
> can use the log4j standard approach [2].
>
> Darren
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions
> [2] http://logging.apache.org/log4j/1.2/manual.html
>
> On Thu, Oct 3, 2013 at 3:29 PM, SuichII, Christopher
> <Ch...@netapp.com> wrote:
>> Darren,
>>
>> I read through this thread and your docs on the wiki, but I'd appreciate it if you could answer a couple questions for me:
>>
>> -When creating an extension, such as a DataStoreProvider, those items are currently added into the list of providers for the appropriate bean, such as:
>> <bean id="dataStoreProviderManager"
>> class="org.apache.cloudstack.storage.datastore.provider.DataStoreProviderManagerImpl">
>> <property name="providers">
>> <list merge="true">
>> <ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
>> <ref local="cloudStackImageStoreProviderImpl"/>
>> <ref local="s3ImageStoreProviderImpl"/>
>> <ref local="swiftImageStoreProviderImpl"/>
>> <ref local="solidFireDataStoreProvider"/>
>> </list>
>> </property>
>> </bean>
>>
>> So, how do we add our bean to that list?
>>
>> -There are a number of extensions that are not currently listed, such as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are omitted from https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
>>
>> -I know somewhere in this thread you talked about the order of beans, but can you document how the ordering or precedence works in the wiki? For example, if I create a DataMotionStrategy, how do I ensure that my strategy's canHandle() method is invoked before the AncientDataStoreStrategy?
>>
>> -Is there any progress on modularizing commands.properties and the log4j configuration?
>>
>> Thanks,
>> Chris
>> --
>> Chris Suich
>> chris.suich@netapp.com<ma...@netapp.com>
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Chris,
I've updated the wiki [1]. Copying from the wiki
"Extensions are automatically discovered based on the interfaces they
implement and which module is their parent. For example, if you place
a storage extension in a child module of the network module, it will
not be discovered. Additionally, depending on the extension, the
ordering may be important in how they extension is used. For the
extensions that ordering is important there is also a "Global
Configuration" setting for the ordering. The value of the setting is a
comma seperated list like SHA256SALT,MD5,LDAP,PLAINTEXT. The values
are the names of the extension. The name will be the result of
getName(), if the bean implements the Named interface, or the sort
class name (ie getClass().getSimpleName())."
I setup all the extensible types based on what was in
componentContext.xml and also trial an error. The types you have
mentioned were not in componetContext.xml so I missed them. It works
today, but is not properly extensible. I will update the
configuration and wiki today.
I've haven't made progress on the commands.properties as I realized it
will overlap with the rbac work being done. I will start a thread
regarding that so maybe it will be address in the rbac work.
Regarding log4j. First, why do you have to change that? Regardless,
at the end of the day you are limited to whatever log4j can provide.
So log4j AFAIK does not support multiple log4j config files so the
best we can do is allow somebody to supply a different log4j config
file and configure cloudstack to use that. One of the things I don't
particularily like is that ACS forces log4j to its own configuration
file and will not allow the regular log4j config discovery. So what I
could do is add a JVM property like -Dcloudstack.log4jstandardconfig
that will disable the ACS behavior of configuring log4j and then you
can use the log4j standard approach [2].
Darren
[1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions
[2] http://logging.apache.org/log4j/1.2/manual.html
On Thu, Oct 3, 2013 at 3:29 PM, SuichII, Christopher
<Ch...@netapp.com> wrote:
> Darren,
>
> I read through this thread and your docs on the wiki, but I'd appreciate it if you could answer a couple questions for me:
>
> -When creating an extension, such as a DataStoreProvider, those items are currently added into the list of providers for the appropriate bean, such as:
> <bean id="dataStoreProviderManager"
> class="org.apache.cloudstack.storage.datastore.provider.DataStoreProviderManagerImpl">
> <property name="providers">
> <list merge="true">
> <ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
> <ref local="cloudStackImageStoreProviderImpl"/>
> <ref local="s3ImageStoreProviderImpl"/>
> <ref local="swiftImageStoreProviderImpl"/>
> <ref local="solidFireDataStoreProvider"/>
> </list>
> </property>
> </bean>
>
> So, how do we add our bean to that list?
>
> -There are a number of extensions that are not currently listed, such as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are omitted from https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
>
> -I know somewhere in this thread you talked about the order of beans, but can you document how the ordering or precedence works in the wiki? For example, if I create a DataMotionStrategy, how do I ensure that my strategy's canHandle() method is invoked before the AncientDataStoreStrategy?
>
> -Is there any progress on modularizing commands.properties and the log4j configuration?
>
> Thanks,
> Chris
> --
> Chris Suich
> chris.suich@netapp.com<ma...@netapp.com>
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
Darren,
I read through this thread and your docs on the wiki, but I'd appreciate it if you could answer a couple questions for me:
-When creating an extension, such as a DataStoreProvider, those items are currently added into the list of providers for the appropriate bean, such as:
<bean id="dataStoreProviderManager"
class="org.apache.cloudstack.storage.datastore.provider.DataStoreProviderManagerImpl">
<property name="providers">
<list merge="true">
<ref bean="cloudStackPrimaryDataStoreProviderImpl"/>
<ref local="cloudStackImageStoreProviderImpl"/>
<ref local="s3ImageStoreProviderImpl"/>
<ref local="swiftImageStoreProviderImpl"/>
<ref local="solidFireDataStoreProvider"/>
</list>
</property>
</bean>
So, how do we add our bean to that list?
-There are a number of extensions that are not currently listed, such as DataMotionStrategy, SnapshotStrategy, etc. Is it a problem that those are omitted from https://cwiki.apache.org/confluence/display/CLOUDSTACK/Extensions?
-I know somewhere in this thread you talked about the order of beans, but can you document how the ordering or precedence works in the wiki? For example, if I create a DataMotionStrategy, how do I ensure that my strategy's canHandle() method is invoked before the AncientDataStoreStrategy?
-Is there any progress on modularizing commands.properties and the log4j configuration?
Thanks,
Chris
--
Chris Suich
chris.suich@netapp.com<ma...@netapp.com>
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Sep 24, 2013, at 2:35 AM, Daan Hoogland <da...@gmail.com>> wrote:
touching on thread hijack but; How does this work relate to the css
modularization going on at the moment as well? It is proposed there to
do merging at build time. Try to beat me if I am to much off topic,
Daan
On Tue, Sep 24, 2013 at 12:43 AM, Darren Shepherd
<da...@gmail.com>> wrote:
Ah right, okay. So your talking about the order of the adapters.
Currently that is maintained as the order in the AdapterList in the
componentContext.xml. So what I've done is that extensions get added
to "Registries." Registries that need to be ordered can specify an
ordering configuration variable so that when the extensions are found
they are added to the list in a specific order. So the registry
definition for the auth stuff looks something like
<bean id="userAuthenticatorsRegistry"
class="org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry">
<property name="orderConfigKey" value="user.authenticators.order" />
<property name="excludeKey" value="user.authenticators.exclude" />
<property name="orderConfigDefault"
value="SHA256SALT,MD5,LDAP,PLAINTEXT" />
</bean>
So one can use user.authenticators.order to change the order and
user.authenticators.exclude to exclude certain extensions from being
used. The default value is also specified in that example.
Darren
On Mon, Sep 23, 2013 at 3:28 PM, Kelven Yang <ke...@citrix.com>> wrote:
I understand that loading order can be completely solved by Spring with
dependency injection, either within flat context or hierarchy context
structure. But some sibling plugins in CloudStack do require a flexible
way to declare ordering for runtime. For example, allocator plugins or
authenticator plugins. The order itself is currently designated in their
parent manager to reference to a ordered adapter list.
This order semantics has nothing to do with dependency injection, but
unfortunately in previous version of CloudStack, it does mix the
requirement into injection framework and there are business logic relying
on it. The fact for parent manager to compose ordering has made an
assumption at compile/load time binding to subject plugins, which we don't
want to see in the future since we want drop-in jar for plugins, what is
our answer for this?
Kelven
On 9/23/13 1:40 PM, "Darren Shepherd" <da...@gmail.com>> wrote:
Siblings have no relationship to each other really. The load order
doesn't matter as one sibling has no visibility to another. Child
contexts are pretty much for plug-ins. Core components of ACS will
live in the core context as they are all interdependent.
I will explain the whole Spring lifecycle though and how it works in the
model.
Contexts are initialized from parent to child. So the top most parent
context is initialized first, then its child, then its grandchildren.
When a context is initialized the following happens in the below order
1) All beans are instantiated
2) Dependencies are wire up (@Inject)
3) @PostConstruct is call for all beans in dependency graph order
4) Extensions are discovered an registered (so NetworkElement for
example will be discovered an registered as a NetworkElement)
5) configure() on all ComponentLifecycle beans will be called in the
getRunLevel() order
Once all modules have been initialized in this fashion then in a
parent first child last order start() on all ComponentLifecycle beans
is called following the getRunLevel() order for the beans in the
context.
Darren
On Mon, Sep 23, 2013 at 11:34 AM, Kelven Yang <ke...@citrix.com>>
wrote:
Darren,
Due to internal release work, I haven't got a chance to review it but
I'm
planning to do so later today and tomorrow. Before that, I have one
question about hierarchy-orginzated context structure, could you
elaborate
an example to the ML on how two sibling plugins to declare their runtime
load order? I'd like to get a feeling on how hard or easy for developers
to do things that involve with structural change under the new hierarchy
mode.
Kelven
On 9/23/13 12:19 AM, "Darren Shepherd" <da...@gmail.com>>
wrote:
So how do I proceed forward on this? I basically already have this all
working. I'd really like to get it all committed as soon as possible if
there are no objections to the approach. The sooner the better.
I already have a bunch of patches pending on review board that change a
bunch of random but related things. I need all of those patches
committed before I can submit the next round of patches. I have about 4
or 5 more. What I'll do is that everything will get committed and then
their will be one small patch that will be last that flips some config
files to enable this all. All changes to code will work in both a
modular and monolithic spring context. So it will be really easy to
turn
this off if suddenly something goes terribly wrong.
So I need people to agree this is good and then start
reviewing/committing my patches. I really want to get this wrapped up
this week if I can.
Darren
On Sep 18, 2013, at 7:06 PM, Darren Shepherd
<da...@gmail.com>> wrote:
Yes this is one of the many things this is trying to address. Adding
a
plugin should be plopping your jar in a directory and restarting. You
pointed out two things I didn't think about though, command.properties
and log4j xml. Let me think about those twos as they should be address
also. Basically you should never have to edit a file that is packaged
as part of ACS. Only add your artifacts to some directory, ideally
just
a jar.
Darren
On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher
<Ch...@netapp.com>> wrote:
I've been following this conversation somewhat and would like to
throw
in some background as a plugin writer.
One thing that concerns me in the current plugin model is the number
of XML/text files that need to be edited to deploy my plugin.
-applicationContext must be edited to add our PluginMangerImpl.
-commands.properties file must be edited to included the permissions
for the APIs we contributed.
-componentContext.xml & nonossComponentContext.xml must be edited to
add our Storage Subsystem Provider API.
-log4j-cloud.xml must be edited to ensure that our logger is enabled
and logging to our necessary default level.
I know our situation is a bit different than the current plug in
model, but I think it is something we, as a community, are going to
begin seeing more of. For a variety of reasons that I won't get in to
right now, our plugin will be closed source and kept separate from the
ACS source tree. We want our users to be able to simply drop in our
jar
file to the CS directory or run and installer and have it picked up by
the MS upon a restart.
It sounds like what you are proposing here would be very beneficial
to
plugins that are targeting a deployment model like this.
Is this something you're looking/hoping/would like to solve, Darren?
-Chris
--
Chris Suich
chris.suich@netapp.com<ma...@netapp.com>
NetApp Software Engineer
Data Center Platforms Cloud Solutions
Citrix, Cisco & Red Hat
On Sep 18, 2013, at 6:44 PM, Darren Shepherd
<da...@gmail.com> wrote:
I'm not a committer
On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang
<Fr...@citrix.com> wrote:
Well. The codes explain more than words.
It seems the only extra work is adding a property file that
specifies
parent context and current context name, it's not much complex.
BTW: any reason for working on repo outside ACS?
-----Original Message-----
From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
Sent: Wednesday, September 18, 2013 2:43 PM
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] Modularize Spring
If you want to see this all working you can just fetch the
"no-at-db4"
branch at https://github.com/ibuildthecloud/cloudstack.git
Plugin composes multiple modules. If modules are siblings they
can't
inject
from each other. But a plugin can augment another module if it
chooses
to.
The reality is that the core cloudstack is a tangled mess of
dependencies such
that most of the core code can't be modularized as it stands. So
there
exists a
context towards the top of the hierarchy called "core" that a lot
of jars
contribute to it. Here is the full hierarchy right now. I'll
probably
rename a
bunch of things, but this gives you an idea.
bootstrap
system
core
allocator
allocator-server
planner
api-planner
baremetal-planner
explicit-dedication
host-anti-affinity
implicit-dedication-planner
server-planner
user-concentrated-pod-planner
random-allocator
api
acl-static-role-based
rate-limit
server-api
user-authenticator-ldap
user-authenticator-md5
user-authenticator-plaintext
user-authenticator-sha256salted
backend
alert-adapter-server-backend
compute
alert-adapter-server-compute
baremetal-compute
fencer-server
investigator-server
kvm-compute
ovm-compute
server-compute
xenserver-compute
network
baremetal-network
elb
midonet
nvp
ovs
server-network
ssp
storage
alert-adapter-server-storage
allocator-storage
baremetal-storage
secondary-storage
server-storage
storage-image-default
storage-image-s3
storage-image-swift
storage-volume-default
storage-volume-solidfire
template-adapter-server
discoverer
baremetal-discoverer
discoverer-server
ovm-discoverer
xenserver-discoverer
If you look at the baremetal hypervisor plugin that is pretty
cross
cutting to
most of ACS. So the modules it contributes to are as follows
resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
resources/META-INF/cloudstack/baremetal-network/spring-context.xml
So it creates child contextes of storage, compute, network,
planner, and
discoverer to add its extensions where it needs to be.
Additionally,
you'll notice,
it adds some beans to the core context from the file
resources/META-
INF/cloudstack/core/spring-baremetal-core-context.xml. This is
because
it has
some manager class that is used by multiple contexts.
Frank, I understand the scare that we are going too complex, but
do you
have
some other suggestion? I don't like the idea of one gigantic
spring
context. So I
feel I am making it as simple as I can while maintaining some
order.
People
just need to create one or more spring xml files and a properties
files
that says
where to put it in the hierarchy.
Additionally, by putting forcing people to put beans in certains
modules
it
forces them to think about what is the role of the code. For
example,
today in
ACS the *ManagerImpl classes are a huge mess. They implement too
many
interfaces and the code crosses to many architectural boundaries.
Its
about
time we start splitting things up to be more maintainable.
If you have some time, please check out what I have on github.
Darren
On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang
<Fr...@citrix.com>
wrote:
I am not against boundary, I am just against making things
unnecessary
complex to enable boundary.
If we are going this way, I hope we can make it as much as
transparent
to developers. That means, as a developer, all a plugin I need
to
do
is 1) provide my separate spring xml 2) inject beans I want
(valid
beans) in my bean and code business logic 3) compile to a jar
and
put
to some place designated by CloudStack. That's it.
I raise this topic because I have seen some projects to create
boundary making things horrible complex. And sometimes
developers
are
hurt by wrong boundaries, as a result, to overcome these
limitations
people write lots of ugly code which makes thing even worse.
However, I am still worry about if we can make things so
simpler.
For example, we may have an orchestration context that contains
major
beans needed by almost every plugin, this context can be easily
set
as parent context for all plugin contexts when bootstrap.
However, if
a plugin A needs to access some bean defined in plugin B, given
they
are sibling, how plugin framework resolves the dependency ?
-----Original Message-----
From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
Sent: Wednesday, September 18, 2013 11:53 AM
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] Modularize Spring
I'm not for OSGi either, but contexts are quite useful and will
lead
to
better
things. First off, we don't want one gigantic spring XML
config
file
like we have
today. I think we can all agree on that. So each plugin will
have
to
supply its
own XML. So the obstacles you mention, will largely be just
that
for
people.
With Spring it is really simple to just inject dependencies and
cross architectural boundaries. Its currently everywhere in
ACS.
You can't
just say
we should review code and make sure nobody doesn't do bad
things. A
little bit
of framework to enforce separation is a good thing. But I'm
guessing
you will
disagree with me there.
Here are some random points on why contexts are good. Say I
want to
use Spring AOP or Spring TX in my plugin. With your own
context
you
can
ensure
that you won't screw with anybody else code by accidentally
having
you pointcut match their bean. You don't have to worry about
bean
name
conflicts.
If two config files specify bean X, Spring will gladly just use
the
last
one. I've
already found multiply defined beans in ACS, but things still
just
happen to work.
Having multiple contexts also defines initialization order. We
can
ensure that
the framework is loaded and ready before child context are
loaded
and
started.
(we kind of do this today with ComponentLifeCycle, but its a
hack in
my
mind).
Additionally, when things start you will know, loading context
"crapping
plug X".
If spring fails to initialize, the issue it there. Today, if
spring
fails to start, it
could be one of over 500 beans causing the weird problem. The
list
goes
on
and on.
Finally, this is the big one and why I really want contexts. I
have
some notes on
the wiki [1] that you might want to read through. Basically I
want
to
get to a
more flexible deployment model that allows both a single
monolithic
JVM
as
today and also a fully distributed system. Having contexts in
hierarchy
will
enable me to accomplish this. By selecting which contexts are
loaded at runtime will determine what role the JVM takes on.
The
contexts help
people
better understand how the distributed architecture will work
too,
when
we get
there.
Frank, trust me, I hate complex things. I don't want OSGi,
classloader
magic,
etc. But I do like organization and a little bit of framework
so
that
people don't
accidentally shoot themselves in the foot. I personally like
knowing
that I
couldn't have screwed something up, because the framework won't
even
allow
it. If we separate everything as I want today, and then
tomorrow we
say
this is
way too much overhead, moving to a flat context is simple.
Don't
think
we are
on some slippery slope to classloaders and dependency hell.
Darren
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
her
e...#Nothingtoseehere...-DeploymentModels
On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
<Fr...@citrix.com>wrote:
What's the point in using separate spring context per plugin?
Separate class loader is the thing I hate most in OSGI, I am
afraid we are on the same way.
Frankly speaking, I never see benefits of this *separate*
model,
our project(or most projects) is not like Chrome which has to
create sandbox for extensions in order to avoid bad plugin
screws
up the whole browser(however, I still see bad plugins screw up
my
Chrome well).
Projects like CloudStack using plugin to decouple architecture
should not introduce many isolations to plugin writer, the
point
preventing wrong use of some components is not making much
sense
to me. If a plugin not following guide(if we have
it) we should kick it out, instead of making obstacles for 99%
good
people.
-----Original Message-----
From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
Sent: Wednesday, September 18, 2013 10:33 AM
To: dev@cloudstack.apache.org
Subject: Re: [PROPOSAL] Modularize Spring
Right, component isn't a thing. I probably shouldn't use
that
term.
I
want to
standarize on the naming convention of plugin, module, and
extension.
It is
explained a bit on the wiki [1] but I'll try to do a little
better job
here. So a
plugin is basically a jar. A jar contains multiple modules.
A
modules
ends up
being a spring application context that composes multiple
configuration
files.
Modules are assembled into a hierarchy at runtime.
Extensions
are implementations of interfaces that exist in a module. A
maven project produces a jar, so a plugin ends up being a
maven
project
also.
So currently today we don't have a strong definition of
Plugin
and I
hope to
address that.
Darren
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
ins%2C+Modules%2C+and+Extensions
On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
<da...@gmail.com>wrote:
sounds great Darren,
By component, you mean maven project or some larger chunk
like
distribution package? (did i miss this definition somewhere
or
do we define the components now?)
regards,
Daan
On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
<da...@gmail.com> wrote:
Currently ACS code is fairly modular in that you can add
plug-ins to ACS
to
extend most functionality. Unfortunately ACS is not
packaged in a
modular
way. It is still delivered essentially as one large unit.
There are
many
reason for this but one large barrier to modularizing ACS
is
that the Spring configuration is managed as one large unit.
I propose that we break apart the Spring XML configuration
such that each component contributes its own configuration.
Additionally each component will be loaded into its own
Spring ApplicationContext such that its beans will not
conflict with the wiring of other beans in ACS. This
change
will
lay the foundation for a richer plugin model and
additionally a more distributed architecture.
The technical details for this proposal can be found on the
wiki
[1].
Darren
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
ize+
Spri
ng
Re: [PROPOSAL] Modularize Spring
Posted by Daan Hoogland <da...@gmail.com>.
touching on thread hijack but; How does this work relate to the css
modularization going on at the moment as well? It is proposed there to
do merging at build time. Try to beat me if I am to much off topic,
Daan
On Tue, Sep 24, 2013 at 12:43 AM, Darren Shepherd
<da...@gmail.com> wrote:
> Ah right, okay. So your talking about the order of the adapters.
> Currently that is maintained as the order in the AdapterList in the
> componentContext.xml. So what I've done is that extensions get added
> to "Registries." Registries that need to be ordered can specify an
> ordering configuration variable so that when the extensions are found
> they are added to the list in a specific order. So the registry
> definition for the auth stuff looks something like
>
> <bean id="userAuthenticatorsRegistry"
> class="org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry">
> <property name="orderConfigKey" value="user.authenticators.order" />
> <property name="excludeKey" value="user.authenticators.exclude" />
> <property name="orderConfigDefault"
> value="SHA256SALT,MD5,LDAP,PLAINTEXT" />
> </bean>
>
> So one can use user.authenticators.order to change the order and
> user.authenticators.exclude to exclude certain extensions from being
> used. The default value is also specified in that example.
>
> Darren
>
> On Mon, Sep 23, 2013 at 3:28 PM, Kelven Yang <ke...@citrix.com> wrote:
>> I understand that loading order can be completely solved by Spring with
>> dependency injection, either within flat context or hierarchy context
>> structure. But some sibling plugins in CloudStack do require a flexible
>> way to declare ordering for runtime. For example, allocator plugins or
>> authenticator plugins. The order itself is currently designated in their
>> parent manager to reference to a ordered adapter list.
>>
>> This order semantics has nothing to do with dependency injection, but
>> unfortunately in previous version of CloudStack, it does mix the
>> requirement into injection framework and there are business logic relying
>> on it. The fact for parent manager to compose ordering has made an
>> assumption at compile/load time binding to subject plugins, which we don't
>> want to see in the future since we want drop-in jar for plugins, what is
>> our answer for this?
>>
>> Kelven
>>
>>
>> On 9/23/13 1:40 PM, "Darren Shepherd" <da...@gmail.com> wrote:
>>
>>>Siblings have no relationship to each other really. The load order
>>>doesn't matter as one sibling has no visibility to another. Child
>>>contexts are pretty much for plug-ins. Core components of ACS will
>>>live in the core context as they are all interdependent.
>>>
>>>I will explain the whole Spring lifecycle though and how it works in the
>>>model.
>>>
>>>Contexts are initialized from parent to child. So the top most parent
>>>context is initialized first, then its child, then its grandchildren.
>>>When a context is initialized the following happens in the below order
>>>
>>>1) All beans are instantiated
>>>2) Dependencies are wire up (@Inject)
>>>3) @PostConstruct is call for all beans in dependency graph order
>>>4) Extensions are discovered an registered (so NetworkElement for
>>>example will be discovered an registered as a NetworkElement)
>>>5) configure() on all ComponentLifecycle beans will be called in the
>>>getRunLevel() order
>>>
>>>Once all modules have been initialized in this fashion then in a
>>>parent first child last order start() on all ComponentLifecycle beans
>>>is called following the getRunLevel() order for the beans in the
>>>context.
>>>
>>>Darren
>>>
>>>
>>>On Mon, Sep 23, 2013 at 11:34 AM, Kelven Yang <ke...@citrix.com>
>>>wrote:
>>>> Darren,
>>>>
>>>> Due to internal release work, I haven't got a chance to review it but
>>>>I'm
>>>> planning to do so later today and tomorrow. Before that, I have one
>>>> question about hierarchy-orginzated context structure, could you
>>>>elaborate
>>>> an example to the ML on how two sibling plugins to declare their runtime
>>>> load order? I'd like to get a feeling on how hard or easy for developers
>>>> to do things that involve with structural change under the new hierarchy
>>>> mode.
>>>>
>>>> Kelven
>>>>
>>>>
>>>> On 9/23/13 12:19 AM, "Darren Shepherd" <da...@gmail.com>
>>>>wrote:
>>>>
>>>>>So how do I proceed forward on this? I basically already have this all
>>>>>working. I'd really like to get it all committed as soon as possible if
>>>>>there are no objections to the approach. The sooner the better.
>>>>>
>>>>>I already have a bunch of patches pending on review board that change a
>>>>>bunch of random but related things. I need all of those patches
>>>>>committed before I can submit the next round of patches. I have about 4
>>>>>or 5 more. What I'll do is that everything will get committed and then
>>>>>their will be one small patch that will be last that flips some config
>>>>>files to enable this all. All changes to code will work in both a
>>>>>modular and monolithic spring context. So it will be really easy to
>>>>>turn
>>>>>this off if suddenly something goes terribly wrong.
>>>>>
>>>>>So I need people to agree this is good and then start
>>>>>reviewing/committing my patches. I really want to get this wrapped up
>>>>>this week if I can.
>>>>>
>>>>>Darren
>>>>>
>>>>>> On Sep 18, 2013, at 7:06 PM, Darren Shepherd
>>>>>><da...@gmail.com> wrote:
>>>>>>
>>>>>> Yes this is one of the many things this is trying to address. Adding
>>>>>>a
>>>>>>plugin should be plopping your jar in a directory and restarting. You
>>>>>>pointed out two things I didn't think about though, command.properties
>>>>>>and log4j xml. Let me think about those twos as they should be address
>>>>>>also. Basically you should never have to edit a file that is packaged
>>>>>>as part of ACS. Only add your artifacts to some directory, ideally
>>>>>>just
>>>>>>a jar.
>>>>>>
>>>>>> Darren
>>>>>>
>>>>>>
>>>>>>> On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher
>>>>>>><Ch...@netapp.com> wrote:
>>>>>>> I've been following this conversation somewhat and would like to
>>>>>>>throw
>>>>>>>in some background as a plugin writer.
>>>>>>>
>>>>>>> One thing that concerns me in the current plugin model is the number
>>>>>>>of XML/text files that need to be edited to deploy my plugin.
>>>>>>> -applicationContext must be edited to add our PluginMangerImpl.
>>>>>>> -commands.properties file must be edited to included the permissions
>>>>>>>for the APIs we contributed.
>>>>>>> -componentContext.xml & nonossComponentContext.xml must be edited to
>>>>>>>add our Storage Subsystem Provider API.
>>>>>>> -log4j-cloud.xml must be edited to ensure that our logger is enabled
>>>>>>>and logging to our necessary default level.
>>>>>>>
>>>>>>> I know our situation is a bit different than the current plug in
>>>>>>>model, but I think it is something we, as a community, are going to
>>>>>>>begin seeing more of. For a variety of reasons that I won't get in to
>>>>>>>right now, our plugin will be closed source and kept separate from the
>>>>>>>ACS source tree. We want our users to be able to simply drop in our
>>>>>>>jar
>>>>>>>file to the CS directory or run and installer and have it picked up by
>>>>>>>the MS upon a restart.
>>>>>>>
>>>>>>> It sounds like what you are proposing here would be very beneficial
>>>>>>>to
>>>>>>>plugins that are targeting a deployment model like this.
>>>>>>>
>>>>>>> Is this something you're looking/hoping/would like to solve, Darren?
>>>>>>>
>>>>>>> -Chris
>>>>>>> --
>>>>>>> Chris Suich
>>>>>>> chris.suich@netapp.com
>>>>>>> NetApp Software Engineer
>>>>>>> Data Center Platforms Cloud Solutions
>>>>>>> Citrix, Cisco & Red Hat
>>>>>>>
>>>>>>> On Sep 18, 2013, at 6:44 PM, Darren Shepherd
>>>>>>><da...@gmail.com> wrote:
>>>>>>>
>>>>>>> > I'm not a committer
>>>>>>> >
>>>>>>> >
>>>>>>> > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang
>>>>>>><Fr...@citrix.com> wrote:
>>>>>>> >
>>>>>>> >> Well. The codes explain more than words.
>>>>>>> >> It seems the only extra work is adding a property file that
>>>>>>>specifies
>>>>>>> >> parent context and current context name, it's not much complex.
>>>>>>> >> BTW: any reason for working on repo outside ACS?
>>>>>>> >>
>>>>>>> >>> -----Original Message-----
>>>>>>> >>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>>> >>> Sent: Wednesday, September 18, 2013 2:43 PM
>>>>>>> >>> To: dev@cloudstack.apache.org
>>>>>>> >>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>>> >>>
>>>>>>> >>> If you want to see this all working you can just fetch the
>>>>>>>"no-at-db4"
>>>>>>> >>> branch at https://github.com/ibuildthecloud/cloudstack.git
>>>>>>> >>>
>>>>>>> >>> Plugin composes multiple modules. If modules are siblings they
>>>>>>>can't
>>>>>>> >> inject
>>>>>>> >>> from each other. But a plugin can augment another module if it
>>>>>>>chooses
>>>>>>> >> to.
>>>>>>> >>> The reality is that the core cloudstack is a tangled mess of
>>>>>>> >> dependencies such
>>>>>>> >>> that most of the core code can't be modularized as it stands. So
>>>>>>>there
>>>>>>> >> exists a
>>>>>>> >>> context towards the top of the hierarchy called "core" that a lot
>>>>>>>of jars
>>>>>>> >>> contribute to it. Here is the full hierarchy right now. I'll
>>>>>>>probably
>>>>>>> >> rename a
>>>>>>> >>> bunch of things, but this gives you an idea.
>>>>>>> >>>
>>>>>>> >>> bootstrap
>>>>>>> >>> system
>>>>>>> >>> core
>>>>>>> >>> allocator
>>>>>>> >>> allocator-server
>>>>>>> >>> planner
>>>>>>> >>> api-planner
>>>>>>> >>> baremetal-planner
>>>>>>> >>> explicit-dedication
>>>>>>> >>> host-anti-affinity
>>>>>>> >>> implicit-dedication-planner
>>>>>>> >>> server-planner
>>>>>>> >>> user-concentrated-pod-planner
>>>>>>> >>> random-allocator
>>>>>>> >>> api
>>>>>>> >>> acl-static-role-based
>>>>>>> >>> rate-limit
>>>>>>> >>> server-api
>>>>>>> >>> user-authenticator-ldap
>>>>>>> >>> user-authenticator-md5
>>>>>>> >>> user-authenticator-plaintext
>>>>>>> >>> user-authenticator-sha256salted
>>>>>>> >>> backend
>>>>>>> >>> alert-adapter-server-backend
>>>>>>> >>> compute
>>>>>>> >>> alert-adapter-server-compute
>>>>>>> >>> baremetal-compute
>>>>>>> >>> fencer-server
>>>>>>> >>> investigator-server
>>>>>>> >>> kvm-compute
>>>>>>> >>> ovm-compute
>>>>>>> >>> server-compute
>>>>>>> >>> xenserver-compute
>>>>>>> >>> network
>>>>>>> >>> baremetal-network
>>>>>>> >>> elb
>>>>>>> >>> midonet
>>>>>>> >>> nvp
>>>>>>> >>> ovs
>>>>>>> >>> server-network
>>>>>>> >>> ssp
>>>>>>> >>> storage
>>>>>>> >>> alert-adapter-server-storage
>>>>>>> >>> allocator-storage
>>>>>>> >>> baremetal-storage
>>>>>>> >>> secondary-storage
>>>>>>> >>> server-storage
>>>>>>> >>> storage-image-default
>>>>>>> >>> storage-image-s3
>>>>>>> >>> storage-image-swift
>>>>>>> >>> storage-volume-default
>>>>>>> >>> storage-volume-solidfire
>>>>>>> >>> template-adapter-server
>>>>>>> >>> discoverer
>>>>>>> >>> baremetal-discoverer
>>>>>>> >>> discoverer-server
>>>>>>> >>> ovm-discoverer
>>>>>>> >>> xenserver-discoverer
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> If you look at the baremetal hypervisor plugin that is pretty
>>>>>>>cross
>>>>>>> >> cutting to
>>>>>>> >>> most of ACS. So the modules it contributes to are as follows
>>>>>>> >>>
>>>>>>> >>>
>>>>>>>resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>>>>>>> >>>
>>>>>>>resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>>>>>>> >>>
>>>>>>>resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>>>>>>> >>>
>>>>>>>resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>>>>>>> >>>
>>>>>>>resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>>>>>>> >>>
>>>>>>>resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>>>>>>> >>>
>>>>>>> >>> So it creates child contextes of storage, compute, network,
>>>>>>>planner, and
>>>>>>> >>> discoverer to add its extensions where it needs to be.
>>>>>>>Additionally,
>>>>>>> >> you'll notice,
>>>>>>> >>> it adds some beans to the core context from the file
>>>>>>>resources/META-
>>>>>>> >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is
>>>>>>>because
>>>>>>> >> it has
>>>>>>> >>> some manager class that is used by multiple contexts.
>>>>>>> >>>
>>>>>>> >>> Frank, I understand the scare that we are going too complex, but
>>>>>>>do you
>>>>>>> >> have
>>>>>>> >>> some other suggestion? I don't like the idea of one gigantic
>>>>>>>spring
>>>>>>> >> context. So I
>>>>>>> >>> feel I am making it as simple as I can while maintaining some
>>>>>>>order.
>>>>>>> >> People
>>>>>>> >>> just need to create one or more spring xml files and a properties
>>>>>>>files
>>>>>>> >> that says
>>>>>>> >>> where to put it in the hierarchy.
>>>>>>> >>>
>>>>>>> >>> Additionally, by putting forcing people to put beans in certains
>>>>>>>modules
>>>>>>> >> it
>>>>>>> >>> forces them to think about what is the role of the code. For
>>>>>>>example,
>>>>>>> >> today in
>>>>>>> >>> ACS the *ManagerImpl classes are a huge mess. They implement too
>>>>>>>many
>>>>>>> >>> interfaces and the code crosses to many architectural boundaries.
>>>>>>>Its
>>>>>>> >> about
>>>>>>> >>> time we start splitting things up to be more maintainable.
>>>>>>> >>>
>>>>>>> >>> If you have some time, please check out what I have on github.
>>>>>>> >>>
>>>>>>> >>> Darren
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang
>>>>>>><Fr...@citrix.com>
>>>>>>> >>> wrote:
>>>>>>> >>>
>>>>>>> >>>> I am not against boundary, I am just against making things
>>>>>>>unnecessary
>>>>>>> >>>> complex to enable boundary.
>>>>>>> >>>> If we are going this way, I hope we can make it as much as
>>>>>>>transparent
>>>>>>> >>>> to developers. That means, as a developer, all a plugin I need
>>>>>>>to
>>>>>>>do
>>>>>>> >>>> is 1) provide my separate spring xml 2) inject beans I want
>>>>>>>(valid
>>>>>>> >>>> beans) in my bean and code business logic 3) compile to a jar
>>>>>>>and
>>>>>>>put
>>>>>>> >>>> to some place designated by CloudStack. That's it.
>>>>>>> >>>>
>>>>>>> >>>> I raise this topic because I have seen some projects to create
>>>>>>> >>>> boundary making things horrible complex. And sometimes
>>>>>>>developers
>>>>>>>are
>>>>>>> >>>> hurt by wrong boundaries, as a result, to overcome these
>>>>>>>limitations
>>>>>>> >>>> people write lots of ugly code which makes thing even worse.
>>>>>>> >>>>
>>>>>>> >>>> However, I am still worry about if we can make things so
>>>>>>>simpler.
>>>>>>> >>>> For example, we may have an orchestration context that contains
>>>>>>>major
>>>>>>> >>>> beans needed by almost every plugin, this context can be easily
>>>>>>>set
>>>>>>> >>>> as parent context for all plugin contexts when bootstrap.
>>>>>>>However, if
>>>>>>> >>>> a plugin A needs to access some bean defined in plugin B, given
>>>>>>>they
>>>>>>> >>>> are sibling, how plugin framework resolves the dependency ?
>>>>>>> >>>>
>>>>>>> >>>>> -----Original Message-----
>>>>>>> >>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>>> >>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>>>>>>> >>>>> To: dev@cloudstack.apache.org
>>>>>>> >>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>>> >>>>>
>>>>>>> >>>>> I'm not for OSGi either, but contexts are quite useful and will
>>>>>>>lead
>>>>>>> >>>>> to
>>>>>>> >>>> better
>>>>>>> >>>>> things. First off, we don't want one gigantic spring XML
>>>>>>>config
>>>>>>> >>>>> file
>>>>>>> >>>> like we have
>>>>>>> >>>>> today. I think we can all agree on that. So each plugin will
>>>>>>>have
>>>>>>> >>>>> to
>>>>>>> >>>> supply its
>>>>>>> >>>>> own XML. So the obstacles you mention, will largely be just
>>>>>>>that
>>>>>>> >>>>> for
>>>>>>> >>>> people.
>>>>>>> >>>>>
>>>>>>> >>>>> With Spring it is really simple to just inject dependencies and
>>>>>>> >>>>> cross architectural boundaries. Its currently everywhere in
>>>>>>>ACS.
>>>>>>> >>>>> You can't
>>>>>>> >>>> just say
>>>>>>> >>>>> we should review code and make sure nobody doesn't do bad
>>>>>>>things. A
>>>>>>> >>>> little bit
>>>>>>> >>>>> of framework to enforce separation is a good thing. But I'm
>>>>>>> >>>>> guessing
>>>>>>> >>>> you will
>>>>>>> >>>>> disagree with me there.
>>>>>>> >>>>>
>>>>>>> >>>>> Here are some random points on why contexts are good. Say I
>>>>>>>want to
>>>>>>> >>>>> use Spring AOP or Spring TX in my plugin. With your own
>>>>>>>context
>>>>>>>you
>>>>>>> >>>>> can
>>>>>>> >>>> ensure
>>>>>>> >>>>> that you won't screw with anybody else code by accidentally
>>>>>>>having
>>>>>>> >>>>> you pointcut match their bean. You don't have to worry about
>>>>>>>bean
>>>>>>> >>>>> name
>>>>>>> >>>> conflicts.
>>>>>>> >>>>> If two config files specify bean X, Spring will gladly just use
>>>>>>>the
>>>>>>> >>>>> last
>>>>>>> >>>> one. I've
>>>>>>> >>>>> already found multiply defined beans in ACS, but things still
>>>>>>>just
>>>>>>> >>>> happen to work.
>>>>>>> >>>>> Having multiple contexts also defines initialization order. We
>>>>>>>can
>>>>>>> >>>> ensure that
>>>>>>> >>>>> the framework is loaded and ready before child context are
>>>>>>>loaded
>>>>>>> >>>>> and
>>>>>>> >>>> started.
>>>>>>> >>>>> (we kind of do this today with ComponentLifeCycle, but its a
>>>>>>>hack in
>>>>>>> >>>>> my
>>>>>>> >>>> mind).
>>>>>>> >>>>> Additionally, when things start you will know, loading context
>>>>>>> >>>>> "crapping
>>>>>>> >>>> plug X".
>>>>>>> >>>>> If spring fails to initialize, the issue it there. Today, if
>>>>>>>spring
>>>>>>> >>>> fails to start, it
>>>>>>> >>>>> could be one of over 500 beans causing the weird problem. The
>>>>>>>list
>>>>>>> >>>>> goes
>>>>>>> >>>> on
>>>>>>> >>>>> and on.
>>>>>>> >>>>>
>>>>>>> >>>>> Finally, this is the big one and why I really want contexts. I
>>>>>>>have
>>>>>>> >>>> some notes on
>>>>>>> >>>>> the wiki [1] that you might want to read through. Basically I
>>>>>>>want
>>>>>>> >>>>> to
>>>>>>> >>>> get to a
>>>>>>> >>>>> more flexible deployment model that allows both a single
>>>>>>>monolithic
>>>>>>> >>>>> JVM
>>>>>>> >>>> as
>>>>>>> >>>>> today and also a fully distributed system. Having contexts in
>>>>>>> >>>>> hierarchy
>>>>>>> >>>> will
>>>>>>> >>>>> enable me to accomplish this. By selecting which contexts are
>>>>>>> >>>>> loaded at runtime will determine what role the JVM takes on.
>>>>>>>The
>>>>>>> >>>>> contexts help
>>>>>>> >>>> people
>>>>>>> >>>>> better understand how the distributed architecture will work
>>>>>>>too,
>>>>>>> >>>>> when
>>>>>>> >>>> we get
>>>>>>> >>>>> there.
>>>>>>> >>>>>
>>>>>>> >>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>>>>>>> >>>>> classloader
>>>>>>> >>>> magic,
>>>>>>> >>>>> etc. But I do like organization and a little bit of framework
>>>>>>>so
>>>>>>> >>>>> that
>>>>>>> >>>> people don't
>>>>>>> >>>>> accidentally shoot themselves in the foot. I personally like
>>>>>>> >>>>> knowing
>>>>>>> >>>> that I
>>>>>>> >>>>> couldn't have screwed something up, because the framework won't
>>>>>>>even
>>>>>>> >>>> allow
>>>>>>> >>>>> it. If we separate everything as I want today, and then
>>>>>>>tomorrow we
>>>>>>> >>>>> say
>>>>>>> >>>> this is
>>>>>>> >>>>> way too much overhead, moving to a flat context is simple.
>>>>>>>Don't
>>>>>>> >>>>> think
>>>>>>> >>>> we are
>>>>>>> >>>>> on some slippery slope to classloaders and dependency hell.
>>>>>>> >>>>>
>>>>>>> >>>>> Darren
>>>>>>> >>>>>
>>>>>>> >>>>> [1]
>>>>>>> >>>>>
>>>>>>> >>>>
>>>>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>>>>>>> >>>> her
>>>>>>> >>>>> e...#Nothingtoseehere...-DeploymentModels
>>>>>>> >>>>>
>>>>>>> >>>>>
>>>>>>> >>>>>
>>>>>>> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>>>>>>> >>>>> <Fr...@citrix.com>wrote:
>>>>>>> >>>>>
>>>>>>> >>>>>> What's the point in using separate spring context per plugin?
>>>>>>> >>>>>> Separate class loader is the thing I hate most in OSGI, I am
>>>>>>> >>>>>> afraid we are on the same way.
>>>>>>> >>>>>> Frankly speaking, I never see benefits of this *separate*
>>>>>>>model,
>>>>>>> >>>>>> our project(or most projects) is not like Chrome which has to
>>>>>>> >>>>>> create sandbox for extensions in order to avoid bad plugin
>>>>>>>screws
>>>>>>> >>>>>> up the whole browser(however, I still see bad plugins screw up
>>>>>>>my
>>>>>>> >>>>>> Chrome well).
>>>>>>> >>>>>> Projects like CloudStack using plugin to decouple architecture
>>>>>>> >>>>>> should not introduce many isolations to plugin writer, the
>>>>>>>point
>>>>>>> >>>>>> preventing wrong use of some components is not making much
>>>>>>>sense
>>>>>>> >>>>>> to me. If a plugin not following guide(if we have
>>>>>>> >>>>>> it) we should kick it out, instead of making obstacles for 99%
>>>>>>> >>>>>> good
>>>>>>> >>>> people.
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>>>> -----Original Message-----
>>>>>>> >>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>>> >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>>>>>>> >>>>>>> To: dev@cloudstack.apache.org
>>>>>>> >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Right, component isn't a thing. I probably shouldn't use
>>>>>>>that
>>>>>>> >> term.
>>>>>>> >>>>>>> I
>>>>>>> >>>>>> want to
>>>>>>> >>>>>>> standarize on the naming convention of plugin, module, and
>>>>>>> >> extension.
>>>>>>> >>>>>> It is
>>>>>>> >>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>>>>>>> >>>>>>> better job
>>>>>>> >>>>>> here. So a
>>>>>>> >>>>>>> plugin is basically a jar. A jar contains multiple modules.
>>>>>>> A
>>>>>>> >>>>>>> modules
>>>>>>> >>>>>> ends up
>>>>>>> >>>>>>> being a spring application context that composes multiple
>>>>>>> >>>>>>> configuration
>>>>>>> >>>>>> files.
>>>>>>> >>>>>>> Modules are assembled into a hierarchy at runtime.
>>>>>>>Extensions
>>>>>>> >>>>>>> are implementations of interfaces that exist in a module. A
>>>>>>> >>>>>>> maven project produces a jar, so a plugin ends up being a
>>>>>>>maven
>>>>>>> >>>>>>> project
>>>>>>> >>>> also.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> So currently today we don't have a strong definition of
>>>>>>>Plugin
>>>>>>> >>>>>>> and I
>>>>>>> >>>>>> hope to
>>>>>>> >>>>>>> address that.
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Darren
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> [1]
>>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>>>>>>> >>>>>>> ins%2C+Modules%2C+and+Extensions
>>>>>>> >>>>>>>
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>>>>>>> >>>>>>> <da...@gmail.com>wrote:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>>> sounds great Darren,
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> By component, you mean maven project or some larger chunk
>>>>>>>like
>>>>>>> >>>>>>>> distribution package? (did i miss this definition somewhere
>>>>>>>or
>>>>>>> >>>>>>>> do we define the components now?)
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> regards,
>>>>>>> >>>>>>>> Daan
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>>>>>>> >>>>>>>> <da...@gmail.com> wrote:
>>>>>>> >>>>>>>>> Currently ACS code is fairly modular in that you can add
>>>>>>> >>>>>>>>> plug-ins to ACS
>>>>>>> >>>>>>>> to
>>>>>>> >>>>>>>>> extend most functionality. Unfortunately ACS is not
>>>>>>> >>>>>>>>> packaged in a
>>>>>>> >>>>>>>> modular
>>>>>>> >>>>>>>>> way. It is still delivered essentially as one large unit.
>>>>>>> >>>>>>>>> There are
>>>>>>> >>>>>>>> many
>>>>>>> >>>>>>>>> reason for this but one large barrier to modularizing ACS
>>>>>>>is
>>>>>>> >>>>>>>>> that the Spring configuration is managed as one large unit.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> I propose that we break apart the Spring XML configuration
>>>>>>> >>>>>>>>> such that each component contributes its own configuration.
>>>>>>> >>>>>>>>> Additionally each component will be loaded into its own
>>>>>>> >>>>>>>>> Spring ApplicationContext such that its beans will not
>>>>>>> >>>>>>>>> conflict with the wiring of other beans in ACS. This
>>>>>>>change
>>>>>>> >>>>>>>> will
>>>>>>> >>>>>>>>> lay the foundation for a richer plugin model and
>>>>>>> >>>>>>>>> additionally a more distributed architecture.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> The technical details for this proposal can be found on the
>>>>>>> >>>>>>>>> wiki
>>>>>>> >>>> [1].
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> Darren
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> [1]
>>>>>>> >>>>>>>>
>>>>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>>>>>>> >>>>>>>> ize+
>>>>>>> >>>>>>>> Spri
>>>>>>> >>>>>>>> ng
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>
>>>>>>> >>>>
>>>>>>> >>
>>>>>>
>>>>
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Ah right, okay. So your talking about the order of the adapters.
Currently that is maintained as the order in the AdapterList in the
componentContext.xml. So what I've done is that extensions get added
to "Registries." Registries that need to be ordered can specify an
ordering configuration variable so that when the extensions are found
they are added to the list in a specific order. So the registry
definition for the auth stuff looks something like
<bean id="userAuthenticatorsRegistry"
class="org.apache.cloudstack.spring.lifecycle.registry.ExtensionRegistry">
<property name="orderConfigKey" value="user.authenticators.order" />
<property name="excludeKey" value="user.authenticators.exclude" />
<property name="orderConfigDefault"
value="SHA256SALT,MD5,LDAP,PLAINTEXT" />
</bean>
So one can use user.authenticators.order to change the order and
user.authenticators.exclude to exclude certain extensions from being
used. The default value is also specified in that example.
Darren
On Mon, Sep 23, 2013 at 3:28 PM, Kelven Yang <ke...@citrix.com> wrote:
> I understand that loading order can be completely solved by Spring with
> dependency injection, either within flat context or hierarchy context
> structure. But some sibling plugins in CloudStack do require a flexible
> way to declare ordering for runtime. For example, allocator plugins or
> authenticator plugins. The order itself is currently designated in their
> parent manager to reference to a ordered adapter list.
>
> This order semantics has nothing to do with dependency injection, but
> unfortunately in previous version of CloudStack, it does mix the
> requirement into injection framework and there are business logic relying
> on it. The fact for parent manager to compose ordering has made an
> assumption at compile/load time binding to subject plugins, which we don't
> want to see in the future since we want drop-in jar for plugins, what is
> our answer for this?
>
> Kelven
>
>
> On 9/23/13 1:40 PM, "Darren Shepherd" <da...@gmail.com> wrote:
>
>>Siblings have no relationship to each other really. The load order
>>doesn't matter as one sibling has no visibility to another. Child
>>contexts are pretty much for plug-ins. Core components of ACS will
>>live in the core context as they are all interdependent.
>>
>>I will explain the whole Spring lifecycle though and how it works in the
>>model.
>>
>>Contexts are initialized from parent to child. So the top most parent
>>context is initialized first, then its child, then its grandchildren.
>>When a context is initialized the following happens in the below order
>>
>>1) All beans are instantiated
>>2) Dependencies are wire up (@Inject)
>>3) @PostConstruct is call for all beans in dependency graph order
>>4) Extensions are discovered an registered (so NetworkElement for
>>example will be discovered an registered as a NetworkElement)
>>5) configure() on all ComponentLifecycle beans will be called in the
>>getRunLevel() order
>>
>>Once all modules have been initialized in this fashion then in a
>>parent first child last order start() on all ComponentLifecycle beans
>>is called following the getRunLevel() order for the beans in the
>>context.
>>
>>Darren
>>
>>
>>On Mon, Sep 23, 2013 at 11:34 AM, Kelven Yang <ke...@citrix.com>
>>wrote:
>>> Darren,
>>>
>>> Due to internal release work, I haven't got a chance to review it but
>>>I'm
>>> planning to do so later today and tomorrow. Before that, I have one
>>> question about hierarchy-orginzated context structure, could you
>>>elaborate
>>> an example to the ML on how two sibling plugins to declare their runtime
>>> load order? I'd like to get a feeling on how hard or easy for developers
>>> to do things that involve with structural change under the new hierarchy
>>> mode.
>>>
>>> Kelven
>>>
>>>
>>> On 9/23/13 12:19 AM, "Darren Shepherd" <da...@gmail.com>
>>>wrote:
>>>
>>>>So how do I proceed forward on this? I basically already have this all
>>>>working. I'd really like to get it all committed as soon as possible if
>>>>there are no objections to the approach. The sooner the better.
>>>>
>>>>I already have a bunch of patches pending on review board that change a
>>>>bunch of random but related things. I need all of those patches
>>>>committed before I can submit the next round of patches. I have about 4
>>>>or 5 more. What I'll do is that everything will get committed and then
>>>>their will be one small patch that will be last that flips some config
>>>>files to enable this all. All changes to code will work in both a
>>>>modular and monolithic spring context. So it will be really easy to
>>>>turn
>>>>this off if suddenly something goes terribly wrong.
>>>>
>>>>So I need people to agree this is good and then start
>>>>reviewing/committing my patches. I really want to get this wrapped up
>>>>this week if I can.
>>>>
>>>>Darren
>>>>
>>>>> On Sep 18, 2013, at 7:06 PM, Darren Shepherd
>>>>><da...@gmail.com> wrote:
>>>>>
>>>>> Yes this is one of the many things this is trying to address. Adding
>>>>>a
>>>>>plugin should be plopping your jar in a directory and restarting. You
>>>>>pointed out two things I didn't think about though, command.properties
>>>>>and log4j xml. Let me think about those twos as they should be address
>>>>>also. Basically you should never have to edit a file that is packaged
>>>>>as part of ACS. Only add your artifacts to some directory, ideally
>>>>>just
>>>>>a jar.
>>>>>
>>>>> Darren
>>>>>
>>>>>
>>>>>> On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher
>>>>>><Ch...@netapp.com> wrote:
>>>>>> I've been following this conversation somewhat and would like to
>>>>>>throw
>>>>>>in some background as a plugin writer.
>>>>>>
>>>>>> One thing that concerns me in the current plugin model is the number
>>>>>>of XML/text files that need to be edited to deploy my plugin.
>>>>>> -applicationContext must be edited to add our PluginMangerImpl.
>>>>>> -commands.properties file must be edited to included the permissions
>>>>>>for the APIs we contributed.
>>>>>> -componentContext.xml & nonossComponentContext.xml must be edited to
>>>>>>add our Storage Subsystem Provider API.
>>>>>> -log4j-cloud.xml must be edited to ensure that our logger is enabled
>>>>>>and logging to our necessary default level.
>>>>>>
>>>>>> I know our situation is a bit different than the current plug in
>>>>>>model, but I think it is something we, as a community, are going to
>>>>>>begin seeing more of. For a variety of reasons that I won't get in to
>>>>>>right now, our plugin will be closed source and kept separate from the
>>>>>>ACS source tree. We want our users to be able to simply drop in our
>>>>>>jar
>>>>>>file to the CS directory or run and installer and have it picked up by
>>>>>>the MS upon a restart.
>>>>>>
>>>>>> It sounds like what you are proposing here would be very beneficial
>>>>>>to
>>>>>>plugins that are targeting a deployment model like this.
>>>>>>
>>>>>> Is this something you're looking/hoping/would like to solve, Darren?
>>>>>>
>>>>>> -Chris
>>>>>> --
>>>>>> Chris Suich
>>>>>> chris.suich@netapp.com
>>>>>> NetApp Software Engineer
>>>>>> Data Center Platforms Cloud Solutions
>>>>>> Citrix, Cisco & Red Hat
>>>>>>
>>>>>> On Sep 18, 2013, at 6:44 PM, Darren Shepherd
>>>>>><da...@gmail.com> wrote:
>>>>>>
>>>>>> > I'm not a committer
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang
>>>>>><Fr...@citrix.com> wrote:
>>>>>> >
>>>>>> >> Well. The codes explain more than words.
>>>>>> >> It seems the only extra work is adding a property file that
>>>>>>specifies
>>>>>> >> parent context and current context name, it's not much complex.
>>>>>> >> BTW: any reason for working on repo outside ACS?
>>>>>> >>
>>>>>> >>> -----Original Message-----
>>>>>> >>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>> >>> Sent: Wednesday, September 18, 2013 2:43 PM
>>>>>> >>> To: dev@cloudstack.apache.org
>>>>>> >>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>> >>>
>>>>>> >>> If you want to see this all working you can just fetch the
>>>>>>"no-at-db4"
>>>>>> >>> branch at https://github.com/ibuildthecloud/cloudstack.git
>>>>>> >>>
>>>>>> >>> Plugin composes multiple modules. If modules are siblings they
>>>>>>can't
>>>>>> >> inject
>>>>>> >>> from each other. But a plugin can augment another module if it
>>>>>>chooses
>>>>>> >> to.
>>>>>> >>> The reality is that the core cloudstack is a tangled mess of
>>>>>> >> dependencies such
>>>>>> >>> that most of the core code can't be modularized as it stands. So
>>>>>>there
>>>>>> >> exists a
>>>>>> >>> context towards the top of the hierarchy called "core" that a lot
>>>>>>of jars
>>>>>> >>> contribute to it. Here is the full hierarchy right now. I'll
>>>>>>probably
>>>>>> >> rename a
>>>>>> >>> bunch of things, but this gives you an idea.
>>>>>> >>>
>>>>>> >>> bootstrap
>>>>>> >>> system
>>>>>> >>> core
>>>>>> >>> allocator
>>>>>> >>> allocator-server
>>>>>> >>> planner
>>>>>> >>> api-planner
>>>>>> >>> baremetal-planner
>>>>>> >>> explicit-dedication
>>>>>> >>> host-anti-affinity
>>>>>> >>> implicit-dedication-planner
>>>>>> >>> server-planner
>>>>>> >>> user-concentrated-pod-planner
>>>>>> >>> random-allocator
>>>>>> >>> api
>>>>>> >>> acl-static-role-based
>>>>>> >>> rate-limit
>>>>>> >>> server-api
>>>>>> >>> user-authenticator-ldap
>>>>>> >>> user-authenticator-md5
>>>>>> >>> user-authenticator-plaintext
>>>>>> >>> user-authenticator-sha256salted
>>>>>> >>> backend
>>>>>> >>> alert-adapter-server-backend
>>>>>> >>> compute
>>>>>> >>> alert-adapter-server-compute
>>>>>> >>> baremetal-compute
>>>>>> >>> fencer-server
>>>>>> >>> investigator-server
>>>>>> >>> kvm-compute
>>>>>> >>> ovm-compute
>>>>>> >>> server-compute
>>>>>> >>> xenserver-compute
>>>>>> >>> network
>>>>>> >>> baremetal-network
>>>>>> >>> elb
>>>>>> >>> midonet
>>>>>> >>> nvp
>>>>>> >>> ovs
>>>>>> >>> server-network
>>>>>> >>> ssp
>>>>>> >>> storage
>>>>>> >>> alert-adapter-server-storage
>>>>>> >>> allocator-storage
>>>>>> >>> baremetal-storage
>>>>>> >>> secondary-storage
>>>>>> >>> server-storage
>>>>>> >>> storage-image-default
>>>>>> >>> storage-image-s3
>>>>>> >>> storage-image-swift
>>>>>> >>> storage-volume-default
>>>>>> >>> storage-volume-solidfire
>>>>>> >>> template-adapter-server
>>>>>> >>> discoverer
>>>>>> >>> baremetal-discoverer
>>>>>> >>> discoverer-server
>>>>>> >>> ovm-discoverer
>>>>>> >>> xenserver-discoverer
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> If you look at the baremetal hypervisor plugin that is pretty
>>>>>>cross
>>>>>> >> cutting to
>>>>>> >>> most of ACS. So the modules it contributes to are as follows
>>>>>> >>>
>>>>>> >>>
>>>>>>resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>>>>>> >>>
>>>>>>resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>>>>>> >>>
>>>>>>resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>>>>>> >>>
>>>>>>resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>>>>>> >>>
>>>>>>resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>>>>>> >>>
>>>>>>resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>>>>>> >>>
>>>>>> >>> So it creates child contextes of storage, compute, network,
>>>>>>planner, and
>>>>>> >>> discoverer to add its extensions where it needs to be.
>>>>>>Additionally,
>>>>>> >> you'll notice,
>>>>>> >>> it adds some beans to the core context from the file
>>>>>>resources/META-
>>>>>> >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is
>>>>>>because
>>>>>> >> it has
>>>>>> >>> some manager class that is used by multiple contexts.
>>>>>> >>>
>>>>>> >>> Frank, I understand the scare that we are going too complex, but
>>>>>>do you
>>>>>> >> have
>>>>>> >>> some other suggestion? I don't like the idea of one gigantic
>>>>>>spring
>>>>>> >> context. So I
>>>>>> >>> feel I am making it as simple as I can while maintaining some
>>>>>>order.
>>>>>> >> People
>>>>>> >>> just need to create one or more spring xml files and a properties
>>>>>>files
>>>>>> >> that says
>>>>>> >>> where to put it in the hierarchy.
>>>>>> >>>
>>>>>> >>> Additionally, by putting forcing people to put beans in certains
>>>>>>modules
>>>>>> >> it
>>>>>> >>> forces them to think about what is the role of the code. For
>>>>>>example,
>>>>>> >> today in
>>>>>> >>> ACS the *ManagerImpl classes are a huge mess. They implement too
>>>>>>many
>>>>>> >>> interfaces and the code crosses to many architectural boundaries.
>>>>>>Its
>>>>>> >> about
>>>>>> >>> time we start splitting things up to be more maintainable.
>>>>>> >>>
>>>>>> >>> If you have some time, please check out what I have on github.
>>>>>> >>>
>>>>>> >>> Darren
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang
>>>>>><Fr...@citrix.com>
>>>>>> >>> wrote:
>>>>>> >>>
>>>>>> >>>> I am not against boundary, I am just against making things
>>>>>>unnecessary
>>>>>> >>>> complex to enable boundary.
>>>>>> >>>> If we are going this way, I hope we can make it as much as
>>>>>>transparent
>>>>>> >>>> to developers. That means, as a developer, all a plugin I need
>>>>>>to
>>>>>>do
>>>>>> >>>> is 1) provide my separate spring xml 2) inject beans I want
>>>>>>(valid
>>>>>> >>>> beans) in my bean and code business logic 3) compile to a jar
>>>>>>and
>>>>>>put
>>>>>> >>>> to some place designated by CloudStack. That's it.
>>>>>> >>>>
>>>>>> >>>> I raise this topic because I have seen some projects to create
>>>>>> >>>> boundary making things horrible complex. And sometimes
>>>>>>developers
>>>>>>are
>>>>>> >>>> hurt by wrong boundaries, as a result, to overcome these
>>>>>>limitations
>>>>>> >>>> people write lots of ugly code which makes thing even worse.
>>>>>> >>>>
>>>>>> >>>> However, I am still worry about if we can make things so
>>>>>>simpler.
>>>>>> >>>> For example, we may have an orchestration context that contains
>>>>>>major
>>>>>> >>>> beans needed by almost every plugin, this context can be easily
>>>>>>set
>>>>>> >>>> as parent context for all plugin contexts when bootstrap.
>>>>>>However, if
>>>>>> >>>> a plugin A needs to access some bean defined in plugin B, given
>>>>>>they
>>>>>> >>>> are sibling, how plugin framework resolves the dependency ?
>>>>>> >>>>
>>>>>> >>>>> -----Original Message-----
>>>>>> >>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>> >>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>>>>>> >>>>> To: dev@cloudstack.apache.org
>>>>>> >>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>> >>>>>
>>>>>> >>>>> I'm not for OSGi either, but contexts are quite useful and will
>>>>>>lead
>>>>>> >>>>> to
>>>>>> >>>> better
>>>>>> >>>>> things. First off, we don't want one gigantic spring XML
>>>>>>config
>>>>>> >>>>> file
>>>>>> >>>> like we have
>>>>>> >>>>> today. I think we can all agree on that. So each plugin will
>>>>>>have
>>>>>> >>>>> to
>>>>>> >>>> supply its
>>>>>> >>>>> own XML. So the obstacles you mention, will largely be just
>>>>>>that
>>>>>> >>>>> for
>>>>>> >>>> people.
>>>>>> >>>>>
>>>>>> >>>>> With Spring it is really simple to just inject dependencies and
>>>>>> >>>>> cross architectural boundaries. Its currently everywhere in
>>>>>>ACS.
>>>>>> >>>>> You can't
>>>>>> >>>> just say
>>>>>> >>>>> we should review code and make sure nobody doesn't do bad
>>>>>>things. A
>>>>>> >>>> little bit
>>>>>> >>>>> of framework to enforce separation is a good thing. But I'm
>>>>>> >>>>> guessing
>>>>>> >>>> you will
>>>>>> >>>>> disagree with me there.
>>>>>> >>>>>
>>>>>> >>>>> Here are some random points on why contexts are good. Say I
>>>>>>want to
>>>>>> >>>>> use Spring AOP or Spring TX in my plugin. With your own
>>>>>>context
>>>>>>you
>>>>>> >>>>> can
>>>>>> >>>> ensure
>>>>>> >>>>> that you won't screw with anybody else code by accidentally
>>>>>>having
>>>>>> >>>>> you pointcut match their bean. You don't have to worry about
>>>>>>bean
>>>>>> >>>>> name
>>>>>> >>>> conflicts.
>>>>>> >>>>> If two config files specify bean X, Spring will gladly just use
>>>>>>the
>>>>>> >>>>> last
>>>>>> >>>> one. I've
>>>>>> >>>>> already found multiply defined beans in ACS, but things still
>>>>>>just
>>>>>> >>>> happen to work.
>>>>>> >>>>> Having multiple contexts also defines initialization order. We
>>>>>>can
>>>>>> >>>> ensure that
>>>>>> >>>>> the framework is loaded and ready before child context are
>>>>>>loaded
>>>>>> >>>>> and
>>>>>> >>>> started.
>>>>>> >>>>> (we kind of do this today with ComponentLifeCycle, but its a
>>>>>>hack in
>>>>>> >>>>> my
>>>>>> >>>> mind).
>>>>>> >>>>> Additionally, when things start you will know, loading context
>>>>>> >>>>> "crapping
>>>>>> >>>> plug X".
>>>>>> >>>>> If spring fails to initialize, the issue it there. Today, if
>>>>>>spring
>>>>>> >>>> fails to start, it
>>>>>> >>>>> could be one of over 500 beans causing the weird problem. The
>>>>>>list
>>>>>> >>>>> goes
>>>>>> >>>> on
>>>>>> >>>>> and on.
>>>>>> >>>>>
>>>>>> >>>>> Finally, this is the big one and why I really want contexts. I
>>>>>>have
>>>>>> >>>> some notes on
>>>>>> >>>>> the wiki [1] that you might want to read through. Basically I
>>>>>>want
>>>>>> >>>>> to
>>>>>> >>>> get to a
>>>>>> >>>>> more flexible deployment model that allows both a single
>>>>>>monolithic
>>>>>> >>>>> JVM
>>>>>> >>>> as
>>>>>> >>>>> today and also a fully distributed system. Having contexts in
>>>>>> >>>>> hierarchy
>>>>>> >>>> will
>>>>>> >>>>> enable me to accomplish this. By selecting which contexts are
>>>>>> >>>>> loaded at runtime will determine what role the JVM takes on.
>>>>>>The
>>>>>> >>>>> contexts help
>>>>>> >>>> people
>>>>>> >>>>> better understand how the distributed architecture will work
>>>>>>too,
>>>>>> >>>>> when
>>>>>> >>>> we get
>>>>>> >>>>> there.
>>>>>> >>>>>
>>>>>> >>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>>>>>> >>>>> classloader
>>>>>> >>>> magic,
>>>>>> >>>>> etc. But I do like organization and a little bit of framework
>>>>>>so
>>>>>> >>>>> that
>>>>>> >>>> people don't
>>>>>> >>>>> accidentally shoot themselves in the foot. I personally like
>>>>>> >>>>> knowing
>>>>>> >>>> that I
>>>>>> >>>>> couldn't have screwed something up, because the framework won't
>>>>>>even
>>>>>> >>>> allow
>>>>>> >>>>> it. If we separate everything as I want today, and then
>>>>>>tomorrow we
>>>>>> >>>>> say
>>>>>> >>>> this is
>>>>>> >>>>> way too much overhead, moving to a flat context is simple.
>>>>>>Don't
>>>>>> >>>>> think
>>>>>> >>>> we are
>>>>>> >>>>> on some slippery slope to classloaders and dependency hell.
>>>>>> >>>>>
>>>>>> >>>>> Darren
>>>>>> >>>>>
>>>>>> >>>>> [1]
>>>>>> >>>>>
>>>>>> >>>>
>>>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>>>>>> >>>> her
>>>>>> >>>>> e...#Nothingtoseehere...-DeploymentModels
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>>
>>>>>> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>>>>>> >>>>> <Fr...@citrix.com>wrote:
>>>>>> >>>>>
>>>>>> >>>>>> What's the point in using separate spring context per plugin?
>>>>>> >>>>>> Separate class loader is the thing I hate most in OSGI, I am
>>>>>> >>>>>> afraid we are on the same way.
>>>>>> >>>>>> Frankly speaking, I never see benefits of this *separate*
>>>>>>model,
>>>>>> >>>>>> our project(or most projects) is not like Chrome which has to
>>>>>> >>>>>> create sandbox for extensions in order to avoid bad plugin
>>>>>>screws
>>>>>> >>>>>> up the whole browser(however, I still see bad plugins screw up
>>>>>>my
>>>>>> >>>>>> Chrome well).
>>>>>> >>>>>> Projects like CloudStack using plugin to decouple architecture
>>>>>> >>>>>> should not introduce many isolations to plugin writer, the
>>>>>>point
>>>>>> >>>>>> preventing wrong use of some components is not making much
>>>>>>sense
>>>>>> >>>>>> to me. If a plugin not following guide(if we have
>>>>>> >>>>>> it) we should kick it out, instead of making obstacles for 99%
>>>>>> >>>>>> good
>>>>>> >>>> people.
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>> -----Original Message-----
>>>>>> >>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>> >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>>>>>> >>>>>>> To: dev@cloudstack.apache.org
>>>>>> >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>> >>>>>>>
>>>>>> >>>>>>> Right, component isn't a thing. I probably shouldn't use
>>>>>>that
>>>>>> >> term.
>>>>>> >>>>>>> I
>>>>>> >>>>>> want to
>>>>>> >>>>>>> standarize on the naming convention of plugin, module, and
>>>>>> >> extension.
>>>>>> >>>>>> It is
>>>>>> >>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>>>>>> >>>>>>> better job
>>>>>> >>>>>> here. So a
>>>>>> >>>>>>> plugin is basically a jar. A jar contains multiple modules.
>>>>>> A
>>>>>> >>>>>>> modules
>>>>>> >>>>>> ends up
>>>>>> >>>>>>> being a spring application context that composes multiple
>>>>>> >>>>>>> configuration
>>>>>> >>>>>> files.
>>>>>> >>>>>>> Modules are assembled into a hierarchy at runtime.
>>>>>>Extensions
>>>>>> >>>>>>> are implementations of interfaces that exist in a module. A
>>>>>> >>>>>>> maven project produces a jar, so a plugin ends up being a
>>>>>>maven
>>>>>> >>>>>>> project
>>>>>> >>>> also.
>>>>>> >>>>>>>
>>>>>> >>>>>>> So currently today we don't have a strong definition of
>>>>>>Plugin
>>>>>> >>>>>>> and I
>>>>>> >>>>>> hope to
>>>>>> >>>>>>> address that.
>>>>>> >>>>>>>
>>>>>> >>>>>>> Darren
>>>>>> >>>>>>>
>>>>>> >>>>>>> [1]
>>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>>>>>> >>>>>>> ins%2C+Modules%2C+and+Extensions
>>>>>> >>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>>>>>> >>>>>>> <da...@gmail.com>wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>>> sounds great Darren,
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> By component, you mean maven project or some larger chunk
>>>>>>like
>>>>>> >>>>>>>> distribution package? (did i miss this definition somewhere
>>>>>>or
>>>>>> >>>>>>>> do we define the components now?)
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> regards,
>>>>>> >>>>>>>> Daan
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>>>>>> >>>>>>>> <da...@gmail.com> wrote:
>>>>>> >>>>>>>>> Currently ACS code is fairly modular in that you can add
>>>>>> >>>>>>>>> plug-ins to ACS
>>>>>> >>>>>>>> to
>>>>>> >>>>>>>>> extend most functionality. Unfortunately ACS is not
>>>>>> >>>>>>>>> packaged in a
>>>>>> >>>>>>>> modular
>>>>>> >>>>>>>>> way. It is still delivered essentially as one large unit.
>>>>>> >>>>>>>>> There are
>>>>>> >>>>>>>> many
>>>>>> >>>>>>>>> reason for this but one large barrier to modularizing ACS
>>>>>>is
>>>>>> >>>>>>>>> that the Spring configuration is managed as one large unit.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> I propose that we break apart the Spring XML configuration
>>>>>> >>>>>>>>> such that each component contributes its own configuration.
>>>>>> >>>>>>>>> Additionally each component will be loaded into its own
>>>>>> >>>>>>>>> Spring ApplicationContext such that its beans will not
>>>>>> >>>>>>>>> conflict with the wiring of other beans in ACS. This
>>>>>>change
>>>>>> >>>>>>>> will
>>>>>> >>>>>>>>> lay the foundation for a richer plugin model and
>>>>>> >>>>>>>>> additionally a more distributed architecture.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> The technical details for this proposal can be found on the
>>>>>> >>>>>>>>> wiki
>>>>>> >>>> [1].
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Darren
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> [1]
>>>>>> >>>>>>>>
>>>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>>>>>> >>>>>>>> ize+
>>>>>> >>>>>>>> Spri
>>>>>> >>>>>>>> ng
>>>>>> >>>>>>>>
>>>>>> >>>>>>
>>>>>> >>>>
>>>>>> >>
>>>>>
>>>
>
Re: [PROPOSAL] Modularize Spring
Posted by Kelven Yang <ke...@citrix.com>.
I understand that loading order can be completely solved by Spring with
dependency injection, either within flat context or hierarchy context
structure. But some sibling plugins in CloudStack do require a flexible
way to declare ordering for runtime. For example, allocator plugins or
authenticator plugins. The order itself is currently designated in their
parent manager to reference to a ordered adapter list.
This order semantics has nothing to do with dependency injection, but
unfortunately in previous version of CloudStack, it does mix the
requirement into injection framework and there are business logic relying
on it. The fact for parent manager to compose ordering has made an
assumption at compile/load time binding to subject plugins, which we don't
want to see in the future since we want drop-in jar for plugins, what is
our answer for this?
Kelven
On 9/23/13 1:40 PM, "Darren Shepherd" <da...@gmail.com> wrote:
>Siblings have no relationship to each other really. The load order
>doesn't matter as one sibling has no visibility to another. Child
>contexts are pretty much for plug-ins. Core components of ACS will
>live in the core context as they are all interdependent.
>
>I will explain the whole Spring lifecycle though and how it works in the
>model.
>
>Contexts are initialized from parent to child. So the top most parent
>context is initialized first, then its child, then its grandchildren.
>When a context is initialized the following happens in the below order
>
>1) All beans are instantiated
>2) Dependencies are wire up (@Inject)
>3) @PostConstruct is call for all beans in dependency graph order
>4) Extensions are discovered an registered (so NetworkElement for
>example will be discovered an registered as a NetworkElement)
>5) configure() on all ComponentLifecycle beans will be called in the
>getRunLevel() order
>
>Once all modules have been initialized in this fashion then in a
>parent first child last order start() on all ComponentLifecycle beans
>is called following the getRunLevel() order for the beans in the
>context.
>
>Darren
>
>
>On Mon, Sep 23, 2013 at 11:34 AM, Kelven Yang <ke...@citrix.com>
>wrote:
>> Darren,
>>
>> Due to internal release work, I haven't got a chance to review it but
>>I'm
>> planning to do so later today and tomorrow. Before that, I have one
>> question about hierarchy-orginzated context structure, could you
>>elaborate
>> an example to the ML on how two sibling plugins to declare their runtime
>> load order? I'd like to get a feeling on how hard or easy for developers
>> to do things that involve with structural change under the new hierarchy
>> mode.
>>
>> Kelven
>>
>>
>> On 9/23/13 12:19 AM, "Darren Shepherd" <da...@gmail.com>
>>wrote:
>>
>>>So how do I proceed forward on this? I basically already have this all
>>>working. I'd really like to get it all committed as soon as possible if
>>>there are no objections to the approach. The sooner the better.
>>>
>>>I already have a bunch of patches pending on review board that change a
>>>bunch of random but related things. I need all of those patches
>>>committed before I can submit the next round of patches. I have about 4
>>>or 5 more. What I'll do is that everything will get committed and then
>>>their will be one small patch that will be last that flips some config
>>>files to enable this all. All changes to code will work in both a
>>>modular and monolithic spring context. So it will be really easy to
>>>turn
>>>this off if suddenly something goes terribly wrong.
>>>
>>>So I need people to agree this is good and then start
>>>reviewing/committing my patches. I really want to get this wrapped up
>>>this week if I can.
>>>
>>>Darren
>>>
>>>> On Sep 18, 2013, at 7:06 PM, Darren Shepherd
>>>><da...@gmail.com> wrote:
>>>>
>>>> Yes this is one of the many things this is trying to address. Adding
>>>>a
>>>>plugin should be plopping your jar in a directory and restarting. You
>>>>pointed out two things I didn't think about though, command.properties
>>>>and log4j xml. Let me think about those twos as they should be address
>>>>also. Basically you should never have to edit a file that is packaged
>>>>as part of ACS. Only add your artifacts to some directory, ideally
>>>>just
>>>>a jar.
>>>>
>>>> Darren
>>>>
>>>>
>>>>> On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher
>>>>><Ch...@netapp.com> wrote:
>>>>> I've been following this conversation somewhat and would like to
>>>>>throw
>>>>>in some background as a plugin writer.
>>>>>
>>>>> One thing that concerns me in the current plugin model is the number
>>>>>of XML/text files that need to be edited to deploy my plugin.
>>>>> -applicationContext must be edited to add our PluginMangerImpl.
>>>>> -commands.properties file must be edited to included the permissions
>>>>>for the APIs we contributed.
>>>>> -componentContext.xml & nonossComponentContext.xml must be edited to
>>>>>add our Storage Subsystem Provider API.
>>>>> -log4j-cloud.xml must be edited to ensure that our logger is enabled
>>>>>and logging to our necessary default level.
>>>>>
>>>>> I know our situation is a bit different than the current plug in
>>>>>model, but I think it is something we, as a community, are going to
>>>>>begin seeing more of. For a variety of reasons that I won't get in to
>>>>>right now, our plugin will be closed source and kept separate from the
>>>>>ACS source tree. We want our users to be able to simply drop in our
>>>>>jar
>>>>>file to the CS directory or run and installer and have it picked up by
>>>>>the MS upon a restart.
>>>>>
>>>>> It sounds like what you are proposing here would be very beneficial
>>>>>to
>>>>>plugins that are targeting a deployment model like this.
>>>>>
>>>>> Is this something you're looking/hoping/would like to solve, Darren?
>>>>>
>>>>> -Chris
>>>>> --
>>>>> Chris Suich
>>>>> chris.suich@netapp.com
>>>>> NetApp Software Engineer
>>>>> Data Center Platforms Cloud Solutions
>>>>> Citrix, Cisco & Red Hat
>>>>>
>>>>> On Sep 18, 2013, at 6:44 PM, Darren Shepherd
>>>>><da...@gmail.com> wrote:
>>>>>
>>>>> > I'm not a committer
>>>>> >
>>>>> >
>>>>> > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang
>>>>><Fr...@citrix.com> wrote:
>>>>> >
>>>>> >> Well. The codes explain more than words.
>>>>> >> It seems the only extra work is adding a property file that
>>>>>specifies
>>>>> >> parent context and current context name, it's not much complex.
>>>>> >> BTW: any reason for working on repo outside ACS?
>>>>> >>
>>>>> >>> -----Original Message-----
>>>>> >>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>> >>> Sent: Wednesday, September 18, 2013 2:43 PM
>>>>> >>> To: dev@cloudstack.apache.org
>>>>> >>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>> >>>
>>>>> >>> If you want to see this all working you can just fetch the
>>>>>"no-at-db4"
>>>>> >>> branch at https://github.com/ibuildthecloud/cloudstack.git
>>>>> >>>
>>>>> >>> Plugin composes multiple modules. If modules are siblings they
>>>>>can't
>>>>> >> inject
>>>>> >>> from each other. But a plugin can augment another module if it
>>>>>chooses
>>>>> >> to.
>>>>> >>> The reality is that the core cloudstack is a tangled mess of
>>>>> >> dependencies such
>>>>> >>> that most of the core code can't be modularized as it stands. So
>>>>>there
>>>>> >> exists a
>>>>> >>> context towards the top of the hierarchy called "core" that a lot
>>>>>of jars
>>>>> >>> contribute to it. Here is the full hierarchy right now. I'll
>>>>>probably
>>>>> >> rename a
>>>>> >>> bunch of things, but this gives you an idea.
>>>>> >>>
>>>>> >>> bootstrap
>>>>> >>> system
>>>>> >>> core
>>>>> >>> allocator
>>>>> >>> allocator-server
>>>>> >>> planner
>>>>> >>> api-planner
>>>>> >>> baremetal-planner
>>>>> >>> explicit-dedication
>>>>> >>> host-anti-affinity
>>>>> >>> implicit-dedication-planner
>>>>> >>> server-planner
>>>>> >>> user-concentrated-pod-planner
>>>>> >>> random-allocator
>>>>> >>> api
>>>>> >>> acl-static-role-based
>>>>> >>> rate-limit
>>>>> >>> server-api
>>>>> >>> user-authenticator-ldap
>>>>> >>> user-authenticator-md5
>>>>> >>> user-authenticator-plaintext
>>>>> >>> user-authenticator-sha256salted
>>>>> >>> backend
>>>>> >>> alert-adapter-server-backend
>>>>> >>> compute
>>>>> >>> alert-adapter-server-compute
>>>>> >>> baremetal-compute
>>>>> >>> fencer-server
>>>>> >>> investigator-server
>>>>> >>> kvm-compute
>>>>> >>> ovm-compute
>>>>> >>> server-compute
>>>>> >>> xenserver-compute
>>>>> >>> network
>>>>> >>> baremetal-network
>>>>> >>> elb
>>>>> >>> midonet
>>>>> >>> nvp
>>>>> >>> ovs
>>>>> >>> server-network
>>>>> >>> ssp
>>>>> >>> storage
>>>>> >>> alert-adapter-server-storage
>>>>> >>> allocator-storage
>>>>> >>> baremetal-storage
>>>>> >>> secondary-storage
>>>>> >>> server-storage
>>>>> >>> storage-image-default
>>>>> >>> storage-image-s3
>>>>> >>> storage-image-swift
>>>>> >>> storage-volume-default
>>>>> >>> storage-volume-solidfire
>>>>> >>> template-adapter-server
>>>>> >>> discoverer
>>>>> >>> baremetal-discoverer
>>>>> >>> discoverer-server
>>>>> >>> ovm-discoverer
>>>>> >>> xenserver-discoverer
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> If you look at the baremetal hypervisor plugin that is pretty
>>>>>cross
>>>>> >> cutting to
>>>>> >>> most of ACS. So the modules it contributes to are as follows
>>>>> >>>
>>>>> >>>
>>>>>resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>>>>> >>>
>>>>>resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>>>>> >>>
>>>>>resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>>>>> >>>
>>>>>resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>>>>> >>>
>>>>>resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>>>>> >>>
>>>>>resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>>>>> >>>
>>>>> >>> So it creates child contextes of storage, compute, network,
>>>>>planner, and
>>>>> >>> discoverer to add its extensions where it needs to be.
>>>>>Additionally,
>>>>> >> you'll notice,
>>>>> >>> it adds some beans to the core context from the file
>>>>>resources/META-
>>>>> >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is
>>>>>because
>>>>> >> it has
>>>>> >>> some manager class that is used by multiple contexts.
>>>>> >>>
>>>>> >>> Frank, I understand the scare that we are going too complex, but
>>>>>do you
>>>>> >> have
>>>>> >>> some other suggestion? I don't like the idea of one gigantic
>>>>>spring
>>>>> >> context. So I
>>>>> >>> feel I am making it as simple as I can while maintaining some
>>>>>order.
>>>>> >> People
>>>>> >>> just need to create one or more spring xml files and a properties
>>>>>files
>>>>> >> that says
>>>>> >>> where to put it in the hierarchy.
>>>>> >>>
>>>>> >>> Additionally, by putting forcing people to put beans in certains
>>>>>modules
>>>>> >> it
>>>>> >>> forces them to think about what is the role of the code. For
>>>>>example,
>>>>> >> today in
>>>>> >>> ACS the *ManagerImpl classes are a huge mess. They implement too
>>>>>many
>>>>> >>> interfaces and the code crosses to many architectural boundaries.
>>>>>Its
>>>>> >> about
>>>>> >>> time we start splitting things up to be more maintainable.
>>>>> >>>
>>>>> >>> If you have some time, please check out what I have on github.
>>>>> >>>
>>>>> >>> Darren
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang
>>>>><Fr...@citrix.com>
>>>>> >>> wrote:
>>>>> >>>
>>>>> >>>> I am not against boundary, I am just against making things
>>>>>unnecessary
>>>>> >>>> complex to enable boundary.
>>>>> >>>> If we are going this way, I hope we can make it as much as
>>>>>transparent
>>>>> >>>> to developers. That means, as a developer, all a plugin I need
>>>>>to
>>>>>do
>>>>> >>>> is 1) provide my separate spring xml 2) inject beans I want
>>>>>(valid
>>>>> >>>> beans) in my bean and code business logic 3) compile to a jar
>>>>>and
>>>>>put
>>>>> >>>> to some place designated by CloudStack. That's it.
>>>>> >>>>
>>>>> >>>> I raise this topic because I have seen some projects to create
>>>>> >>>> boundary making things horrible complex. And sometimes
>>>>>developers
>>>>>are
>>>>> >>>> hurt by wrong boundaries, as a result, to overcome these
>>>>>limitations
>>>>> >>>> people write lots of ugly code which makes thing even worse.
>>>>> >>>>
>>>>> >>>> However, I am still worry about if we can make things so
>>>>>simpler.
>>>>> >>>> For example, we may have an orchestration context that contains
>>>>>major
>>>>> >>>> beans needed by almost every plugin, this context can be easily
>>>>>set
>>>>> >>>> as parent context for all plugin contexts when bootstrap.
>>>>>However, if
>>>>> >>>> a plugin A needs to access some bean defined in plugin B, given
>>>>>they
>>>>> >>>> are sibling, how plugin framework resolves the dependency ?
>>>>> >>>>
>>>>> >>>>> -----Original Message-----
>>>>> >>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>> >>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>>>>> >>>>> To: dev@cloudstack.apache.org
>>>>> >>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>> >>>>>
>>>>> >>>>> I'm not for OSGi either, but contexts are quite useful and will
>>>>>lead
>>>>> >>>>> to
>>>>> >>>> better
>>>>> >>>>> things. First off, we don't want one gigantic spring XML
>>>>>config
>>>>> >>>>> file
>>>>> >>>> like we have
>>>>> >>>>> today. I think we can all agree on that. So each plugin will
>>>>>have
>>>>> >>>>> to
>>>>> >>>> supply its
>>>>> >>>>> own XML. So the obstacles you mention, will largely be just
>>>>>that
>>>>> >>>>> for
>>>>> >>>> people.
>>>>> >>>>>
>>>>> >>>>> With Spring it is really simple to just inject dependencies and
>>>>> >>>>> cross architectural boundaries. Its currently everywhere in
>>>>>ACS.
>>>>> >>>>> You can't
>>>>> >>>> just say
>>>>> >>>>> we should review code and make sure nobody doesn't do bad
>>>>>things. A
>>>>> >>>> little bit
>>>>> >>>>> of framework to enforce separation is a good thing. But I'm
>>>>> >>>>> guessing
>>>>> >>>> you will
>>>>> >>>>> disagree with me there.
>>>>> >>>>>
>>>>> >>>>> Here are some random points on why contexts are good. Say I
>>>>>want to
>>>>> >>>>> use Spring AOP or Spring TX in my plugin. With your own
>>>>>context
>>>>>you
>>>>> >>>>> can
>>>>> >>>> ensure
>>>>> >>>>> that you won't screw with anybody else code by accidentally
>>>>>having
>>>>> >>>>> you pointcut match their bean. You don't have to worry about
>>>>>bean
>>>>> >>>>> name
>>>>> >>>> conflicts.
>>>>> >>>>> If two config files specify bean X, Spring will gladly just use
>>>>>the
>>>>> >>>>> last
>>>>> >>>> one. I've
>>>>> >>>>> already found multiply defined beans in ACS, but things still
>>>>>just
>>>>> >>>> happen to work.
>>>>> >>>>> Having multiple contexts also defines initialization order. We
>>>>>can
>>>>> >>>> ensure that
>>>>> >>>>> the framework is loaded and ready before child context are
>>>>>loaded
>>>>> >>>>> and
>>>>> >>>> started.
>>>>> >>>>> (we kind of do this today with ComponentLifeCycle, but its a
>>>>>hack in
>>>>> >>>>> my
>>>>> >>>> mind).
>>>>> >>>>> Additionally, when things start you will know, loading context
>>>>> >>>>> "crapping
>>>>> >>>> plug X".
>>>>> >>>>> If spring fails to initialize, the issue it there. Today, if
>>>>>spring
>>>>> >>>> fails to start, it
>>>>> >>>>> could be one of over 500 beans causing the weird problem. The
>>>>>list
>>>>> >>>>> goes
>>>>> >>>> on
>>>>> >>>>> and on.
>>>>> >>>>>
>>>>> >>>>> Finally, this is the big one and why I really want contexts. I
>>>>>have
>>>>> >>>> some notes on
>>>>> >>>>> the wiki [1] that you might want to read through. Basically I
>>>>>want
>>>>> >>>>> to
>>>>> >>>> get to a
>>>>> >>>>> more flexible deployment model that allows both a single
>>>>>monolithic
>>>>> >>>>> JVM
>>>>> >>>> as
>>>>> >>>>> today and also a fully distributed system. Having contexts in
>>>>> >>>>> hierarchy
>>>>> >>>> will
>>>>> >>>>> enable me to accomplish this. By selecting which contexts are
>>>>> >>>>> loaded at runtime will determine what role the JVM takes on.
>>>>>The
>>>>> >>>>> contexts help
>>>>> >>>> people
>>>>> >>>>> better understand how the distributed architecture will work
>>>>>too,
>>>>> >>>>> when
>>>>> >>>> we get
>>>>> >>>>> there.
>>>>> >>>>>
>>>>> >>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>>>>> >>>>> classloader
>>>>> >>>> magic,
>>>>> >>>>> etc. But I do like organization and a little bit of framework
>>>>>so
>>>>> >>>>> that
>>>>> >>>> people don't
>>>>> >>>>> accidentally shoot themselves in the foot. I personally like
>>>>> >>>>> knowing
>>>>> >>>> that I
>>>>> >>>>> couldn't have screwed something up, because the framework won't
>>>>>even
>>>>> >>>> allow
>>>>> >>>>> it. If we separate everything as I want today, and then
>>>>>tomorrow we
>>>>> >>>>> say
>>>>> >>>> this is
>>>>> >>>>> way too much overhead, moving to a flat context is simple.
>>>>>Don't
>>>>> >>>>> think
>>>>> >>>> we are
>>>>> >>>>> on some slippery slope to classloaders and dependency hell.
>>>>> >>>>>
>>>>> >>>>> Darren
>>>>> >>>>>
>>>>> >>>>> [1]
>>>>> >>>>>
>>>>> >>>>
>>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>>>>> >>>> her
>>>>> >>>>> e...#Nothingtoseehere...-DeploymentModels
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>>
>>>>> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>>>>> >>>>> <Fr...@citrix.com>wrote:
>>>>> >>>>>
>>>>> >>>>>> What's the point in using separate spring context per plugin?
>>>>> >>>>>> Separate class loader is the thing I hate most in OSGI, I am
>>>>> >>>>>> afraid we are on the same way.
>>>>> >>>>>> Frankly speaking, I never see benefits of this *separate*
>>>>>model,
>>>>> >>>>>> our project(or most projects) is not like Chrome which has to
>>>>> >>>>>> create sandbox for extensions in order to avoid bad plugin
>>>>>screws
>>>>> >>>>>> up the whole browser(however, I still see bad plugins screw up
>>>>>my
>>>>> >>>>>> Chrome well).
>>>>> >>>>>> Projects like CloudStack using plugin to decouple architecture
>>>>> >>>>>> should not introduce many isolations to plugin writer, the
>>>>>point
>>>>> >>>>>> preventing wrong use of some components is not making much
>>>>>sense
>>>>> >>>>>> to me. If a plugin not following guide(if we have
>>>>> >>>>>> it) we should kick it out, instead of making obstacles for 99%
>>>>> >>>>>> good
>>>>> >>>> people.
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>> -----Original Message-----
>>>>> >>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>> >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>>>>> >>>>>>> To: dev@cloudstack.apache.org
>>>>> >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>> >>>>>>>
>>>>> >>>>>>> Right, component isn't a thing. I probably shouldn't use
>>>>>that
>>>>> >> term.
>>>>> >>>>>>> I
>>>>> >>>>>> want to
>>>>> >>>>>>> standarize on the naming convention of plugin, module, and
>>>>> >> extension.
>>>>> >>>>>> It is
>>>>> >>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>>>>> >>>>>>> better job
>>>>> >>>>>> here. So a
>>>>> >>>>>>> plugin is basically a jar. A jar contains multiple modules.
>>>>> A
>>>>> >>>>>>> modules
>>>>> >>>>>> ends up
>>>>> >>>>>>> being a spring application context that composes multiple
>>>>> >>>>>>> configuration
>>>>> >>>>>> files.
>>>>> >>>>>>> Modules are assembled into a hierarchy at runtime.
>>>>>Extensions
>>>>> >>>>>>> are implementations of interfaces that exist in a module. A
>>>>> >>>>>>> maven project produces a jar, so a plugin ends up being a
>>>>>maven
>>>>> >>>>>>> project
>>>>> >>>> also.
>>>>> >>>>>>>
>>>>> >>>>>>> So currently today we don't have a strong definition of
>>>>>Plugin
>>>>> >>>>>>> and I
>>>>> >>>>>> hope to
>>>>> >>>>>>> address that.
>>>>> >>>>>>>
>>>>> >>>>>>> Darren
>>>>> >>>>>>>
>>>>> >>>>>>> [1]
>>>>> >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>>>>> >>>>>>> ins%2C+Modules%2C+and+Extensions
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>>>>> >>>>>>> <da...@gmail.com>wrote:
>>>>> >>>>>>>
>>>>> >>>>>>>> sounds great Darren,
>>>>> >>>>>>>>
>>>>> >>>>>>>> By component, you mean maven project or some larger chunk
>>>>>like
>>>>> >>>>>>>> distribution package? (did i miss this definition somewhere
>>>>>or
>>>>> >>>>>>>> do we define the components now?)
>>>>> >>>>>>>>
>>>>> >>>>>>>> regards,
>>>>> >>>>>>>> Daan
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>>>>> >>>>>>>> <da...@gmail.com> wrote:
>>>>> >>>>>>>>> Currently ACS code is fairly modular in that you can add
>>>>> >>>>>>>>> plug-ins to ACS
>>>>> >>>>>>>> to
>>>>> >>>>>>>>> extend most functionality. Unfortunately ACS is not
>>>>> >>>>>>>>> packaged in a
>>>>> >>>>>>>> modular
>>>>> >>>>>>>>> way. It is still delivered essentially as one large unit.
>>>>> >>>>>>>>> There are
>>>>> >>>>>>>> many
>>>>> >>>>>>>>> reason for this but one large barrier to modularizing ACS
>>>>>is
>>>>> >>>>>>>>> that the Spring configuration is managed as one large unit.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I propose that we break apart the Spring XML configuration
>>>>> >>>>>>>>> such that each component contributes its own configuration.
>>>>> >>>>>>>>> Additionally each component will be loaded into its own
>>>>> >>>>>>>>> Spring ApplicationContext such that its beans will not
>>>>> >>>>>>>>> conflict with the wiring of other beans in ACS. This
>>>>>change
>>>>> >>>>>>>> will
>>>>> >>>>>>>>> lay the foundation for a richer plugin model and
>>>>> >>>>>>>>> additionally a more distributed architecture.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> The technical details for this proposal can be found on the
>>>>> >>>>>>>>> wiki
>>>>> >>>> [1].
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Darren
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> [1]
>>>>> >>>>>>>>
>>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>>>>> >>>>>>>> ize+
>>>>> >>>>>>>> Spri
>>>>> >>>>>>>> ng
>>>>> >>>>>>>>
>>>>> >>>>>>
>>>>> >>>>
>>>>> >>
>>>>
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Siblings have no relationship to each other really. The load order
doesn't matter as one sibling has no visibility to another. Child
contexts are pretty much for plug-ins. Core components of ACS will
live in the core context as they are all interdependent.
I will explain the whole Spring lifecycle though and how it works in the model.
Contexts are initialized from parent to child. So the top most parent
context is initialized first, then its child, then its grandchildren.
When a context is initialized the following happens in the below order
1) All beans are instantiated
2) Dependencies are wire up (@Inject)
3) @PostConstruct is call for all beans in dependency graph order
4) Extensions are discovered an registered (so NetworkElement for
example will be discovered an registered as a NetworkElement)
5) configure() on all ComponentLifecycle beans will be called in the
getRunLevel() order
Once all modules have been initialized in this fashion then in a
parent first child last order start() on all ComponentLifecycle beans
is called following the getRunLevel() order for the beans in the
context.
Darren
On Mon, Sep 23, 2013 at 11:34 AM, Kelven Yang <ke...@citrix.com> wrote:
> Darren,
>
> Due to internal release work, I haven't got a chance to review it but I'm
> planning to do so later today and tomorrow. Before that, I have one
> question about hierarchy-orginzated context structure, could you elaborate
> an example to the ML on how two sibling plugins to declare their runtime
> load order? I'd like to get a feeling on how hard or easy for developers
> to do things that involve with structural change under the new hierarchy
> mode.
>
> Kelven
>
>
> On 9/23/13 12:19 AM, "Darren Shepherd" <da...@gmail.com> wrote:
>
>>So how do I proceed forward on this? I basically already have this all
>>working. I'd really like to get it all committed as soon as possible if
>>there are no objections to the approach. The sooner the better.
>>
>>I already have a bunch of patches pending on review board that change a
>>bunch of random but related things. I need all of those patches
>>committed before I can submit the next round of patches. I have about 4
>>or 5 more. What I'll do is that everything will get committed and then
>>their will be one small patch that will be last that flips some config
>>files to enable this all. All changes to code will work in both a
>>modular and monolithic spring context. So it will be really easy to turn
>>this off if suddenly something goes terribly wrong.
>>
>>So I need people to agree this is good and then start
>>reviewing/committing my patches. I really want to get this wrapped up
>>this week if I can.
>>
>>Darren
>>
>>> On Sep 18, 2013, at 7:06 PM, Darren Shepherd
>>><da...@gmail.com> wrote:
>>>
>>> Yes this is one of the many things this is trying to address. Adding a
>>>plugin should be plopping your jar in a directory and restarting. You
>>>pointed out two things I didn't think about though, command.properties
>>>and log4j xml. Let me think about those twos as they should be address
>>>also. Basically you should never have to edit a file that is packaged
>>>as part of ACS. Only add your artifacts to some directory, ideally just
>>>a jar.
>>>
>>> Darren
>>>
>>>
>>>> On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher
>>>><Ch...@netapp.com> wrote:
>>>> I've been following this conversation somewhat and would like to throw
>>>>in some background as a plugin writer.
>>>>
>>>> One thing that concerns me in the current plugin model is the number
>>>>of XML/text files that need to be edited to deploy my plugin.
>>>> -applicationContext must be edited to add our PluginMangerImpl.
>>>> -commands.properties file must be edited to included the permissions
>>>>for the APIs we contributed.
>>>> -componentContext.xml & nonossComponentContext.xml must be edited to
>>>>add our Storage Subsystem Provider API.
>>>> -log4j-cloud.xml must be edited to ensure that our logger is enabled
>>>>and logging to our necessary default level.
>>>>
>>>> I know our situation is a bit different than the current plug in
>>>>model, but I think it is something we, as a community, are going to
>>>>begin seeing more of. For a variety of reasons that I won't get in to
>>>>right now, our plugin will be closed source and kept separate from the
>>>>ACS source tree. We want our users to be able to simply drop in our jar
>>>>file to the CS directory or run and installer and have it picked up by
>>>>the MS upon a restart.
>>>>
>>>> It sounds like what you are proposing here would be very beneficial to
>>>>plugins that are targeting a deployment model like this.
>>>>
>>>> Is this something you're looking/hoping/would like to solve, Darren?
>>>>
>>>> -Chris
>>>> --
>>>> Chris Suich
>>>> chris.suich@netapp.com
>>>> NetApp Software Engineer
>>>> Data Center Platforms Cloud Solutions
>>>> Citrix, Cisco & Red Hat
>>>>
>>>> On Sep 18, 2013, at 6:44 PM, Darren Shepherd
>>>><da...@gmail.com> wrote:
>>>>
>>>> > I'm not a committer
>>>> >
>>>> >
>>>> > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang
>>>><Fr...@citrix.com> wrote:
>>>> >
>>>> >> Well. The codes explain more than words.
>>>> >> It seems the only extra work is adding a property file that
>>>>specifies
>>>> >> parent context and current context name, it's not much complex.
>>>> >> BTW: any reason for working on repo outside ACS?
>>>> >>
>>>> >>> -----Original Message-----
>>>> >>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>> >>> Sent: Wednesday, September 18, 2013 2:43 PM
>>>> >>> To: dev@cloudstack.apache.org
>>>> >>> Subject: Re: [PROPOSAL] Modularize Spring
>>>> >>>
>>>> >>> If you want to see this all working you can just fetch the
>>>>"no-at-db4"
>>>> >>> branch at https://github.com/ibuildthecloud/cloudstack.git
>>>> >>>
>>>> >>> Plugin composes multiple modules. If modules are siblings they
>>>>can't
>>>> >> inject
>>>> >>> from each other. But a plugin can augment another module if it
>>>>chooses
>>>> >> to.
>>>> >>> The reality is that the core cloudstack is a tangled mess of
>>>> >> dependencies such
>>>> >>> that most of the core code can't be modularized as it stands. So
>>>>there
>>>> >> exists a
>>>> >>> context towards the top of the hierarchy called "core" that a lot
>>>>of jars
>>>> >>> contribute to it. Here is the full hierarchy right now. I'll
>>>>probably
>>>> >> rename a
>>>> >>> bunch of things, but this gives you an idea.
>>>> >>>
>>>> >>> bootstrap
>>>> >>> system
>>>> >>> core
>>>> >>> allocator
>>>> >>> allocator-server
>>>> >>> planner
>>>> >>> api-planner
>>>> >>> baremetal-planner
>>>> >>> explicit-dedication
>>>> >>> host-anti-affinity
>>>> >>> implicit-dedication-planner
>>>> >>> server-planner
>>>> >>> user-concentrated-pod-planner
>>>> >>> random-allocator
>>>> >>> api
>>>> >>> acl-static-role-based
>>>> >>> rate-limit
>>>> >>> server-api
>>>> >>> user-authenticator-ldap
>>>> >>> user-authenticator-md5
>>>> >>> user-authenticator-plaintext
>>>> >>> user-authenticator-sha256salted
>>>> >>> backend
>>>> >>> alert-adapter-server-backend
>>>> >>> compute
>>>> >>> alert-adapter-server-compute
>>>> >>> baremetal-compute
>>>> >>> fencer-server
>>>> >>> investigator-server
>>>> >>> kvm-compute
>>>> >>> ovm-compute
>>>> >>> server-compute
>>>> >>> xenserver-compute
>>>> >>> network
>>>> >>> baremetal-network
>>>> >>> elb
>>>> >>> midonet
>>>> >>> nvp
>>>> >>> ovs
>>>> >>> server-network
>>>> >>> ssp
>>>> >>> storage
>>>> >>> alert-adapter-server-storage
>>>> >>> allocator-storage
>>>> >>> baremetal-storage
>>>> >>> secondary-storage
>>>> >>> server-storage
>>>> >>> storage-image-default
>>>> >>> storage-image-s3
>>>> >>> storage-image-swift
>>>> >>> storage-volume-default
>>>> >>> storage-volume-solidfire
>>>> >>> template-adapter-server
>>>> >>> discoverer
>>>> >>> baremetal-discoverer
>>>> >>> discoverer-server
>>>> >>> ovm-discoverer
>>>> >>> xenserver-discoverer
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> If you look at the baremetal hypervisor plugin that is pretty cross
>>>> >> cutting to
>>>> >>> most of ACS. So the modules it contributes to are as follows
>>>> >>>
>>>> >>> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>>>> >>> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>>>> >>>
>>>>resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>>>> >>>
>>>>resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>>>> >>> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>>>> >>> resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>>>> >>>
>>>> >>> So it creates child contextes of storage, compute, network,
>>>>planner, and
>>>> >>> discoverer to add its extensions where it needs to be.
>>>>Additionally,
>>>> >> you'll notice,
>>>> >>> it adds some beans to the core context from the file
>>>>resources/META-
>>>> >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is
>>>>because
>>>> >> it has
>>>> >>> some manager class that is used by multiple contexts.
>>>> >>>
>>>> >>> Frank, I understand the scare that we are going too complex, but
>>>>do you
>>>> >> have
>>>> >>> some other suggestion? I don't like the idea of one gigantic
>>>>spring
>>>> >> context. So I
>>>> >>> feel I am making it as simple as I can while maintaining some
>>>>order.
>>>> >> People
>>>> >>> just need to create one or more spring xml files and a properties
>>>>files
>>>> >> that says
>>>> >>> where to put it in the hierarchy.
>>>> >>>
>>>> >>> Additionally, by putting forcing people to put beans in certains
>>>>modules
>>>> >> it
>>>> >>> forces them to think about what is the role of the code. For
>>>>example,
>>>> >> today in
>>>> >>> ACS the *ManagerImpl classes are a huge mess. They implement too
>>>>many
>>>> >>> interfaces and the code crosses to many architectural boundaries.
>>>>Its
>>>> >> about
>>>> >>> time we start splitting things up to be more maintainable.
>>>> >>>
>>>> >>> If you have some time, please check out what I have on github.
>>>> >>>
>>>> >>> Darren
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang
>>>><Fr...@citrix.com>
>>>> >>> wrote:
>>>> >>>
>>>> >>>> I am not against boundary, I am just against making things
>>>>unnecessary
>>>> >>>> complex to enable boundary.
>>>> >>>> If we are going this way, I hope we can make it as much as
>>>>transparent
>>>> >>>> to developers. That means, as a developer, all a plugin I need to
>>>>do
>>>> >>>> is 1) provide my separate spring xml 2) inject beans I want (valid
>>>> >>>> beans) in my bean and code business logic 3) compile to a jar and
>>>>put
>>>> >>>> to some place designated by CloudStack. That's it.
>>>> >>>>
>>>> >>>> I raise this topic because I have seen some projects to create
>>>> >>>> boundary making things horrible complex. And sometimes developers
>>>>are
>>>> >>>> hurt by wrong boundaries, as a result, to overcome these
>>>>limitations
>>>> >>>> people write lots of ugly code which makes thing even worse.
>>>> >>>>
>>>> >>>> However, I am still worry about if we can make things so simpler.
>>>> >>>> For example, we may have an orchestration context that contains
>>>>major
>>>> >>>> beans needed by almost every plugin, this context can be easily
>>>>set
>>>> >>>> as parent context for all plugin contexts when bootstrap.
>>>>However, if
>>>> >>>> a plugin A needs to access some bean defined in plugin B, given
>>>>they
>>>> >>>> are sibling, how plugin framework resolves the dependency ?
>>>> >>>>
>>>> >>>>> -----Original Message-----
>>>> >>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>> >>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>>>> >>>>> To: dev@cloudstack.apache.org
>>>> >>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>> >>>>>
>>>> >>>>> I'm not for OSGi either, but contexts are quite useful and will
>>>>lead
>>>> >>>>> to
>>>> >>>> better
>>>> >>>>> things. First off, we don't want one gigantic spring XML config
>>>> >>>>> file
>>>> >>>> like we have
>>>> >>>>> today. I think we can all agree on that. So each plugin will
>>>>have
>>>> >>>>> to
>>>> >>>> supply its
>>>> >>>>> own XML. So the obstacles you mention, will largely be just that
>>>> >>>>> for
>>>> >>>> people.
>>>> >>>>>
>>>> >>>>> With Spring it is really simple to just inject dependencies and
>>>> >>>>> cross architectural boundaries. Its currently everywhere in ACS.
>>>> >>>>> You can't
>>>> >>>> just say
>>>> >>>>> we should review code and make sure nobody doesn't do bad
>>>>things. A
>>>> >>>> little bit
>>>> >>>>> of framework to enforce separation is a good thing. But I'm
>>>> >>>>> guessing
>>>> >>>> you will
>>>> >>>>> disagree with me there.
>>>> >>>>>
>>>> >>>>> Here are some random points on why contexts are good. Say I
>>>>want to
>>>> >>>>> use Spring AOP or Spring TX in my plugin. With your own context
>>>>you
>>>> >>>>> can
>>>> >>>> ensure
>>>> >>>>> that you won't screw with anybody else code by accidentally
>>>>having
>>>> >>>>> you pointcut match their bean. You don't have to worry about
>>>>bean
>>>> >>>>> name
>>>> >>>> conflicts.
>>>> >>>>> If two config files specify bean X, Spring will gladly just use
>>>>the
>>>> >>>>> last
>>>> >>>> one. I've
>>>> >>>>> already found multiply defined beans in ACS, but things still
>>>>just
>>>> >>>> happen to work.
>>>> >>>>> Having multiple contexts also defines initialization order. We
>>>>can
>>>> >>>> ensure that
>>>> >>>>> the framework is loaded and ready before child context are loaded
>>>> >>>>> and
>>>> >>>> started.
>>>> >>>>> (we kind of do this today with ComponentLifeCycle, but its a
>>>>hack in
>>>> >>>>> my
>>>> >>>> mind).
>>>> >>>>> Additionally, when things start you will know, loading context
>>>> >>>>> "crapping
>>>> >>>> plug X".
>>>> >>>>> If spring fails to initialize, the issue it there. Today, if
>>>>spring
>>>> >>>> fails to start, it
>>>> >>>>> could be one of over 500 beans causing the weird problem. The
>>>>list
>>>> >>>>> goes
>>>> >>>> on
>>>> >>>>> and on.
>>>> >>>>>
>>>> >>>>> Finally, this is the big one and why I really want contexts. I
>>>>have
>>>> >>>> some notes on
>>>> >>>>> the wiki [1] that you might want to read through. Basically I
>>>>want
>>>> >>>>> to
>>>> >>>> get to a
>>>> >>>>> more flexible deployment model that allows both a single
>>>>monolithic
>>>> >>>>> JVM
>>>> >>>> as
>>>> >>>>> today and also a fully distributed system. Having contexts in
>>>> >>>>> hierarchy
>>>> >>>> will
>>>> >>>>> enable me to accomplish this. By selecting which contexts are
>>>> >>>>> loaded at runtime will determine what role the JVM takes on. The
>>>> >>>>> contexts help
>>>> >>>> people
>>>> >>>>> better understand how the distributed architecture will work too,
>>>> >>>>> when
>>>> >>>> we get
>>>> >>>>> there.
>>>> >>>>>
>>>> >>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>>>> >>>>> classloader
>>>> >>>> magic,
>>>> >>>>> etc. But I do like organization and a little bit of framework so
>>>> >>>>> that
>>>> >>>> people don't
>>>> >>>>> accidentally shoot themselves in the foot. I personally like
>>>> >>>>> knowing
>>>> >>>> that I
>>>> >>>>> couldn't have screwed something up, because the framework won't
>>>>even
>>>> >>>> allow
>>>> >>>>> it. If we separate everything as I want today, and then
>>>>tomorrow we
>>>> >>>>> say
>>>> >>>> this is
>>>> >>>>> way too much overhead, moving to a flat context is simple. Don't
>>>> >>>>> think
>>>> >>>> we are
>>>> >>>>> on some slippery slope to classloaders and dependency hell.
>>>> >>>>>
>>>> >>>>> Darren
>>>> >>>>>
>>>> >>>>> [1]
>>>> >>>>>
>>>> >>>>
>>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>>>> >>>> her
>>>> >>>>> e...#Nothingtoseehere...-DeploymentModels
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>>>> >>>>> <Fr...@citrix.com>wrote:
>>>> >>>>>
>>>> >>>>>> What's the point in using separate spring context per plugin?
>>>> >>>>>> Separate class loader is the thing I hate most in OSGI, I am
>>>> >>>>>> afraid we are on the same way.
>>>> >>>>>> Frankly speaking, I never see benefits of this *separate* model,
>>>> >>>>>> our project(or most projects) is not like Chrome which has to
>>>> >>>>>> create sandbox for extensions in order to avoid bad plugin
>>>>screws
>>>> >>>>>> up the whole browser(however, I still see bad plugins screw up
>>>>my
>>>> >>>>>> Chrome well).
>>>> >>>>>> Projects like CloudStack using plugin to decouple architecture
>>>> >>>>>> should not introduce many isolations to plugin writer, the point
>>>> >>>>>> preventing wrong use of some components is not making much sense
>>>> >>>>>> to me. If a plugin not following guide(if we have
>>>> >>>>>> it) we should kick it out, instead of making obstacles for 99%
>>>> >>>>>> good
>>>> >>>> people.
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>> -----Original Message-----
>>>> >>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>> >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>>>> >>>>>>> To: dev@cloudstack.apache.org
>>>> >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>> >>>>>>>
>>>> >>>>>>> Right, component isn't a thing. I probably shouldn't use that
>>>> >> term.
>>>> >>>>>>> I
>>>> >>>>>> want to
>>>> >>>>>>> standarize on the naming convention of plugin, module, and
>>>> >> extension.
>>>> >>>>>> It is
>>>> >>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>>>> >>>>>>> better job
>>>> >>>>>> here. So a
>>>> >>>>>>> plugin is basically a jar. A jar contains multiple modules. A
>>>> >>>>>>> modules
>>>> >>>>>> ends up
>>>> >>>>>>> being a spring application context that composes multiple
>>>> >>>>>>> configuration
>>>> >>>>>> files.
>>>> >>>>>>> Modules are assembled into a hierarchy at runtime. Extensions
>>>> >>>>>>> are implementations of interfaces that exist in a module. A
>>>> >>>>>>> maven project produces a jar, so a plugin ends up being a maven
>>>> >>>>>>> project
>>>> >>>> also.
>>>> >>>>>>>
>>>> >>>>>>> So currently today we don't have a strong definition of Plugin
>>>> >>>>>>> and I
>>>> >>>>>> hope to
>>>> >>>>>>> address that.
>>>> >>>>>>>
>>>> >>>>>>> Darren
>>>> >>>>>>>
>>>> >>>>>>> [1]
>>>> >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>>>> >>>>>>> ins%2C+Modules%2C+and+Extensions
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>>>> >>>>>>> <da...@gmail.com>wrote:
>>>> >>>>>>>
>>>> >>>>>>>> sounds great Darren,
>>>> >>>>>>>>
>>>> >>>>>>>> By component, you mean maven project or some larger chunk like
>>>> >>>>>>>> distribution package? (did i miss this definition somewhere or
>>>> >>>>>>>> do we define the components now?)
>>>> >>>>>>>>
>>>> >>>>>>>> regards,
>>>> >>>>>>>> Daan
>>>> >>>>>>>>
>>>> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>>>> >>>>>>>> <da...@gmail.com> wrote:
>>>> >>>>>>>>> Currently ACS code is fairly modular in that you can add
>>>> >>>>>>>>> plug-ins to ACS
>>>> >>>>>>>> to
>>>> >>>>>>>>> extend most functionality. Unfortunately ACS is not
>>>> >>>>>>>>> packaged in a
>>>> >>>>>>>> modular
>>>> >>>>>>>>> way. It is still delivered essentially as one large unit.
>>>> >>>>>>>>> There are
>>>> >>>>>>>> many
>>>> >>>>>>>>> reason for this but one large barrier to modularizing ACS is
>>>> >>>>>>>>> that the Spring configuration is managed as one large unit.
>>>> >>>>>>>>>
>>>> >>>>>>>>> I propose that we break apart the Spring XML configuration
>>>> >>>>>>>>> such that each component contributes its own configuration.
>>>> >>>>>>>>> Additionally each component will be loaded into its own
>>>> >>>>>>>>> Spring ApplicationContext such that its beans will not
>>>> >>>>>>>>> conflict with the wiring of other beans in ACS. This change
>>>> >>>>>>>> will
>>>> >>>>>>>>> lay the foundation for a richer plugin model and
>>>> >>>>>>>>> additionally a more distributed architecture.
>>>> >>>>>>>>>
>>>> >>>>>>>>> The technical details for this proposal can be found on the
>>>> >>>>>>>>> wiki
>>>> >>>> [1].
>>>> >>>>>>>>>
>>>> >>>>>>>>> Darren
>>>> >>>>>>>>>
>>>> >>>>>>>>> [1]
>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>>>> >>>>>>>> ize+
>>>> >>>>>>>> Spri
>>>> >>>>>>>> ng
>>>> >>>>>>>>
>>>> >>>>>>
>>>> >>>>
>>>> >>
>>>
>
Re: [PROPOSAL] Modularize Spring
Posted by Kelven Yang <ke...@citrix.com>.
Darren,
Due to internal release work, I haven't got a chance to review it but I'm
planning to do so later today and tomorrow. Before that, I have one
question about hierarchy-orginzated context structure, could you elaborate
an example to the ML on how two sibling plugins to declare their runtime
load order? I'd like to get a feeling on how hard or easy for developers
to do things that involve with structural change under the new hierarchy
mode.
Kelven
On 9/23/13 12:19 AM, "Darren Shepherd" <da...@gmail.com> wrote:
>So how do I proceed forward on this? I basically already have this all
>working. I'd really like to get it all committed as soon as possible if
>there are no objections to the approach. The sooner the better.
>
>I already have a bunch of patches pending on review board that change a
>bunch of random but related things. I need all of those patches
>committed before I can submit the next round of patches. I have about 4
>or 5 more. What I'll do is that everything will get committed and then
>their will be one small patch that will be last that flips some config
>files to enable this all. All changes to code will work in both a
>modular and monolithic spring context. So it will be really easy to turn
>this off if suddenly something goes terribly wrong.
>
>So I need people to agree this is good and then start
>reviewing/committing my patches. I really want to get this wrapped up
>this week if I can.
>
>Darren
>
>> On Sep 18, 2013, at 7:06 PM, Darren Shepherd
>><da...@gmail.com> wrote:
>>
>> Yes this is one of the many things this is trying to address. Adding a
>>plugin should be plopping your jar in a directory and restarting. You
>>pointed out two things I didn't think about though, command.properties
>>and log4j xml. Let me think about those twos as they should be address
>>also. Basically you should never have to edit a file that is packaged
>>as part of ACS. Only add your artifacts to some directory, ideally just
>>a jar.
>>
>> Darren
>>
>>
>>> On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher
>>><Ch...@netapp.com> wrote:
>>> I've been following this conversation somewhat and would like to throw
>>>in some background as a plugin writer.
>>>
>>> One thing that concerns me in the current plugin model is the number
>>>of XML/text files that need to be edited to deploy my plugin.
>>> -applicationContext must be edited to add our PluginMangerImpl.
>>> -commands.properties file must be edited to included the permissions
>>>for the APIs we contributed.
>>> -componentContext.xml & nonossComponentContext.xml must be edited to
>>>add our Storage Subsystem Provider API.
>>> -log4j-cloud.xml must be edited to ensure that our logger is enabled
>>>and logging to our necessary default level.
>>>
>>> I know our situation is a bit different than the current plug in
>>>model, but I think it is something we, as a community, are going to
>>>begin seeing more of. For a variety of reasons that I won't get in to
>>>right now, our plugin will be closed source and kept separate from the
>>>ACS source tree. We want our users to be able to simply drop in our jar
>>>file to the CS directory or run and installer and have it picked up by
>>>the MS upon a restart.
>>>
>>> It sounds like what you are proposing here would be very beneficial to
>>>plugins that are targeting a deployment model like this.
>>>
>>> Is this something you're looking/hoping/would like to solve, Darren?
>>>
>>> -Chris
>>> --
>>> Chris Suich
>>> chris.suich@netapp.com
>>> NetApp Software Engineer
>>> Data Center Platforms Cloud Solutions
>>> Citrix, Cisco & Red Hat
>>>
>>> On Sep 18, 2013, at 6:44 PM, Darren Shepherd
>>><da...@gmail.com> wrote:
>>>
>>> > I'm not a committer
>>> >
>>> >
>>> > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang
>>><Fr...@citrix.com> wrote:
>>> >
>>> >> Well. The codes explain more than words.
>>> >> It seems the only extra work is adding a property file that
>>>specifies
>>> >> parent context and current context name, it's not much complex.
>>> >> BTW: any reason for working on repo outside ACS?
>>> >>
>>> >>> -----Original Message-----
>>> >>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>> >>> Sent: Wednesday, September 18, 2013 2:43 PM
>>> >>> To: dev@cloudstack.apache.org
>>> >>> Subject: Re: [PROPOSAL] Modularize Spring
>>> >>>
>>> >>> If you want to see this all working you can just fetch the
>>>"no-at-db4"
>>> >>> branch at https://github.com/ibuildthecloud/cloudstack.git
>>> >>>
>>> >>> Plugin composes multiple modules. If modules are siblings they
>>>can't
>>> >> inject
>>> >>> from each other. But a plugin can augment another module if it
>>>chooses
>>> >> to.
>>> >>> The reality is that the core cloudstack is a tangled mess of
>>> >> dependencies such
>>> >>> that most of the core code can't be modularized as it stands. So
>>>there
>>> >> exists a
>>> >>> context towards the top of the hierarchy called "core" that a lot
>>>of jars
>>> >>> contribute to it. Here is the full hierarchy right now. I'll
>>>probably
>>> >> rename a
>>> >>> bunch of things, but this gives you an idea.
>>> >>>
>>> >>> bootstrap
>>> >>> system
>>> >>> core
>>> >>> allocator
>>> >>> allocator-server
>>> >>> planner
>>> >>> api-planner
>>> >>> baremetal-planner
>>> >>> explicit-dedication
>>> >>> host-anti-affinity
>>> >>> implicit-dedication-planner
>>> >>> server-planner
>>> >>> user-concentrated-pod-planner
>>> >>> random-allocator
>>> >>> api
>>> >>> acl-static-role-based
>>> >>> rate-limit
>>> >>> server-api
>>> >>> user-authenticator-ldap
>>> >>> user-authenticator-md5
>>> >>> user-authenticator-plaintext
>>> >>> user-authenticator-sha256salted
>>> >>> backend
>>> >>> alert-adapter-server-backend
>>> >>> compute
>>> >>> alert-adapter-server-compute
>>> >>> baremetal-compute
>>> >>> fencer-server
>>> >>> investigator-server
>>> >>> kvm-compute
>>> >>> ovm-compute
>>> >>> server-compute
>>> >>> xenserver-compute
>>> >>> network
>>> >>> baremetal-network
>>> >>> elb
>>> >>> midonet
>>> >>> nvp
>>> >>> ovs
>>> >>> server-network
>>> >>> ssp
>>> >>> storage
>>> >>> alert-adapter-server-storage
>>> >>> allocator-storage
>>> >>> baremetal-storage
>>> >>> secondary-storage
>>> >>> server-storage
>>> >>> storage-image-default
>>> >>> storage-image-s3
>>> >>> storage-image-swift
>>> >>> storage-volume-default
>>> >>> storage-volume-solidfire
>>> >>> template-adapter-server
>>> >>> discoverer
>>> >>> baremetal-discoverer
>>> >>> discoverer-server
>>> >>> ovm-discoverer
>>> >>> xenserver-discoverer
>>> >>>
>>> >>>
>>> >>>
>>> >>> If you look at the baremetal hypervisor plugin that is pretty cross
>>> >> cutting to
>>> >>> most of ACS. So the modules it contributes to are as follows
>>> >>>
>>> >>> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>>> >>> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>>> >>>
>>>resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>>> >>>
>>>resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>>> >>> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>>> >>> resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>>> >>>
>>> >>> So it creates child contextes of storage, compute, network,
>>>planner, and
>>> >>> discoverer to add its extensions where it needs to be.
>>>Additionally,
>>> >> you'll notice,
>>> >>> it adds some beans to the core context from the file
>>>resources/META-
>>> >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is
>>>because
>>> >> it has
>>> >>> some manager class that is used by multiple contexts.
>>> >>>
>>> >>> Frank, I understand the scare that we are going too complex, but
>>>do you
>>> >> have
>>> >>> some other suggestion? I don't like the idea of one gigantic
>>>spring
>>> >> context. So I
>>> >>> feel I am making it as simple as I can while maintaining some
>>>order.
>>> >> People
>>> >>> just need to create one or more spring xml files and a properties
>>>files
>>> >> that says
>>> >>> where to put it in the hierarchy.
>>> >>>
>>> >>> Additionally, by putting forcing people to put beans in certains
>>>modules
>>> >> it
>>> >>> forces them to think about what is the role of the code. For
>>>example,
>>> >> today in
>>> >>> ACS the *ManagerImpl classes are a huge mess. They implement too
>>>many
>>> >>> interfaces and the code crosses to many architectural boundaries.
>>>Its
>>> >> about
>>> >>> time we start splitting things up to be more maintainable.
>>> >>>
>>> >>> If you have some time, please check out what I have on github.
>>> >>>
>>> >>> Darren
>>> >>>
>>> >>>
>>> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang
>>><Fr...@citrix.com>
>>> >>> wrote:
>>> >>>
>>> >>>> I am not against boundary, I am just against making things
>>>unnecessary
>>> >>>> complex to enable boundary.
>>> >>>> If we are going this way, I hope we can make it as much as
>>>transparent
>>> >>>> to developers. That means, as a developer, all a plugin I need to
>>>do
>>> >>>> is 1) provide my separate spring xml 2) inject beans I want (valid
>>> >>>> beans) in my bean and code business logic 3) compile to a jar and
>>>put
>>> >>>> to some place designated by CloudStack. That's it.
>>> >>>>
>>> >>>> I raise this topic because I have seen some projects to create
>>> >>>> boundary making things horrible complex. And sometimes developers
>>>are
>>> >>>> hurt by wrong boundaries, as a result, to overcome these
>>>limitations
>>> >>>> people write lots of ugly code which makes thing even worse.
>>> >>>>
>>> >>>> However, I am still worry about if we can make things so simpler.
>>> >>>> For example, we may have an orchestration context that contains
>>>major
>>> >>>> beans needed by almost every plugin, this context can be easily
>>>set
>>> >>>> as parent context for all plugin contexts when bootstrap.
>>>However, if
>>> >>>> a plugin A needs to access some bean defined in plugin B, given
>>>they
>>> >>>> are sibling, how plugin framework resolves the dependency ?
>>> >>>>
>>> >>>>> -----Original Message-----
>>> >>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>> >>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>>> >>>>> To: dev@cloudstack.apache.org
>>> >>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>> >>>>>
>>> >>>>> I'm not for OSGi either, but contexts are quite useful and will
>>>lead
>>> >>>>> to
>>> >>>> better
>>> >>>>> things. First off, we don't want one gigantic spring XML config
>>> >>>>> file
>>> >>>> like we have
>>> >>>>> today. I think we can all agree on that. So each plugin will
>>>have
>>> >>>>> to
>>> >>>> supply its
>>> >>>>> own XML. So the obstacles you mention, will largely be just that
>>> >>>>> for
>>> >>>> people.
>>> >>>>>
>>> >>>>> With Spring it is really simple to just inject dependencies and
>>> >>>>> cross architectural boundaries. Its currently everywhere in ACS.
>>> >>>>> You can't
>>> >>>> just say
>>> >>>>> we should review code and make sure nobody doesn't do bad
>>>things. A
>>> >>>> little bit
>>> >>>>> of framework to enforce separation is a good thing. But I'm
>>> >>>>> guessing
>>> >>>> you will
>>> >>>>> disagree with me there.
>>> >>>>>
>>> >>>>> Here are some random points on why contexts are good. Say I
>>>want to
>>> >>>>> use Spring AOP or Spring TX in my plugin. With your own context
>>>you
>>> >>>>> can
>>> >>>> ensure
>>> >>>>> that you won't screw with anybody else code by accidentally
>>>having
>>> >>>>> you pointcut match their bean. You don't have to worry about
>>>bean
>>> >>>>> name
>>> >>>> conflicts.
>>> >>>>> If two config files specify bean X, Spring will gladly just use
>>>the
>>> >>>>> last
>>> >>>> one. I've
>>> >>>>> already found multiply defined beans in ACS, but things still
>>>just
>>> >>>> happen to work.
>>> >>>>> Having multiple contexts also defines initialization order. We
>>>can
>>> >>>> ensure that
>>> >>>>> the framework is loaded and ready before child context are loaded
>>> >>>>> and
>>> >>>> started.
>>> >>>>> (we kind of do this today with ComponentLifeCycle, but its a
>>>hack in
>>> >>>>> my
>>> >>>> mind).
>>> >>>>> Additionally, when things start you will know, loading context
>>> >>>>> "crapping
>>> >>>> plug X".
>>> >>>>> If spring fails to initialize, the issue it there. Today, if
>>>spring
>>> >>>> fails to start, it
>>> >>>>> could be one of over 500 beans causing the weird problem. The
>>>list
>>> >>>>> goes
>>> >>>> on
>>> >>>>> and on.
>>> >>>>>
>>> >>>>> Finally, this is the big one and why I really want contexts. I
>>>have
>>> >>>> some notes on
>>> >>>>> the wiki [1] that you might want to read through. Basically I
>>>want
>>> >>>>> to
>>> >>>> get to a
>>> >>>>> more flexible deployment model that allows both a single
>>>monolithic
>>> >>>>> JVM
>>> >>>> as
>>> >>>>> today and also a fully distributed system. Having contexts in
>>> >>>>> hierarchy
>>> >>>> will
>>> >>>>> enable me to accomplish this. By selecting which contexts are
>>> >>>>> loaded at runtime will determine what role the JVM takes on. The
>>> >>>>> contexts help
>>> >>>> people
>>> >>>>> better understand how the distributed architecture will work too,
>>> >>>>> when
>>> >>>> we get
>>> >>>>> there.
>>> >>>>>
>>> >>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>>> >>>>> classloader
>>> >>>> magic,
>>> >>>>> etc. But I do like organization and a little bit of framework so
>>> >>>>> that
>>> >>>> people don't
>>> >>>>> accidentally shoot themselves in the foot. I personally like
>>> >>>>> knowing
>>> >>>> that I
>>> >>>>> couldn't have screwed something up, because the framework won't
>>>even
>>> >>>> allow
>>> >>>>> it. If we separate everything as I want today, and then
>>>tomorrow we
>>> >>>>> say
>>> >>>> this is
>>> >>>>> way too much overhead, moving to a flat context is simple. Don't
>>> >>>>> think
>>> >>>> we are
>>> >>>>> on some slippery slope to classloaders and dependency hell.
>>> >>>>>
>>> >>>>> Darren
>>> >>>>>
>>> >>>>> [1]
>>> >>>>>
>>> >>>>
>>>https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>>> >>>> her
>>> >>>>> e...#Nothingtoseehere...-DeploymentModels
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>>> >>>>> <Fr...@citrix.com>wrote:
>>> >>>>>
>>> >>>>>> What's the point in using separate spring context per plugin?
>>> >>>>>> Separate class loader is the thing I hate most in OSGI, I am
>>> >>>>>> afraid we are on the same way.
>>> >>>>>> Frankly speaking, I never see benefits of this *separate* model,
>>> >>>>>> our project(or most projects) is not like Chrome which has to
>>> >>>>>> create sandbox for extensions in order to avoid bad plugin
>>>screws
>>> >>>>>> up the whole browser(however, I still see bad plugins screw up
>>>my
>>> >>>>>> Chrome well).
>>> >>>>>> Projects like CloudStack using plugin to decouple architecture
>>> >>>>>> should not introduce many isolations to plugin writer, the point
>>> >>>>>> preventing wrong use of some components is not making much sense
>>> >>>>>> to me. If a plugin not following guide(if we have
>>> >>>>>> it) we should kick it out, instead of making obstacles for 99%
>>> >>>>>> good
>>> >>>> people.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>> -----Original Message-----
>>> >>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>> >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>>> >>>>>>> To: dev@cloudstack.apache.org
>>> >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>> >>>>>>>
>>> >>>>>>> Right, component isn't a thing. I probably shouldn't use that
>>> >> term.
>>> >>>>>>> I
>>> >>>>>> want to
>>> >>>>>>> standarize on the naming convention of plugin, module, and
>>> >> extension.
>>> >>>>>> It is
>>> >>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>>> >>>>>>> better job
>>> >>>>>> here. So a
>>> >>>>>>> plugin is basically a jar. A jar contains multiple modules. A
>>> >>>>>>> modules
>>> >>>>>> ends up
>>> >>>>>>> being a spring application context that composes multiple
>>> >>>>>>> configuration
>>> >>>>>> files.
>>> >>>>>>> Modules are assembled into a hierarchy at runtime. Extensions
>>> >>>>>>> are implementations of interfaces that exist in a module. A
>>> >>>>>>> maven project produces a jar, so a plugin ends up being a maven
>>> >>>>>>> project
>>> >>>> also.
>>> >>>>>>>
>>> >>>>>>> So currently today we don't have a strong definition of Plugin
>>> >>>>>>> and I
>>> >>>>>> hope to
>>> >>>>>>> address that.
>>> >>>>>>>
>>> >>>>>>> Darren
>>> >>>>>>>
>>> >>>>>>> [1]
>>> >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>>> >>>>>>> ins%2C+Modules%2C+and+Extensions
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>>> >>>>>>> <da...@gmail.com>wrote:
>>> >>>>>>>
>>> >>>>>>>> sounds great Darren,
>>> >>>>>>>>
>>> >>>>>>>> By component, you mean maven project or some larger chunk like
>>> >>>>>>>> distribution package? (did i miss this definition somewhere or
>>> >>>>>>>> do we define the components now?)
>>> >>>>>>>>
>>> >>>>>>>> regards,
>>> >>>>>>>> Daan
>>> >>>>>>>>
>>> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>>> >>>>>>>> <da...@gmail.com> wrote:
>>> >>>>>>>>> Currently ACS code is fairly modular in that you can add
>>> >>>>>>>>> plug-ins to ACS
>>> >>>>>>>> to
>>> >>>>>>>>> extend most functionality. Unfortunately ACS is not
>>> >>>>>>>>> packaged in a
>>> >>>>>>>> modular
>>> >>>>>>>>> way. It is still delivered essentially as one large unit.
>>> >>>>>>>>> There are
>>> >>>>>>>> many
>>> >>>>>>>>> reason for this but one large barrier to modularizing ACS is
>>> >>>>>>>>> that the Spring configuration is managed as one large unit.
>>> >>>>>>>>>
>>> >>>>>>>>> I propose that we break apart the Spring XML configuration
>>> >>>>>>>>> such that each component contributes its own configuration.
>>> >>>>>>>>> Additionally each component will be loaded into its own
>>> >>>>>>>>> Spring ApplicationContext such that its beans will not
>>> >>>>>>>>> conflict with the wiring of other beans in ACS. This change
>>> >>>>>>>> will
>>> >>>>>>>>> lay the foundation for a richer plugin model and
>>> >>>>>>>>> additionally a more distributed architecture.
>>> >>>>>>>>>
>>> >>>>>>>>> The technical details for this proposal can be found on the
>>> >>>>>>>>> wiki
>>> >>>> [1].
>>> >>>>>>>>>
>>> >>>>>>>>> Darren
>>> >>>>>>>>>
>>> >>>>>>>>> [1]
>>> >>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>>> >>>>>>>> ize+
>>> >>>>>>>> Spri
>>> >>>>>>>> ng
>>> >>>>>>>>
>>> >>>>>>
>>> >>>>
>>> >>
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
So how do I proceed forward on this? I basically already have this all working. I'd really like to get it all committed as soon as possible if there are no objections to the approach. The sooner the better.
I already have a bunch of patches pending on review board that change a bunch of random but related things. I need all of those patches committed before I can submit the next round of patches. I have about 4 or 5 more. What I'll do is that everything will get committed and then their will be one small patch that will be last that flips some config files to enable this all. All changes to code will work in both a modular and monolithic spring context. So it will be really easy to turn this off if suddenly something goes terribly wrong.
So I need people to agree this is good and then start reviewing/committing my patches. I really want to get this wrapped up this week if I can.
Darren
> On Sep 18, 2013, at 7:06 PM, Darren Shepherd <da...@gmail.com> wrote:
>
> Yes this is one of the many things this is trying to address. Adding a plugin should be plopping your jar in a directory and restarting. You pointed out two things I didn't think about though, command.properties and log4j xml. Let me think about those twos as they should be address also. Basically you should never have to edit a file that is packaged as part of ACS. Only add your artifacts to some directory, ideally just a jar.
>
> Darren
>
>
>> On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher <Ch...@netapp.com> wrote:
>> I've been following this conversation somewhat and would like to throw in some background as a plugin writer.
>>
>> One thing that concerns me in the current plugin model is the number of XML/text files that need to be edited to deploy my plugin.
>> -applicationContext must be edited to add our PluginMangerImpl.
>> -commands.properties file must be edited to included the permissions for the APIs we contributed.
>> -componentContext.xml & nonossComponentContext.xml must be edited to add our Storage Subsystem Provider API.
>> -log4j-cloud.xml must be edited to ensure that our logger is enabled and logging to our necessary default level.
>>
>> I know our situation is a bit different than the current plug in model, but I think it is something we, as a community, are going to begin seeing more of. For a variety of reasons that I won't get in to right now, our plugin will be closed source and kept separate from the ACS source tree. We want our users to be able to simply drop in our jar file to the CS directory or run and installer and have it picked up by the MS upon a restart.
>>
>> It sounds like what you are proposing here would be very beneficial to plugins that are targeting a deployment model like this.
>>
>> Is this something you're looking/hoping/would like to solve, Darren?
>>
>> -Chris
>> --
>> Chris Suich
>> chris.suich@netapp.com
>> NetApp Software Engineer
>> Data Center Platforms – Cloud Solutions
>> Citrix, Cisco & Red Hat
>>
>> On Sep 18, 2013, at 6:44 PM, Darren Shepherd <da...@gmail.com> wrote:
>>
>> > I'm not a committer
>> >
>> >
>> > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang <Fr...@citrix.com> wrote:
>> >
>> >> Well. The codes explain more than words.
>> >> It seems the only extra work is adding a property file that specifies
>> >> parent context and current context name, it's not much complex.
>> >> BTW: any reason for working on repo outside ACS?
>> >>
>> >>> -----Original Message-----
>> >>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>> >>> Sent: Wednesday, September 18, 2013 2:43 PM
>> >>> To: dev@cloudstack.apache.org
>> >>> Subject: Re: [PROPOSAL] Modularize Spring
>> >>>
>> >>> If you want to see this all working you can just fetch the "no-at-db4"
>> >>> branch at https://github.com/ibuildthecloud/cloudstack.git
>> >>>
>> >>> Plugin composes multiple modules. If modules are siblings they can't
>> >> inject
>> >>> from each other. But a plugin can augment another module if it chooses
>> >> to.
>> >>> The reality is that the core cloudstack is a tangled mess of
>> >> dependencies such
>> >>> that most of the core code can't be modularized as it stands. So there
>> >> exists a
>> >>> context towards the top of the hierarchy called "core" that a lot of jars
>> >>> contribute to it. Here is the full hierarchy right now. I'll probably
>> >> rename a
>> >>> bunch of things, but this gives you an idea.
>> >>>
>> >>> bootstrap
>> >>> system
>> >>> core
>> >>> allocator
>> >>> allocator-server
>> >>> planner
>> >>> api-planner
>> >>> baremetal-planner
>> >>> explicit-dedication
>> >>> host-anti-affinity
>> >>> implicit-dedication-planner
>> >>> server-planner
>> >>> user-concentrated-pod-planner
>> >>> random-allocator
>> >>> api
>> >>> acl-static-role-based
>> >>> rate-limit
>> >>> server-api
>> >>> user-authenticator-ldap
>> >>> user-authenticator-md5
>> >>> user-authenticator-plaintext
>> >>> user-authenticator-sha256salted
>> >>> backend
>> >>> alert-adapter-server-backend
>> >>> compute
>> >>> alert-adapter-server-compute
>> >>> baremetal-compute
>> >>> fencer-server
>> >>> investigator-server
>> >>> kvm-compute
>> >>> ovm-compute
>> >>> server-compute
>> >>> xenserver-compute
>> >>> network
>> >>> baremetal-network
>> >>> elb
>> >>> midonet
>> >>> nvp
>> >>> ovs
>> >>> server-network
>> >>> ssp
>> >>> storage
>> >>> alert-adapter-server-storage
>> >>> allocator-storage
>> >>> baremetal-storage
>> >>> secondary-storage
>> >>> server-storage
>> >>> storage-image-default
>> >>> storage-image-s3
>> >>> storage-image-swift
>> >>> storage-volume-default
>> >>> storage-volume-solidfire
>> >>> template-adapter-server
>> >>> discoverer
>> >>> baremetal-discoverer
>> >>> discoverer-server
>> >>> ovm-discoverer
>> >>> xenserver-discoverer
>> >>>
>> >>>
>> >>>
>> >>> If you look at the baremetal hypervisor plugin that is pretty cross
>> >> cutting to
>> >>> most of ACS. So the modules it contributes to are as follows
>> >>>
>> >>> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>> >>> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>> >>> resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>> >>> resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>> >>> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>> >>> resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>> >>>
>> >>> So it creates child contextes of storage, compute, network, planner, and
>> >>> discoverer to add its extensions where it needs to be. Additionally,
>> >> you'll notice,
>> >>> it adds some beans to the core context from the file resources/META-
>> >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is because
>> >> it has
>> >>> some manager class that is used by multiple contexts.
>> >>>
>> >>> Frank, I understand the scare that we are going too complex, but do you
>> >> have
>> >>> some other suggestion? I don't like the idea of one gigantic spring
>> >> context. So I
>> >>> feel I am making it as simple as I can while maintaining some order.
>> >> People
>> >>> just need to create one or more spring xml files and a properties files
>> >> that says
>> >>> where to put it in the hierarchy.
>> >>>
>> >>> Additionally, by putting forcing people to put beans in certains modules
>> >> it
>> >>> forces them to think about what is the role of the code. For example,
>> >> today in
>> >>> ACS the *ManagerImpl classes are a huge mess. They implement too many
>> >>> interfaces and the code crosses to many architectural boundaries. Its
>> >> about
>> >>> time we start splitting things up to be more maintainable.
>> >>>
>> >>> If you have some time, please check out what I have on github.
>> >>>
>> >>> Darren
>> >>>
>> >>>
>> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <Fr...@citrix.com>
>> >>> wrote:
>> >>>
>> >>>> I am not against boundary, I am just against making things unnecessary
>> >>>> complex to enable boundary.
>> >>>> If we are going this way, I hope we can make it as much as transparent
>> >>>> to developers. That means, as a developer, all a plugin I need to do
>> >>>> is 1) provide my separate spring xml 2) inject beans I want (valid
>> >>>> beans) in my bean and code business logic 3) compile to a jar and put
>> >>>> to some place designated by CloudStack. That's it.
>> >>>>
>> >>>> I raise this topic because I have seen some projects to create
>> >>>> boundary making things horrible complex. And sometimes developers are
>> >>>> hurt by wrong boundaries, as a result, to overcome these limitations
>> >>>> people write lots of ugly code which makes thing even worse.
>> >>>>
>> >>>> However, I am still worry about if we can make things so simpler.
>> >>>> For example, we may have an orchestration context that contains major
>> >>>> beans needed by almost every plugin, this context can be easily set
>> >>>> as parent context for all plugin contexts when bootstrap. However, if
>> >>>> a plugin A needs to access some bean defined in plugin B, given they
>> >>>> are sibling, how plugin framework resolves the dependency ?
>> >>>>
>> >>>>> -----Original Message-----
>> >>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>> >>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>> >>>>> To: dev@cloudstack.apache.org
>> >>>>> Subject: Re: [PROPOSAL] Modularize Spring
>> >>>>>
>> >>>>> I'm not for OSGi either, but contexts are quite useful and will lead
>> >>>>> to
>> >>>> better
>> >>>>> things. First off, we don't want one gigantic spring XML config
>> >>>>> file
>> >>>> like we have
>> >>>>> today. I think we can all agree on that. So each plugin will have
>> >>>>> to
>> >>>> supply its
>> >>>>> own XML. So the obstacles you mention, will largely be just that
>> >>>>> for
>> >>>> people.
>> >>>>>
>> >>>>> With Spring it is really simple to just inject dependencies and
>> >>>>> cross architectural boundaries. Its currently everywhere in ACS.
>> >>>>> You can't
>> >>>> just say
>> >>>>> we should review code and make sure nobody doesn't do bad things. A
>> >>>> little bit
>> >>>>> of framework to enforce separation is a good thing. But I'm
>> >>>>> guessing
>> >>>> you will
>> >>>>> disagree with me there.
>> >>>>>
>> >>>>> Here are some random points on why contexts are good. Say I want to
>> >>>>> use Spring AOP or Spring TX in my plugin. With your own context you
>> >>>>> can
>> >>>> ensure
>> >>>>> that you won't screw with anybody else code by accidentally having
>> >>>>> you pointcut match their bean. You don't have to worry about bean
>> >>>>> name
>> >>>> conflicts.
>> >>>>> If two config files specify bean X, Spring will gladly just use the
>> >>>>> last
>> >>>> one. I've
>> >>>>> already found multiply defined beans in ACS, but things still just
>> >>>> happen to work.
>> >>>>> Having multiple contexts also defines initialization order. We can
>> >>>> ensure that
>> >>>>> the framework is loaded and ready before child context are loaded
>> >>>>> and
>> >>>> started.
>> >>>>> (we kind of do this today with ComponentLifeCycle, but its a hack in
>> >>>>> my
>> >>>> mind).
>> >>>>> Additionally, when things start you will know, loading context
>> >>>>> "crapping
>> >>>> plug X".
>> >>>>> If spring fails to initialize, the issue it there. Today, if spring
>> >>>> fails to start, it
>> >>>>> could be one of over 500 beans causing the weird problem. The list
>> >>>>> goes
>> >>>> on
>> >>>>> and on.
>> >>>>>
>> >>>>> Finally, this is the big one and why I really want contexts. I have
>> >>>> some notes on
>> >>>>> the wiki [1] that you might want to read through. Basically I want
>> >>>>> to
>> >>>> get to a
>> >>>>> more flexible deployment model that allows both a single monolithic
>> >>>>> JVM
>> >>>> as
>> >>>>> today and also a fully distributed system. Having contexts in
>> >>>>> hierarchy
>> >>>> will
>> >>>>> enable me to accomplish this. By selecting which contexts are
>> >>>>> loaded at runtime will determine what role the JVM takes on. The
>> >>>>> contexts help
>> >>>> people
>> >>>>> better understand how the distributed architecture will work too,
>> >>>>> when
>> >>>> we get
>> >>>>> there.
>> >>>>>
>> >>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>> >>>>> classloader
>> >>>> magic,
>> >>>>> etc. But I do like organization and a little bit of framework so
>> >>>>> that
>> >>>> people don't
>> >>>>> accidentally shoot themselves in the foot. I personally like
>> >>>>> knowing
>> >>>> that I
>> >>>>> couldn't have screwed something up, because the framework won't even
>> >>>> allow
>> >>>>> it. If we separate everything as I want today, and then tomorrow we
>> >>>>> say
>> >>>> this is
>> >>>>> way too much overhead, moving to a flat context is simple. Don't
>> >>>>> think
>> >>>> we are
>> >>>>> on some slippery slope to classloaders and dependency hell.
>> >>>>>
>> >>>>> Darren
>> >>>>>
>> >>>>> [1]
>> >>>>>
>> >>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>> >>>> her
>> >>>>> e...#Nothingtoseehere...-DeploymentModels
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>> >>>>> <Fr...@citrix.com>wrote:
>> >>>>>
>> >>>>>> What's the point in using separate spring context per plugin?
>> >>>>>> Separate class loader is the thing I hate most in OSGI, I am
>> >>>>>> afraid we are on the same way.
>> >>>>>> Frankly speaking, I never see benefits of this *separate* model,
>> >>>>>> our project(or most projects) is not like Chrome which has to
>> >>>>>> create sandbox for extensions in order to avoid bad plugin screws
>> >>>>>> up the whole browser(however, I still see bad plugins screw up my
>> >>>>>> Chrome well).
>> >>>>>> Projects like CloudStack using plugin to decouple architecture
>> >>>>>> should not introduce many isolations to plugin writer, the point
>> >>>>>> preventing wrong use of some components is not making much sense
>> >>>>>> to me. If a plugin not following guide(if we have
>> >>>>>> it) we should kick it out, instead of making obstacles for 99%
>> >>>>>> good
>> >>>> people.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>> -----Original Message-----
>> >>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>> >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>> >>>>>>> To: dev@cloudstack.apache.org
>> >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>> >>>>>>>
>> >>>>>>> Right, component isn't a thing. I probably shouldn't use that
>> >> term.
>> >>>>>>> I
>> >>>>>> want to
>> >>>>>>> standarize on the naming convention of plugin, module, and
>> >> extension.
>> >>>>>> It is
>> >>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>> >>>>>>> better job
>> >>>>>> here. So a
>> >>>>>>> plugin is basically a jar. A jar contains multiple modules. A
>> >>>>>>> modules
>> >>>>>> ends up
>> >>>>>>> being a spring application context that composes multiple
>> >>>>>>> configuration
>> >>>>>> files.
>> >>>>>>> Modules are assembled into a hierarchy at runtime. Extensions
>> >>>>>>> are implementations of interfaces that exist in a module. A
>> >>>>>>> maven project produces a jar, so a plugin ends up being a maven
>> >>>>>>> project
>> >>>> also.
>> >>>>>>>
>> >>>>>>> So currently today we don't have a strong definition of Plugin
>> >>>>>>> and I
>> >>>>>> hope to
>> >>>>>>> address that.
>> >>>>>>>
>> >>>>>>> Darren
>> >>>>>>>
>> >>>>>>> [1]
>> >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>> >>>>>>> ins%2C+Modules%2C+and+Extensions
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>> >>>>>>> <da...@gmail.com>wrote:
>> >>>>>>>
>> >>>>>>>> sounds great Darren,
>> >>>>>>>>
>> >>>>>>>> By component, you mean maven project or some larger chunk like
>> >>>>>>>> distribution package? (did i miss this definition somewhere or
>> >>>>>>>> do we define the components now?)
>> >>>>>>>>
>> >>>>>>>> regards,
>> >>>>>>>> Daan
>> >>>>>>>>
>> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>> >>>>>>>> <da...@gmail.com> wrote:
>> >>>>>>>>> Currently ACS code is fairly modular in that you can add
>> >>>>>>>>> plug-ins to ACS
>> >>>>>>>> to
>> >>>>>>>>> extend most functionality. Unfortunately ACS is not
>> >>>>>>>>> packaged in a
>> >>>>>>>> modular
>> >>>>>>>>> way. It is still delivered essentially as one large unit.
>> >>>>>>>>> There are
>> >>>>>>>> many
>> >>>>>>>>> reason for this but one large barrier to modularizing ACS is
>> >>>>>>>>> that the Spring configuration is managed as one large unit.
>> >>>>>>>>>
>> >>>>>>>>> I propose that we break apart the Spring XML configuration
>> >>>>>>>>> such that each component contributes its own configuration.
>> >>>>>>>>> Additionally each component will be loaded into its own
>> >>>>>>>>> Spring ApplicationContext such that its beans will not
>> >>>>>>>>> conflict with the wiring of other beans in ACS. This change
>> >>>>>>>> will
>> >>>>>>>>> lay the foundation for a richer plugin model and
>> >>>>>>>>> additionally a more distributed architecture.
>> >>>>>>>>>
>> >>>>>>>>> The technical details for this proposal can be found on the
>> >>>>>>>>> wiki
>> >>>> [1].
>> >>>>>>>>>
>> >>>>>>>>> Darren
>> >>>>>>>>>
>> >>>>>>>>> [1]
>> >>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>> >>>>>>>> ize+
>> >>>>>>>> Spri
>> >>>>>>>> ng
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> >>
>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Yes this is one of the many things this is trying to address. Adding a
plugin should be plopping your jar in a directory and restarting. You
pointed out two things I didn't think about though, command.properties and
log4j xml. Let me think about those twos as they should be address also.
Basically you should never have to edit a file that is packaged as part of
ACS. Only add your artifacts to some directory, ideally just a jar.
Darren
On Wed, Sep 18, 2013 at 5:46 PM, SuichII, Christopher <
Chris.Suich@netapp.com> wrote:
> I've been following this conversation somewhat and would like to throw in
> some background as a plugin writer.
>
> One thing that concerns me in the current plugin model is the number of
> XML/text files that need to be edited to deploy my plugin.
> -applicationContext must be edited to add our PluginMangerImpl.
> -commands.properties file must be edited to included the permissions for
> the APIs we contributed.
> -componentContext.xml & nonossComponentContext.xml must be edited to add
> our Storage Subsystem Provider API.
> -log4j-cloud.xml must be edited to ensure that our logger is enabled and
> logging to our necessary default level.
>
> I know our situation is a bit different than the current plug in model,
> but I think it is something we, as a community, are going to begin seeing
> more of. For a variety of reasons that I won't get in to right now, our
> plugin will be closed source and kept separate from the ACS source tree. We
> want our users to be able to simply drop in our jar file to the CS
> directory or run and installer and have it picked up by the MS upon a
> restart.
>
> It sounds like what you are proposing here would be very beneficial to
> plugins that are targeting a deployment model like this.
>
> Is this something you're looking/hoping/would like to solve, Darren?
>
> -Chris
> --
> Chris Suich
> chris.suich@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Sep 18, 2013, at 6:44 PM, Darren Shepherd <da...@gmail.com>
> wrote:
>
> > I'm not a committer
> >
> >
> > On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang <Fr...@citrix.com>
> wrote:
> >
> >> Well. The codes explain more than words.
> >> It seems the only extra work is adding a property file that specifies
> >> parent context and current context name, it's not much complex.
> >> BTW: any reason for working on repo outside ACS?
> >>
> >>> -----Original Message-----
> >>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> >>> Sent: Wednesday, September 18, 2013 2:43 PM
> >>> To: dev@cloudstack.apache.org
> >>> Subject: Re: [PROPOSAL] Modularize Spring
> >>>
> >>> If you want to see this all working you can just fetch the "no-at-db4"
> >>> branch at https://github.com/ibuildthecloud/cloudstack.git
> >>>
> >>> Plugin composes multiple modules. If modules are siblings they can't
> >> inject
> >>> from each other. But a plugin can augment another module if it chooses
> >> to.
> >>> The reality is that the core cloudstack is a tangled mess of
> >> dependencies such
> >>> that most of the core code can't be modularized as it stands. So there
> >> exists a
> >>> context towards the top of the hierarchy called "core" that a lot of
> jars
> >>> contribute to it. Here is the full hierarchy right now. I'll probably
> >> rename a
> >>> bunch of things, but this gives you an idea.
> >>>
> >>> bootstrap
> >>> system
> >>> core
> >>> allocator
> >>> allocator-server
> >>> planner
> >>> api-planner
> >>> baremetal-planner
> >>> explicit-dedication
> >>> host-anti-affinity
> >>> implicit-dedication-planner
> >>> server-planner
> >>> user-concentrated-pod-planner
> >>> random-allocator
> >>> api
> >>> acl-static-role-based
> >>> rate-limit
> >>> server-api
> >>> user-authenticator-ldap
> >>> user-authenticator-md5
> >>> user-authenticator-plaintext
> >>> user-authenticator-sha256salted
> >>> backend
> >>> alert-adapter-server-backend
> >>> compute
> >>> alert-adapter-server-compute
> >>> baremetal-compute
> >>> fencer-server
> >>> investigator-server
> >>> kvm-compute
> >>> ovm-compute
> >>> server-compute
> >>> xenserver-compute
> >>> network
> >>> baremetal-network
> >>> elb
> >>> midonet
> >>> nvp
> >>> ovs
> >>> server-network
> >>> ssp
> >>> storage
> >>> alert-adapter-server-storage
> >>> allocator-storage
> >>> baremetal-storage
> >>> secondary-storage
> >>> server-storage
> >>> storage-image-default
> >>> storage-image-s3
> >>> storage-image-swift
> >>> storage-volume-default
> >>> storage-volume-solidfire
> >>> template-adapter-server
> >>> discoverer
> >>> baremetal-discoverer
> >>> discoverer-server
> >>> ovm-discoverer
> >>> xenserver-discoverer
> >>>
> >>>
> >>>
> >>> If you look at the baremetal hypervisor plugin that is pretty cross
> >> cutting to
> >>> most of ACS. So the modules it contributes to are as follows
> >>>
> >>> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
> >>> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
> >>> resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
> >>> resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
> >>> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
> >>> resources/META-INF/cloudstack/baremetal-network/spring-context.xml
> >>>
> >>> So it creates child contextes of storage, compute, network, planner,
> and
> >>> discoverer to add its extensions where it needs to be. Additionally,
> >> you'll notice,
> >>> it adds some beans to the core context from the file resources/META-
> >>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is because
> >> it has
> >>> some manager class that is used by multiple contexts.
> >>>
> >>> Frank, I understand the scare that we are going too complex, but do you
> >> have
> >>> some other suggestion? I don't like the idea of one gigantic spring
> >> context. So I
> >>> feel I am making it as simple as I can while maintaining some order.
> >> People
> >>> just need to create one or more spring xml files and a properties files
> >> that says
> >>> where to put it in the hierarchy.
> >>>
> >>> Additionally, by putting forcing people to put beans in certains
> modules
> >> it
> >>> forces them to think about what is the role of the code. For example,
> >> today in
> >>> ACS the *ManagerImpl classes are a huge mess. They implement too many
> >>> interfaces and the code crosses to many architectural boundaries. Its
> >> about
> >>> time we start splitting things up to be more maintainable.
> >>>
> >>> If you have some time, please check out what I have on github.
> >>>
> >>> Darren
> >>>
> >>>
> >>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <Fr...@citrix.com>
> >>> wrote:
> >>>
> >>>> I am not against boundary, I am just against making things unnecessary
> >>>> complex to enable boundary.
> >>>> If we are going this way, I hope we can make it as much as transparent
> >>>> to developers. That means, as a developer, all a plugin I need to do
> >>>> is 1) provide my separate spring xml 2) inject beans I want (valid
> >>>> beans) in my bean and code business logic 3) compile to a jar and put
> >>>> to some place designated by CloudStack. That's it.
> >>>>
> >>>> I raise this topic because I have seen some projects to create
> >>>> boundary making things horrible complex. And sometimes developers are
> >>>> hurt by wrong boundaries, as a result, to overcome these limitations
> >>>> people write lots of ugly code which makes thing even worse.
> >>>>
> >>>> However, I am still worry about if we can make things so simpler.
> >>>> For example, we may have an orchestration context that contains major
> >>>> beans needed by almost every plugin, this context can be easily set
> >>>> as parent context for all plugin contexts when bootstrap. However, if
> >>>> a plugin A needs to access some bean defined in plugin B, given they
> >>>> are sibling, how plugin framework resolves the dependency ?
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> >>>>> Sent: Wednesday, September 18, 2013 11:53 AM
> >>>>> To: dev@cloudstack.apache.org
> >>>>> Subject: Re: [PROPOSAL] Modularize Spring
> >>>>>
> >>>>> I'm not for OSGi either, but contexts are quite useful and will lead
> >>>>> to
> >>>> better
> >>>>> things. First off, we don't want one gigantic spring XML config
> >>>>> file
> >>>> like we have
> >>>>> today. I think we can all agree on that. So each plugin will have
> >>>>> to
> >>>> supply its
> >>>>> own XML. So the obstacles you mention, will largely be just that
> >>>>> for
> >>>> people.
> >>>>>
> >>>>> With Spring it is really simple to just inject dependencies and
> >>>>> cross architectural boundaries. Its currently everywhere in ACS.
> >>>>> You can't
> >>>> just say
> >>>>> we should review code and make sure nobody doesn't do bad things. A
> >>>> little bit
> >>>>> of framework to enforce separation is a good thing. But I'm
> >>>>> guessing
> >>>> you will
> >>>>> disagree with me there.
> >>>>>
> >>>>> Here are some random points on why contexts are good. Say I want to
> >>>>> use Spring AOP or Spring TX in my plugin. With your own context you
> >>>>> can
> >>>> ensure
> >>>>> that you won't screw with anybody else code by accidentally having
> >>>>> you pointcut match their bean. You don't have to worry about bean
> >>>>> name
> >>>> conflicts.
> >>>>> If two config files specify bean X, Spring will gladly just use the
> >>>>> last
> >>>> one. I've
> >>>>> already found multiply defined beans in ACS, but things still just
> >>>> happen to work.
> >>>>> Having multiple contexts also defines initialization order. We can
> >>>> ensure that
> >>>>> the framework is loaded and ready before child context are loaded
> >>>>> and
> >>>> started.
> >>>>> (we kind of do this today with ComponentLifeCycle, but its a hack in
> >>>>> my
> >>>> mind).
> >>>>> Additionally, when things start you will know, loading context
> >>>>> "crapping
> >>>> plug X".
> >>>>> If spring fails to initialize, the issue it there. Today, if spring
> >>>> fails to start, it
> >>>>> could be one of over 500 beans causing the weird problem. The list
> >>>>> goes
> >>>> on
> >>>>> and on.
> >>>>>
> >>>>> Finally, this is the big one and why I really want contexts. I have
> >>>> some notes on
> >>>>> the wiki [1] that you might want to read through. Basically I want
> >>>>> to
> >>>> get to a
> >>>>> more flexible deployment model that allows both a single monolithic
> >>>>> JVM
> >>>> as
> >>>>> today and also a fully distributed system. Having contexts in
> >>>>> hierarchy
> >>>> will
> >>>>> enable me to accomplish this. By selecting which contexts are
> >>>>> loaded at runtime will determine what role the JVM takes on. The
> >>>>> contexts help
> >>>> people
> >>>>> better understand how the distributed architecture will work too,
> >>>>> when
> >>>> we get
> >>>>> there.
> >>>>>
> >>>>> Frank, trust me, I hate complex things. I don't want OSGi,
> >>>>> classloader
> >>>> magic,
> >>>>> etc. But I do like organization and a little bit of framework so
> >>>>> that
> >>>> people don't
> >>>>> accidentally shoot themselves in the foot. I personally like
> >>>>> knowing
> >>>> that I
> >>>>> couldn't have screwed something up, because the framework won't even
> >>>> allow
> >>>>> it. If we separate everything as I want today, and then tomorrow we
> >>>>> say
> >>>> this is
> >>>>> way too much overhead, moving to a flat context is simple. Don't
> >>>>> think
> >>>> we are
> >>>>> on some slippery slope to classloaders and dependency hell.
> >>>>>
> >>>>> Darren
> >>>>>
> >>>>> [1]
> >>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
> >>>> her
> >>>>> e...#Nothingtoseehere...-DeploymentModels
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
> >>>>> <Fr...@citrix.com>wrote:
> >>>>>
> >>>>>> What's the point in using separate spring context per plugin?
> >>>>>> Separate class loader is the thing I hate most in OSGI, I am
> >>>>>> afraid we are on the same way.
> >>>>>> Frankly speaking, I never see benefits of this *separate* model,
> >>>>>> our project(or most projects) is not like Chrome which has to
> >>>>>> create sandbox for extensions in order to avoid bad plugin screws
> >>>>>> up the whole browser(however, I still see bad plugins screw up my
> >>>>>> Chrome well).
> >>>>>> Projects like CloudStack using plugin to decouple architecture
> >>>>>> should not introduce many isolations to plugin writer, the point
> >>>>>> preventing wrong use of some components is not making much sense
> >>>>>> to me. If a plugin not following guide(if we have
> >>>>>> it) we should kick it out, instead of making obstacles for 99%
> >>>>>> good
> >>>> people.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> >>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
> >>>>>>> To: dev@cloudstack.apache.org
> >>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
> >>>>>>>
> >>>>>>> Right, component isn't a thing. I probably shouldn't use that
> >> term.
> >>>>>>> I
> >>>>>> want to
> >>>>>>> standarize on the naming convention of plugin, module, and
> >> extension.
> >>>>>> It is
> >>>>>>> explained a bit on the wiki [1] but I'll try to do a little
> >>>>>>> better job
> >>>>>> here. So a
> >>>>>>> plugin is basically a jar. A jar contains multiple modules. A
> >>>>>>> modules
> >>>>>> ends up
> >>>>>>> being a spring application context that composes multiple
> >>>>>>> configuration
> >>>>>> files.
> >>>>>>> Modules are assembled into a hierarchy at runtime. Extensions
> >>>>>>> are implementations of interfaces that exist in a module. A
> >>>>>>> maven project produces a jar, so a plugin ends up being a maven
> >>>>>>> project
> >>>> also.
> >>>>>>>
> >>>>>>> So currently today we don't have a strong definition of Plugin
> >>>>>>> and I
> >>>>>> hope to
> >>>>>>> address that.
> >>>>>>>
> >>>>>>> Darren
> >>>>>>>
> >>>>>>> [1]
> >>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> >>>>>>> ins%2C+Modules%2C+and+Extensions
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
> >>>>>>> <da...@gmail.com>wrote:
> >>>>>>>
> >>>>>>>> sounds great Darren,
> >>>>>>>>
> >>>>>>>> By component, you mean maven project or some larger chunk like
> >>>>>>>> distribution package? (did i miss this definition somewhere or
> >>>>>>>> do we define the components now?)
> >>>>>>>>
> >>>>>>>> regards,
> >>>>>>>> Daan
> >>>>>>>>
> >>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> >>>>>>>> <da...@gmail.com> wrote:
> >>>>>>>>> Currently ACS code is fairly modular in that you can add
> >>>>>>>>> plug-ins to ACS
> >>>>>>>> to
> >>>>>>>>> extend most functionality. Unfortunately ACS is not
> >>>>>>>>> packaged in a
> >>>>>>>> modular
> >>>>>>>>> way. It is still delivered essentially as one large unit.
> >>>>>>>>> There are
> >>>>>>>> many
> >>>>>>>>> reason for this but one large barrier to modularizing ACS is
> >>>>>>>>> that the Spring configuration is managed as one large unit.
> >>>>>>>>>
> >>>>>>>>> I propose that we break apart the Spring XML configuration
> >>>>>>>>> such that each component contributes its own configuration.
> >>>>>>>>> Additionally each component will be loaded into its own
> >>>>>>>>> Spring ApplicationContext such that its beans will not
> >>>>>>>>> conflict with the wiring of other beans in ACS. This change
> >>>>>>>> will
> >>>>>>>>> lay the foundation for a richer plugin model and
> >>>>>>>>> additionally a more distributed architecture.
> >>>>>>>>>
> >>>>>>>>> The technical details for this proposal can be found on the
> >>>>>>>>> wiki
> >>>> [1].
> >>>>>>>>>
> >>>>>>>>> Darren
> >>>>>>>>>
> >>>>>>>>> [1]
> >>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
> >>>>>>>> ize+
> >>>>>>>> Spri
> >>>>>>>> ng
> >>>>>>>>
> >>>>>>
> >>>>
> >>
>
>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
s/PluginManagerImpl/PluggableService/
Along with a storage provider plug in, we are also creating a generic API plugin to support the UI plugin we are developing (too many plugins….).
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Sep 18, 2013, at 8:46 PM, "SuichII, Christopher" <Ch...@netapp.com> wrote:
> I've been following this conversation somewhat and would like to throw in some background as a plugin writer.
>
> One thing that concerns me in the current plugin model is the number of XML/text files that need to be edited to deploy my plugin.
> -applicationContext must be edited to add our PluginMangerImpl.
> -commands.properties file must be edited to included the permissions for the APIs we contributed.
> -componentContext.xml & nonossComponentContext.xml must be edited to add our Storage Subsystem Provider API.
> -log4j-cloud.xml must be edited to ensure that our logger is enabled and logging to our necessary default level.
>
> I know our situation is a bit different than the current plug in model, but I think it is something we, as a community, are going to begin seeing more of. For a variety of reasons that I won't get in to right now, our plugin will be closed source and kept separate from the ACS source tree. We want our users to be able to simply drop in our jar file to the CS directory or run and installer and have it picked up by the MS upon a restart.
>
> It sounds like what you are proposing here would be very beneficial to plugins that are targeting a deployment model like this.
>
> Is this something you're looking/hoping/would like to solve, Darren?
>
> -Chris
> --
> Chris Suich
> chris.suich@netapp.com
> NetApp Software Engineer
> Data Center Platforms – Cloud Solutions
> Citrix, Cisco & Red Hat
>
> On Sep 18, 2013, at 6:44 PM, Darren Shepherd <da...@gmail.com> wrote:
>
>> I'm not a committer
>>
>>
>> On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang <Fr...@citrix.com> wrote:
>>
>>> Well. The codes explain more than words.
>>> It seems the only extra work is adding a property file that specifies
>>> parent context and current context name, it's not much complex.
>>> BTW: any reason for working on repo outside ACS?
>>>
>>>> -----Original Message-----
>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>> Sent: Wednesday, September 18, 2013 2:43 PM
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>
>>>> If you want to see this all working you can just fetch the "no-at-db4"
>>>> branch at https://github.com/ibuildthecloud/cloudstack.git
>>>>
>>>> Plugin composes multiple modules. If modules are siblings they can't
>>> inject
>>>> from each other. But a plugin can augment another module if it chooses
>>> to.
>>>> The reality is that the core cloudstack is a tangled mess of
>>> dependencies such
>>>> that most of the core code can't be modularized as it stands. So there
>>> exists a
>>>> context towards the top of the hierarchy called "core" that a lot of jars
>>>> contribute to it. Here is the full hierarchy right now. I'll probably
>>> rename a
>>>> bunch of things, but this gives you an idea.
>>>>
>>>> bootstrap
>>>> system
>>>> core
>>>> allocator
>>>> allocator-server
>>>> planner
>>>> api-planner
>>>> baremetal-planner
>>>> explicit-dedication
>>>> host-anti-affinity
>>>> implicit-dedication-planner
>>>> server-planner
>>>> user-concentrated-pod-planner
>>>> random-allocator
>>>> api
>>>> acl-static-role-based
>>>> rate-limit
>>>> server-api
>>>> user-authenticator-ldap
>>>> user-authenticator-md5
>>>> user-authenticator-plaintext
>>>> user-authenticator-sha256salted
>>>> backend
>>>> alert-adapter-server-backend
>>>> compute
>>>> alert-adapter-server-compute
>>>> baremetal-compute
>>>> fencer-server
>>>> investigator-server
>>>> kvm-compute
>>>> ovm-compute
>>>> server-compute
>>>> xenserver-compute
>>>> network
>>>> baremetal-network
>>>> elb
>>>> midonet
>>>> nvp
>>>> ovs
>>>> server-network
>>>> ssp
>>>> storage
>>>> alert-adapter-server-storage
>>>> allocator-storage
>>>> baremetal-storage
>>>> secondary-storage
>>>> server-storage
>>>> storage-image-default
>>>> storage-image-s3
>>>> storage-image-swift
>>>> storage-volume-default
>>>> storage-volume-solidfire
>>>> template-adapter-server
>>>> discoverer
>>>> baremetal-discoverer
>>>> discoverer-server
>>>> ovm-discoverer
>>>> xenserver-discoverer
>>>>
>>>>
>>>>
>>>> If you look at the baremetal hypervisor plugin that is pretty cross
>>> cutting to
>>>> most of ACS. So the modules it contributes to are as follows
>>>>
>>>> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>>>> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>>>> resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>>>> resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>>>> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>>>> resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>>>>
>>>> So it creates child contextes of storage, compute, network, planner, and
>>>> discoverer to add its extensions where it needs to be. Additionally,
>>> you'll notice,
>>>> it adds some beans to the core context from the file resources/META-
>>>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is because
>>> it has
>>>> some manager class that is used by multiple contexts.
>>>>
>>>> Frank, I understand the scare that we are going too complex, but do you
>>> have
>>>> some other suggestion? I don't like the idea of one gigantic spring
>>> context. So I
>>>> feel I am making it as simple as I can while maintaining some order.
>>> People
>>>> just need to create one or more spring xml files and a properties files
>>> that says
>>>> where to put it in the hierarchy.
>>>>
>>>> Additionally, by putting forcing people to put beans in certains modules
>>> it
>>>> forces them to think about what is the role of the code. For example,
>>> today in
>>>> ACS the *ManagerImpl classes are a huge mess. They implement too many
>>>> interfaces and the code crosses to many architectural boundaries. Its
>>> about
>>>> time we start splitting things up to be more maintainable.
>>>>
>>>> If you have some time, please check out what I have on github.
>>>>
>>>> Darren
>>>>
>>>>
>>>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <Fr...@citrix.com>
>>>> wrote:
>>>>
>>>>> I am not against boundary, I am just against making things unnecessary
>>>>> complex to enable boundary.
>>>>> If we are going this way, I hope we can make it as much as transparent
>>>>> to developers. That means, as a developer, all a plugin I need to do
>>>>> is 1) provide my separate spring xml 2) inject beans I want (valid
>>>>> beans) in my bean and code business logic 3) compile to a jar and put
>>>>> to some place designated by CloudStack. That's it.
>>>>>
>>>>> I raise this topic because I have seen some projects to create
>>>>> boundary making things horrible complex. And sometimes developers are
>>>>> hurt by wrong boundaries, as a result, to overcome these limitations
>>>>> people write lots of ugly code which makes thing even worse.
>>>>>
>>>>> However, I am still worry about if we can make things so simpler.
>>>>> For example, we may have an orchestration context that contains major
>>>>> beans needed by almost every plugin, this context can be easily set
>>>>> as parent context for all plugin contexts when bootstrap. However, if
>>>>> a plugin A needs to access some bean defined in plugin B, given they
>>>>> are sibling, how plugin framework resolves the dependency ?
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>>>>>> To: dev@cloudstack.apache.org
>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>>
>>>>>> I'm not for OSGi either, but contexts are quite useful and will lead
>>>>>> to
>>>>> better
>>>>>> things. First off, we don't want one gigantic spring XML config
>>>>>> file
>>>>> like we have
>>>>>> today. I think we can all agree on that. So each plugin will have
>>>>>> to
>>>>> supply its
>>>>>> own XML. So the obstacles you mention, will largely be just that
>>>>>> for
>>>>> people.
>>>>>>
>>>>>> With Spring it is really simple to just inject dependencies and
>>>>>> cross architectural boundaries. Its currently everywhere in ACS.
>>>>>> You can't
>>>>> just say
>>>>>> we should review code and make sure nobody doesn't do bad things. A
>>>>> little bit
>>>>>> of framework to enforce separation is a good thing. But I'm
>>>>>> guessing
>>>>> you will
>>>>>> disagree with me there.
>>>>>>
>>>>>> Here are some random points on why contexts are good. Say I want to
>>>>>> use Spring AOP or Spring TX in my plugin. With your own context you
>>>>>> can
>>>>> ensure
>>>>>> that you won't screw with anybody else code by accidentally having
>>>>>> you pointcut match their bean. You don't have to worry about bean
>>>>>> name
>>>>> conflicts.
>>>>>> If two config files specify bean X, Spring will gladly just use the
>>>>>> last
>>>>> one. I've
>>>>>> already found multiply defined beans in ACS, but things still just
>>>>> happen to work.
>>>>>> Having multiple contexts also defines initialization order. We can
>>>>> ensure that
>>>>>> the framework is loaded and ready before child context are loaded
>>>>>> and
>>>>> started.
>>>>>> (we kind of do this today with ComponentLifeCycle, but its a hack in
>>>>>> my
>>>>> mind).
>>>>>> Additionally, when things start you will know, loading context
>>>>>> "crapping
>>>>> plug X".
>>>>>> If spring fails to initialize, the issue it there. Today, if spring
>>>>> fails to start, it
>>>>>> could be one of over 500 beans causing the weird problem. The list
>>>>>> goes
>>>>> on
>>>>>> and on.
>>>>>>
>>>>>> Finally, this is the big one and why I really want contexts. I have
>>>>> some notes on
>>>>>> the wiki [1] that you might want to read through. Basically I want
>>>>>> to
>>>>> get to a
>>>>>> more flexible deployment model that allows both a single monolithic
>>>>>> JVM
>>>>> as
>>>>>> today and also a fully distributed system. Having contexts in
>>>>>> hierarchy
>>>>> will
>>>>>> enable me to accomplish this. By selecting which contexts are
>>>>>> loaded at runtime will determine what role the JVM takes on. The
>>>>>> contexts help
>>>>> people
>>>>>> better understand how the distributed architecture will work too,
>>>>>> when
>>>>> we get
>>>>>> there.
>>>>>>
>>>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>>>>>> classloader
>>>>> magic,
>>>>>> etc. But I do like organization and a little bit of framework so
>>>>>> that
>>>>> people don't
>>>>>> accidentally shoot themselves in the foot. I personally like
>>>>>> knowing
>>>>> that I
>>>>>> couldn't have screwed something up, because the framework won't even
>>>>> allow
>>>>>> it. If we separate everything as I want today, and then tomorrow we
>>>>>> say
>>>>> this is
>>>>>> way too much overhead, moving to a flat context is simple. Don't
>>>>>> think
>>>>> we are
>>>>>> on some slippery slope to classloaders and dependency hell.
>>>>>>
>>>>>> Darren
>>>>>>
>>>>>> [1]
>>>>>>
>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>>>>> her
>>>>>> e...#Nothingtoseehere...-DeploymentModels
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>>>>>> <Fr...@citrix.com>wrote:
>>>>>>
>>>>>>> What's the point in using separate spring context per plugin?
>>>>>>> Separate class loader is the thing I hate most in OSGI, I am
>>>>>>> afraid we are on the same way.
>>>>>>> Frankly speaking, I never see benefits of this *separate* model,
>>>>>>> our project(or most projects) is not like Chrome which has to
>>>>>>> create sandbox for extensions in order to avoid bad plugin screws
>>>>>>> up the whole browser(however, I still see bad plugins screw up my
>>>>>>> Chrome well).
>>>>>>> Projects like CloudStack using plugin to decouple architecture
>>>>>>> should not introduce many isolations to plugin writer, the point
>>>>>>> preventing wrong use of some components is not making much sense
>>>>>>> to me. If a plugin not following guide(if we have
>>>>>>> it) we should kick it out, instead of making obstacles for 99%
>>>>>>> good
>>>>> people.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>>>>>>>> To: dev@cloudstack.apache.org
>>>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>>>>
>>>>>>>> Right, component isn't a thing. I probably shouldn't use that
>>> term.
>>>>>>>> I
>>>>>>> want to
>>>>>>>> standarize on the naming convention of plugin, module, and
>>> extension.
>>>>>>> It is
>>>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>>>>>>>> better job
>>>>>>> here. So a
>>>>>>>> plugin is basically a jar. A jar contains multiple modules. A
>>>>>>>> modules
>>>>>>> ends up
>>>>>>>> being a spring application context that composes multiple
>>>>>>>> configuration
>>>>>>> files.
>>>>>>>> Modules are assembled into a hierarchy at runtime. Extensions
>>>>>>>> are implementations of interfaces that exist in a module. A
>>>>>>>> maven project produces a jar, so a plugin ends up being a maven
>>>>>>>> project
>>>>> also.
>>>>>>>>
>>>>>>>> So currently today we don't have a strong definition of Plugin
>>>>>>>> and I
>>>>>>> hope to
>>>>>>>> address that.
>>>>>>>>
>>>>>>>> Darren
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>>>>>>>> ins%2C+Modules%2C+and+Extensions
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>>>>>>>> <da...@gmail.com>wrote:
>>>>>>>>
>>>>>>>>> sounds great Darren,
>>>>>>>>>
>>>>>>>>> By component, you mean maven project or some larger chunk like
>>>>>>>>> distribution package? (did i miss this definition somewhere or
>>>>>>>>> do we define the components now?)
>>>>>>>>>
>>>>>>>>> regards,
>>>>>>>>> Daan
>>>>>>>>>
>>>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>>>>>>>>> <da...@gmail.com> wrote:
>>>>>>>>>> Currently ACS code is fairly modular in that you can add
>>>>>>>>>> plug-ins to ACS
>>>>>>>>> to
>>>>>>>>>> extend most functionality. Unfortunately ACS is not
>>>>>>>>>> packaged in a
>>>>>>>>> modular
>>>>>>>>>> way. It is still delivered essentially as one large unit.
>>>>>>>>>> There are
>>>>>>>>> many
>>>>>>>>>> reason for this but one large barrier to modularizing ACS is
>>>>>>>>>> that the Spring configuration is managed as one large unit.
>>>>>>>>>>
>>>>>>>>>> I propose that we break apart the Spring XML configuration
>>>>>>>>>> such that each component contributes its own configuration.
>>>>>>>>>> Additionally each component will be loaded into its own
>>>>>>>>>> Spring ApplicationContext such that its beans will not
>>>>>>>>>> conflict with the wiring of other beans in ACS. This change
>>>>>>>>> will
>>>>>>>>>> lay the foundation for a richer plugin model and
>>>>>>>>>> additionally a more distributed architecture.
>>>>>>>>>>
>>>>>>>>>> The technical details for this proposal can be found on the
>>>>>>>>>> wiki
>>>>> [1].
>>>>>>>>>>
>>>>>>>>>> Darren
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>>>>>>>>> ize+
>>>>>>>>> Spri
>>>>>>>>> ng
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: [PROPOSAL] Modularize Spring
Posted by "SuichII, Christopher" <Ch...@netapp.com>.
I've been following this conversation somewhat and would like to throw in some background as a plugin writer.
One thing that concerns me in the current plugin model is the number of XML/text files that need to be edited to deploy my plugin.
-applicationContext must be edited to add our PluginMangerImpl.
-commands.properties file must be edited to included the permissions for the APIs we contributed.
-componentContext.xml & nonossComponentContext.xml must be edited to add our Storage Subsystem Provider API.
-log4j-cloud.xml must be edited to ensure that our logger is enabled and logging to our necessary default level.
I know our situation is a bit different than the current plug in model, but I think it is something we, as a community, are going to begin seeing more of. For a variety of reasons that I won't get in to right now, our plugin will be closed source and kept separate from the ACS source tree. We want our users to be able to simply drop in our jar file to the CS directory or run and installer and have it picked up by the MS upon a restart.
It sounds like what you are proposing here would be very beneficial to plugins that are targeting a deployment model like this.
Is this something you're looking/hoping/would like to solve, Darren?
-Chris
--
Chris Suich
chris.suich@netapp.com
NetApp Software Engineer
Data Center Platforms – Cloud Solutions
Citrix, Cisco & Red Hat
On Sep 18, 2013, at 6:44 PM, Darren Shepherd <da...@gmail.com> wrote:
> I'm not a committer
>
>
> On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang <Fr...@citrix.com> wrote:
>
>> Well. The codes explain more than words.
>> It seems the only extra work is adding a property file that specifies
>> parent context and current context name, it's not much complex.
>> BTW: any reason for working on repo outside ACS?
>>
>>> -----Original Message-----
>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>> Sent: Wednesday, September 18, 2013 2:43 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>
>>> If you want to see this all working you can just fetch the "no-at-db4"
>>> branch at https://github.com/ibuildthecloud/cloudstack.git
>>>
>>> Plugin composes multiple modules. If modules are siblings they can't
>> inject
>>> from each other. But a plugin can augment another module if it chooses
>> to.
>>> The reality is that the core cloudstack is a tangled mess of
>> dependencies such
>>> that most of the core code can't be modularized as it stands. So there
>> exists a
>>> context towards the top of the hierarchy called "core" that a lot of jars
>>> contribute to it. Here is the full hierarchy right now. I'll probably
>> rename a
>>> bunch of things, but this gives you an idea.
>>>
>>> bootstrap
>>> system
>>> core
>>> allocator
>>> allocator-server
>>> planner
>>> api-planner
>>> baremetal-planner
>>> explicit-dedication
>>> host-anti-affinity
>>> implicit-dedication-planner
>>> server-planner
>>> user-concentrated-pod-planner
>>> random-allocator
>>> api
>>> acl-static-role-based
>>> rate-limit
>>> server-api
>>> user-authenticator-ldap
>>> user-authenticator-md5
>>> user-authenticator-plaintext
>>> user-authenticator-sha256salted
>>> backend
>>> alert-adapter-server-backend
>>> compute
>>> alert-adapter-server-compute
>>> baremetal-compute
>>> fencer-server
>>> investigator-server
>>> kvm-compute
>>> ovm-compute
>>> server-compute
>>> xenserver-compute
>>> network
>>> baremetal-network
>>> elb
>>> midonet
>>> nvp
>>> ovs
>>> server-network
>>> ssp
>>> storage
>>> alert-adapter-server-storage
>>> allocator-storage
>>> baremetal-storage
>>> secondary-storage
>>> server-storage
>>> storage-image-default
>>> storage-image-s3
>>> storage-image-swift
>>> storage-volume-default
>>> storage-volume-solidfire
>>> template-adapter-server
>>> discoverer
>>> baremetal-discoverer
>>> discoverer-server
>>> ovm-discoverer
>>> xenserver-discoverer
>>>
>>>
>>>
>>> If you look at the baremetal hypervisor plugin that is pretty cross
>> cutting to
>>> most of ACS. So the modules it contributes to are as follows
>>>
>>> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
>>> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
>>> resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
>>> resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
>>> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
>>> resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>>>
>>> So it creates child contextes of storage, compute, network, planner, and
>>> discoverer to add its extensions where it needs to be. Additionally,
>> you'll notice,
>>> it adds some beans to the core context from the file resources/META-
>>> INF/cloudstack/core/spring-baremetal-core-context.xml. This is because
>> it has
>>> some manager class that is used by multiple contexts.
>>>
>>> Frank, I understand the scare that we are going too complex, but do you
>> have
>>> some other suggestion? I don't like the idea of one gigantic spring
>> context. So I
>>> feel I am making it as simple as I can while maintaining some order.
>> People
>>> just need to create one or more spring xml files and a properties files
>> that says
>>> where to put it in the hierarchy.
>>>
>>> Additionally, by putting forcing people to put beans in certains modules
>> it
>>> forces them to think about what is the role of the code. For example,
>> today in
>>> ACS the *ManagerImpl classes are a huge mess. They implement too many
>>> interfaces and the code crosses to many architectural boundaries. Its
>> about
>>> time we start splitting things up to be more maintainable.
>>>
>>> If you have some time, please check out what I have on github.
>>>
>>> Darren
>>>
>>>
>>> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <Fr...@citrix.com>
>>> wrote:
>>>
>>>> I am not against boundary, I am just against making things unnecessary
>>>> complex to enable boundary.
>>>> If we are going this way, I hope we can make it as much as transparent
>>>> to developers. That means, as a developer, all a plugin I need to do
>>>> is 1) provide my separate spring xml 2) inject beans I want (valid
>>>> beans) in my bean and code business logic 3) compile to a jar and put
>>>> to some place designated by CloudStack. That's it.
>>>>
>>>> I raise this topic because I have seen some projects to create
>>>> boundary making things horrible complex. And sometimes developers are
>>>> hurt by wrong boundaries, as a result, to overcome these limitations
>>>> people write lots of ugly code which makes thing even worse.
>>>>
>>>> However, I am still worry about if we can make things so simpler.
>>>> For example, we may have an orchestration context that contains major
>>>> beans needed by almost every plugin, this context can be easily set
>>>> as parent context for all plugin contexts when bootstrap. However, if
>>>> a plugin A needs to access some bean defined in plugin B, given they
>>>> are sibling, how plugin framework resolves the dependency ?
>>>>
>>>>> -----Original Message-----
>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>> Sent: Wednesday, September 18, 2013 11:53 AM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>
>>>>> I'm not for OSGi either, but contexts are quite useful and will lead
>>>>> to
>>>> better
>>>>> things. First off, we don't want one gigantic spring XML config
>>>>> file
>>>> like we have
>>>>> today. I think we can all agree on that. So each plugin will have
>>>>> to
>>>> supply its
>>>>> own XML. So the obstacles you mention, will largely be just that
>>>>> for
>>>> people.
>>>>>
>>>>> With Spring it is really simple to just inject dependencies and
>>>>> cross architectural boundaries. Its currently everywhere in ACS.
>>>>> You can't
>>>> just say
>>>>> we should review code and make sure nobody doesn't do bad things. A
>>>> little bit
>>>>> of framework to enforce separation is a good thing. But I'm
>>>>> guessing
>>>> you will
>>>>> disagree with me there.
>>>>>
>>>>> Here are some random points on why contexts are good. Say I want to
>>>>> use Spring AOP or Spring TX in my plugin. With your own context you
>>>>> can
>>>> ensure
>>>>> that you won't screw with anybody else code by accidentally having
>>>>> you pointcut match their bean. You don't have to worry about bean
>>>>> name
>>>> conflicts.
>>>>> If two config files specify bean X, Spring will gladly just use the
>>>>> last
>>>> one. I've
>>>>> already found multiply defined beans in ACS, but things still just
>>>> happen to work.
>>>>> Having multiple contexts also defines initialization order. We can
>>>> ensure that
>>>>> the framework is loaded and ready before child context are loaded
>>>>> and
>>>> started.
>>>>> (we kind of do this today with ComponentLifeCycle, but its a hack in
>>>>> my
>>>> mind).
>>>>> Additionally, when things start you will know, loading context
>>>>> "crapping
>>>> plug X".
>>>>> If spring fails to initialize, the issue it there. Today, if spring
>>>> fails to start, it
>>>>> could be one of over 500 beans causing the weird problem. The list
>>>>> goes
>>>> on
>>>>> and on.
>>>>>
>>>>> Finally, this is the big one and why I really want contexts. I have
>>>> some notes on
>>>>> the wiki [1] that you might want to read through. Basically I want
>>>>> to
>>>> get to a
>>>>> more flexible deployment model that allows both a single monolithic
>>>>> JVM
>>>> as
>>>>> today and also a fully distributed system. Having contexts in
>>>>> hierarchy
>>>> will
>>>>> enable me to accomplish this. By selecting which contexts are
>>>>> loaded at runtime will determine what role the JVM takes on. The
>>>>> contexts help
>>>> people
>>>>> better understand how the distributed architecture will work too,
>>>>> when
>>>> we get
>>>>> there.
>>>>>
>>>>> Frank, trust me, I hate complex things. I don't want OSGi,
>>>>> classloader
>>>> magic,
>>>>> etc. But I do like organization and a little bit of framework so
>>>>> that
>>>> people don't
>>>>> accidentally shoot themselves in the foot. I personally like
>>>>> knowing
>>>> that I
>>>>> couldn't have screwed something up, because the framework won't even
>>>> allow
>>>>> it. If we separate everything as I want today, and then tomorrow we
>>>>> say
>>>> this is
>>>>> way too much overhead, moving to a flat context is simple. Don't
>>>>> think
>>>> we are
>>>>> on some slippery slope to classloaders and dependency hell.
>>>>>
>>>>> Darren
>>>>>
>>>>> [1]
>>>>>
>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
>>>> her
>>>>> e...#Nothingtoseehere...-DeploymentModels
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
>>>>> <Fr...@citrix.com>wrote:
>>>>>
>>>>>> What's the point in using separate spring context per plugin?
>>>>>> Separate class loader is the thing I hate most in OSGI, I am
>>>>>> afraid we are on the same way.
>>>>>> Frankly speaking, I never see benefits of this *separate* model,
>>>>>> our project(or most projects) is not like Chrome which has to
>>>>>> create sandbox for extensions in order to avoid bad plugin screws
>>>>>> up the whole browser(however, I still see bad plugins screw up my
>>>>>> Chrome well).
>>>>>> Projects like CloudStack using plugin to decouple architecture
>>>>>> should not introduce many isolations to plugin writer, the point
>>>>>> preventing wrong use of some components is not making much sense
>>>>>> to me. If a plugin not following guide(if we have
>>>>>> it) we should kick it out, instead of making obstacles for 99%
>>>>>> good
>>>> people.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
>>>>>>> Sent: Wednesday, September 18, 2013 10:33 AM
>>>>>>> To: dev@cloudstack.apache.org
>>>>>>> Subject: Re: [PROPOSAL] Modularize Spring
>>>>>>>
>>>>>>> Right, component isn't a thing. I probably shouldn't use that
>> term.
>>>>>>> I
>>>>>> want to
>>>>>>> standarize on the naming convention of plugin, module, and
>> extension.
>>>>>> It is
>>>>>>> explained a bit on the wiki [1] but I'll try to do a little
>>>>>>> better job
>>>>>> here. So a
>>>>>>> plugin is basically a jar. A jar contains multiple modules. A
>>>>>>> modules
>>>>>> ends up
>>>>>>> being a spring application context that composes multiple
>>>>>>> configuration
>>>>>> files.
>>>>>>> Modules are assembled into a hierarchy at runtime. Extensions
>>>>>>> are implementations of interfaces that exist in a module. A
>>>>>>> maven project produces a jar, so a plugin ends up being a maven
>>>>>>> project
>>>> also.
>>>>>>>
>>>>>>> So currently today we don't have a strong definition of Plugin
>>>>>>> and I
>>>>>> hope to
>>>>>>> address that.
>>>>>>>
>>>>>>> Darren
>>>>>>>
>>>>>>> [1]
>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
>>>>>>> ins%2C+Modules%2C+and+Extensions
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
>>>>>>> <da...@gmail.com>wrote:
>>>>>>>
>>>>>>>> sounds great Darren,
>>>>>>>>
>>>>>>>> By component, you mean maven project or some larger chunk like
>>>>>>>> distribution package? (did i miss this definition somewhere or
>>>>>>>> do we define the components now?)
>>>>>>>>
>>>>>>>> regards,
>>>>>>>> Daan
>>>>>>>>
>>>>>>>> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
>>>>>>>> <da...@gmail.com> wrote:
>>>>>>>>> Currently ACS code is fairly modular in that you can add
>>>>>>>>> plug-ins to ACS
>>>>>>>> to
>>>>>>>>> extend most functionality. Unfortunately ACS is not
>>>>>>>>> packaged in a
>>>>>>>> modular
>>>>>>>>> way. It is still delivered essentially as one large unit.
>>>>>>>>> There are
>>>>>>>> many
>>>>>>>>> reason for this but one large barrier to modularizing ACS is
>>>>>>>>> that the Spring configuration is managed as one large unit.
>>>>>>>>>
>>>>>>>>> I propose that we break apart the Spring XML configuration
>>>>>>>>> such that each component contributes its own configuration.
>>>>>>>>> Additionally each component will be loaded into its own
>>>>>>>>> Spring ApplicationContext such that its beans will not
>>>>>>>>> conflict with the wiring of other beans in ACS. This change
>>>>>>>> will
>>>>>>>>> lay the foundation for a richer plugin model and
>>>>>>>>> additionally a more distributed architecture.
>>>>>>>>>
>>>>>>>>> The technical details for this proposal can be found on the
>>>>>>>>> wiki
>>>> [1].
>>>>>>>>>
>>>>>>>>> Darren
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
>>>>>>>> ize+
>>>>>>>> Spri
>>>>>>>> ng
>>>>>>>>
>>>>>>
>>>>
>>
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
I'm not a committer
On Wed, Sep 18, 2013 at 3:24 PM, Frank Zhang <Fr...@citrix.com> wrote:
> Well. The codes explain more than words.
> It seems the only extra work is adding a property file that specifies
> parent context and current context name, it's not much complex.
> BTW: any reason for working on repo outside ACS?
>
> > -----Original Message-----
> > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > Sent: Wednesday, September 18, 2013 2:43 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] Modularize Spring
> >
> > If you want to see this all working you can just fetch the "no-at-db4"
> > branch at https://github.com/ibuildthecloud/cloudstack.git
> >
> > Plugin composes multiple modules. If modules are siblings they can't
> inject
> > from each other. But a plugin can augment another module if it chooses
> to.
> > The reality is that the core cloudstack is a tangled mess of
> dependencies such
> > that most of the core code can't be modularized as it stands. So there
> exists a
> > context towards the top of the hierarchy called "core" that a lot of jars
> > contribute to it. Here is the full hierarchy right now. I'll probably
> rename a
> > bunch of things, but this gives you an idea.
> >
> > bootstrap
> > system
> > core
> > allocator
> > allocator-server
> > planner
> > api-planner
> > baremetal-planner
> > explicit-dedication
> > host-anti-affinity
> > implicit-dedication-planner
> > server-planner
> > user-concentrated-pod-planner
> > random-allocator
> > api
> > acl-static-role-based
> > rate-limit
> > server-api
> > user-authenticator-ldap
> > user-authenticator-md5
> > user-authenticator-plaintext
> > user-authenticator-sha256salted
> > backend
> > alert-adapter-server-backend
> > compute
> > alert-adapter-server-compute
> > baremetal-compute
> > fencer-server
> > investigator-server
> > kvm-compute
> > ovm-compute
> > server-compute
> > xenserver-compute
> > network
> > baremetal-network
> > elb
> > midonet
> > nvp
> > ovs
> > server-network
> > ssp
> > storage
> > alert-adapter-server-storage
> > allocator-storage
> > baremetal-storage
> > secondary-storage
> > server-storage
> > storage-image-default
> > storage-image-s3
> > storage-image-swift
> > storage-volume-default
> > storage-volume-solidfire
> > template-adapter-server
> > discoverer
> > baremetal-discoverer
> > discoverer-server
> > ovm-discoverer
> > xenserver-discoverer
> >
> >
> >
> > If you look at the baremetal hypervisor plugin that is pretty cross
> cutting to
> > most of ACS. So the modules it contributes to are as follows
> >
> > resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
> > resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
> > resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
> > resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
> > resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
> > resources/META-INF/cloudstack/baremetal-network/spring-context.xml
> >
> > So it creates child contextes of storage, compute, network, planner, and
> > discoverer to add its extensions where it needs to be. Additionally,
> you'll notice,
> > it adds some beans to the core context from the file resources/META-
> > INF/cloudstack/core/spring-baremetal-core-context.xml. This is because
> it has
> > some manager class that is used by multiple contexts.
> >
> > Frank, I understand the scare that we are going too complex, but do you
> have
> > some other suggestion? I don't like the idea of one gigantic spring
> context. So I
> > feel I am making it as simple as I can while maintaining some order.
> People
> > just need to create one or more spring xml files and a properties files
> that says
> > where to put it in the hierarchy.
> >
> > Additionally, by putting forcing people to put beans in certains modules
> it
> > forces them to think about what is the role of the code. For example,
> today in
> > ACS the *ManagerImpl classes are a huge mess. They implement too many
> > interfaces and the code crosses to many architectural boundaries. Its
> about
> > time we start splitting things up to be more maintainable.
> >
> > If you have some time, please check out what I have on github.
> >
> > Darren
> >
> >
> > On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <Fr...@citrix.com>
> > wrote:
> >
> > > I am not against boundary, I am just against making things unnecessary
> > > complex to enable boundary.
> > > If we are going this way, I hope we can make it as much as transparent
> > > to developers. That means, as a developer, all a plugin I need to do
> > > is 1) provide my separate spring xml 2) inject beans I want (valid
> > > beans) in my bean and code business logic 3) compile to a jar and put
> > > to some place designated by CloudStack. That's it.
> > >
> > > I raise this topic because I have seen some projects to create
> > > boundary making things horrible complex. And sometimes developers are
> > > hurt by wrong boundaries, as a result, to overcome these limitations
> > > people write lots of ugly code which makes thing even worse.
> > >
> > > However, I am still worry about if we can make things so simpler.
> > > For example, we may have an orchestration context that contains major
> > > beans needed by almost every plugin, this context can be easily set
> > > as parent context for all plugin contexts when bootstrap. However, if
> > > a plugin A needs to access some bean defined in plugin B, given they
> > > are sibling, how plugin framework resolves the dependency ?
> > >
> > > > -----Original Message-----
> > > > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > > > Sent: Wednesday, September 18, 2013 11:53 AM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [PROPOSAL] Modularize Spring
> > > >
> > > > I'm not for OSGi either, but contexts are quite useful and will lead
> > > > to
> > > better
> > > > things. First off, we don't want one gigantic spring XML config
> > > > file
> > > like we have
> > > > today. I think we can all agree on that. So each plugin will have
> > > > to
> > > supply its
> > > > own XML. So the obstacles you mention, will largely be just that
> > > > for
> > > people.
> > > >
> > > > With Spring it is really simple to just inject dependencies and
> > > > cross architectural boundaries. Its currently everywhere in ACS.
> > > > You can't
> > > just say
> > > > we should review code and make sure nobody doesn't do bad things. A
> > > little bit
> > > > of framework to enforce separation is a good thing. But I'm
> > > > guessing
> > > you will
> > > > disagree with me there.
> > > >
> > > > Here are some random points on why contexts are good. Say I want to
> > > > use Spring AOP or Spring TX in my plugin. With your own context you
> > > > can
> > > ensure
> > > > that you won't screw with anybody else code by accidentally having
> > > > you pointcut match their bean. You don't have to worry about bean
> > > > name
> > > conflicts.
> > > > If two config files specify bean X, Spring will gladly just use the
> > > > last
> > > one. I've
> > > > already found multiply defined beans in ACS, but things still just
> > > happen to work.
> > > > Having multiple contexts also defines initialization order. We can
> > > ensure that
> > > > the framework is loaded and ready before child context are loaded
> > > > and
> > > started.
> > > > (we kind of do this today with ComponentLifeCycle, but its a hack in
> > > > my
> > > mind).
> > > > Additionally, when things start you will know, loading context
> > > > "crapping
> > > plug X".
> > > > If spring fails to initialize, the issue it there. Today, if spring
> > > fails to start, it
> > > > could be one of over 500 beans causing the weird problem. The list
> > > > goes
> > > on
> > > > and on.
> > > >
> > > > Finally, this is the big one and why I really want contexts. I have
> > > some notes on
> > > > the wiki [1] that you might want to read through. Basically I want
> > > > to
> > > get to a
> > > > more flexible deployment model that allows both a single monolithic
> > > > JVM
> > > as
> > > > today and also a fully distributed system. Having contexts in
> > > > hierarchy
> > > will
> > > > enable me to accomplish this. By selecting which contexts are
> > > > loaded at runtime will determine what role the JVM takes on. The
> > > > contexts help
> > > people
> > > > better understand how the distributed architecture will work too,
> > > > when
> > > we get
> > > > there.
> > > >
> > > > Frank, trust me, I hate complex things. I don't want OSGi,
> > > > classloader
> > > magic,
> > > > etc. But I do like organization and a little bit of framework so
> > > > that
> > > people don't
> > > > accidentally shoot themselves in the foot. I personally like
> > > > knowing
> > > that I
> > > > couldn't have screwed something up, because the framework won't even
> > > allow
> > > > it. If we separate everything as I want today, and then tomorrow we
> > > > say
> > > this is
> > > > way too much overhead, moving to a flat context is simple. Don't
> > > > think
> > > we are
> > > > on some slippery slope to classloaders and dependency hell.
> > > >
> > > > Darren
> > > >
> > > > [1]
> > > >
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
> > > her
> > > > e...#Nothingtoseehere...-DeploymentModels
> > > >
> > > >
> > > >
> > > > On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
> > > > <Fr...@citrix.com>wrote:
> > > >
> > > > > What's the point in using separate spring context per plugin?
> > > > > Separate class loader is the thing I hate most in OSGI, I am
> > > > > afraid we are on the same way.
> > > > > Frankly speaking, I never see benefits of this *separate* model,
> > > > > our project(or most projects) is not like Chrome which has to
> > > > > create sandbox for extensions in order to avoid bad plugin screws
> > > > > up the whole browser(however, I still see bad plugins screw up my
> > > > > Chrome well).
> > > > > Projects like CloudStack using plugin to decouple architecture
> > > > > should not introduce many isolations to plugin writer, the point
> > > > > preventing wrong use of some components is not making much sense
> > > > > to me. If a plugin not following guide(if we have
> > > > > it) we should kick it out, instead of making obstacles for 99%
> > > > > good
> > > people.
> > > > >
> > > > >
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > > > > > Sent: Wednesday, September 18, 2013 10:33 AM
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: Re: [PROPOSAL] Modularize Spring
> > > > > >
> > > > > > Right, component isn't a thing. I probably shouldn't use that
> term.
> > > > > > I
> > > > > want to
> > > > > > standarize on the naming convention of plugin, module, and
> extension.
> > > > > It is
> > > > > > explained a bit on the wiki [1] but I'll try to do a little
> > > > > > better job
> > > > > here. So a
> > > > > > plugin is basically a jar. A jar contains multiple modules. A
> > > > > > modules
> > > > > ends up
> > > > > > being a spring application context that composes multiple
> > > > > > configuration
> > > > > files.
> > > > > > Modules are assembled into a hierarchy at runtime. Extensions
> > > > > > are implementations of interfaces that exist in a module. A
> > > > > > maven project produces a jar, so a plugin ends up being a maven
> > > > > > project
> > > also.
> > > > > >
> > > > > > So currently today we don't have a strong definition of Plugin
> > > > > > and I
> > > > > hope to
> > > > > > address that.
> > > > > >
> > > > > > Darren
> > > > > >
> > > > > > [1]
> > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> > > > > > ins%2C+Modules%2C+and+Extensions
> > > > > >
> > > > > >
> > > > > > On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
> > > > > > <da...@gmail.com>wrote:
> > > > > >
> > > > > > > sounds great Darren,
> > > > > > >
> > > > > > > By component, you mean maven project or some larger chunk like
> > > > > > > distribution package? (did i miss this definition somewhere or
> > > > > > > do we define the components now?)
> > > > > > >
> > > > > > > regards,
> > > > > > > Daan
> > > > > > >
> > > > > > > On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> > > > > > > <da...@gmail.com> wrote:
> > > > > > > > Currently ACS code is fairly modular in that you can add
> > > > > > > > plug-ins to ACS
> > > > > > > to
> > > > > > > > extend most functionality. Unfortunately ACS is not
> > > > > > > > packaged in a
> > > > > > > modular
> > > > > > > > way. It is still delivered essentially as one large unit.
> > > > > > > > There are
> > > > > > > many
> > > > > > > > reason for this but one large barrier to modularizing ACS is
> > > > > > > > that the Spring configuration is managed as one large unit.
> > > > > > > >
> > > > > > > > I propose that we break apart the Spring XML configuration
> > > > > > > > such that each component contributes its own configuration.
> > > > > > > > Additionally each component will be loaded into its own
> > > > > > > > Spring ApplicationContext such that its beans will not
> > > > > > > > conflict with the wiring of other beans in ACS. This change
> > > > > > > will
> > > > > > > > lay the foundation for a richer plugin model and
> > > > > > > > additionally a more distributed architecture.
> > > > > > > >
> > > > > > > > The technical details for this proposal can be found on the
> > > > > > > > wiki
> > > [1].
> > > > > > > >
> > > > > > > > Darren
> > > > > > > >
> > > > > > > > [1]
> > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
> > > > > > > ize+
> > > > > > > Spri
> > > > > > > ng
> > > > > > >
> > > > >
> > >
>
RE: [PROPOSAL] Modularize Spring
Posted by Frank Zhang <Fr...@citrix.com>.
Well. The codes explain more than words.
It seems the only extra work is adding a property file that specifies parent context and current context name, it's not much complex.
BTW: any reason for working on repo outside ACS?
> -----Original Message-----
> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> Sent: Wednesday, September 18, 2013 2:43 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Modularize Spring
>
> If you want to see this all working you can just fetch the "no-at-db4"
> branch at https://github.com/ibuildthecloud/cloudstack.git
>
> Plugin composes multiple modules. If modules are siblings they can't inject
> from each other. But a plugin can augment another module if it chooses to.
> The reality is that the core cloudstack is a tangled mess of dependencies such
> that most of the core code can't be modularized as it stands. So there exists a
> context towards the top of the hierarchy called "core" that a lot of jars
> contribute to it. Here is the full hierarchy right now. I'll probably rename a
> bunch of things, but this gives you an idea.
>
> bootstrap
> system
> core
> allocator
> allocator-server
> planner
> api-planner
> baremetal-planner
> explicit-dedication
> host-anti-affinity
> implicit-dedication-planner
> server-planner
> user-concentrated-pod-planner
> random-allocator
> api
> acl-static-role-based
> rate-limit
> server-api
> user-authenticator-ldap
> user-authenticator-md5
> user-authenticator-plaintext
> user-authenticator-sha256salted
> backend
> alert-adapter-server-backend
> compute
> alert-adapter-server-compute
> baremetal-compute
> fencer-server
> investigator-server
> kvm-compute
> ovm-compute
> server-compute
> xenserver-compute
> network
> baremetal-network
> elb
> midonet
> nvp
> ovs
> server-network
> ssp
> storage
> alert-adapter-server-storage
> allocator-storage
> baremetal-storage
> secondary-storage
> server-storage
> storage-image-default
> storage-image-s3
> storage-image-swift
> storage-volume-default
> storage-volume-solidfire
> template-adapter-server
> discoverer
> baremetal-discoverer
> discoverer-server
> ovm-discoverer
> xenserver-discoverer
>
>
>
> If you look at the baremetal hypervisor plugin that is pretty cross cutting to
> most of ACS. So the modules it contributes to are as follows
>
> resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
> resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
> resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
> resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
> resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
> resources/META-INF/cloudstack/baremetal-network/spring-context.xml
>
> So it creates child contextes of storage, compute, network, planner, and
> discoverer to add its extensions where it needs to be. Additionally, you'll notice,
> it adds some beans to the core context from the file resources/META-
> INF/cloudstack/core/spring-baremetal-core-context.xml. This is because it has
> some manager class that is used by multiple contexts.
>
> Frank, I understand the scare that we are going too complex, but do you have
> some other suggestion? I don't like the idea of one gigantic spring context. So I
> feel I am making it as simple as I can while maintaining some order. People
> just need to create one or more spring xml files and a properties files that says
> where to put it in the hierarchy.
>
> Additionally, by putting forcing people to put beans in certains modules it
> forces them to think about what is the role of the code. For example, today in
> ACS the *ManagerImpl classes are a huge mess. They implement too many
> interfaces and the code crosses to many architectural boundaries. Its about
> time we start splitting things up to be more maintainable.
>
> If you have some time, please check out what I have on github.
>
> Darren
>
>
> On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <Fr...@citrix.com>
> wrote:
>
> > I am not against boundary, I am just against making things unnecessary
> > complex to enable boundary.
> > If we are going this way, I hope we can make it as much as transparent
> > to developers. That means, as a developer, all a plugin I need to do
> > is 1) provide my separate spring xml 2) inject beans I want (valid
> > beans) in my bean and code business logic 3) compile to a jar and put
> > to some place designated by CloudStack. That's it.
> >
> > I raise this topic because I have seen some projects to create
> > boundary making things horrible complex. And sometimes developers are
> > hurt by wrong boundaries, as a result, to overcome these limitations
> > people write lots of ugly code which makes thing even worse.
> >
> > However, I am still worry about if we can make things so simpler.
> > For example, we may have an orchestration context that contains major
> > beans needed by almost every plugin, this context can be easily set
> > as parent context for all plugin contexts when bootstrap. However, if
> > a plugin A needs to access some bean defined in plugin B, given they
> > are sibling, how plugin framework resolves the dependency ?
> >
> > > -----Original Message-----
> > > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > > Sent: Wednesday, September 18, 2013 11:53 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [PROPOSAL] Modularize Spring
> > >
> > > I'm not for OSGi either, but contexts are quite useful and will lead
> > > to
> > better
> > > things. First off, we don't want one gigantic spring XML config
> > > file
> > like we have
> > > today. I think we can all agree on that. So each plugin will have
> > > to
> > supply its
> > > own XML. So the obstacles you mention, will largely be just that
> > > for
> > people.
> > >
> > > With Spring it is really simple to just inject dependencies and
> > > cross architectural boundaries. Its currently everywhere in ACS.
> > > You can't
> > just say
> > > we should review code and make sure nobody doesn't do bad things. A
> > little bit
> > > of framework to enforce separation is a good thing. But I'm
> > > guessing
> > you will
> > > disagree with me there.
> > >
> > > Here are some random points on why contexts are good. Say I want to
> > > use Spring AOP or Spring TX in my plugin. With your own context you
> > > can
> > ensure
> > > that you won't screw with anybody else code by accidentally having
> > > you pointcut match their bean. You don't have to worry about bean
> > > name
> > conflicts.
> > > If two config files specify bean X, Spring will gladly just use the
> > > last
> > one. I've
> > > already found multiply defined beans in ACS, but things still just
> > happen to work.
> > > Having multiple contexts also defines initialization order. We can
> > ensure that
> > > the framework is loaded and ready before child context are loaded
> > > and
> > started.
> > > (we kind of do this today with ComponentLifeCycle, but its a hack in
> > > my
> > mind).
> > > Additionally, when things start you will know, loading context
> > > "crapping
> > plug X".
> > > If spring fails to initialize, the issue it there. Today, if spring
> > fails to start, it
> > > could be one of over 500 beans causing the weird problem. The list
> > > goes
> > on
> > > and on.
> > >
> > > Finally, this is the big one and why I really want contexts. I have
> > some notes on
> > > the wiki [1] that you might want to read through. Basically I want
> > > to
> > get to a
> > > more flexible deployment model that allows both a single monolithic
> > > JVM
> > as
> > > today and also a fully distributed system. Having contexts in
> > > hierarchy
> > will
> > > enable me to accomplish this. By selecting which contexts are
> > > loaded at runtime will determine what role the JVM takes on. The
> > > contexts help
> > people
> > > better understand how the distributed architecture will work too,
> > > when
> > we get
> > > there.
> > >
> > > Frank, trust me, I hate complex things. I don't want OSGi,
> > > classloader
> > magic,
> > > etc. But I do like organization and a little bit of framework so
> > > that
> > people don't
> > > accidentally shoot themselves in the foot. I personally like
> > > knowing
> > that I
> > > couldn't have screwed something up, because the framework won't even
> > allow
> > > it. If we separate everything as I want today, and then tomorrow we
> > > say
> > this is
> > > way too much overhead, moving to a flat context is simple. Don't
> > > think
> > we are
> > > on some slippery slope to classloaders and dependency hell.
> > >
> > > Darren
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+
> > her
> > > e...#Nothingtoseehere...-DeploymentModels
> > >
> > >
> > >
> > > On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
> > > <Fr...@citrix.com>wrote:
> > >
> > > > What's the point in using separate spring context per plugin?
> > > > Separate class loader is the thing I hate most in OSGI, I am
> > > > afraid we are on the same way.
> > > > Frankly speaking, I never see benefits of this *separate* model,
> > > > our project(or most projects) is not like Chrome which has to
> > > > create sandbox for extensions in order to avoid bad plugin screws
> > > > up the whole browser(however, I still see bad plugins screw up my
> > > > Chrome well).
> > > > Projects like CloudStack using plugin to decouple architecture
> > > > should not introduce many isolations to plugin writer, the point
> > > > preventing wrong use of some components is not making much sense
> > > > to me. If a plugin not following guide(if we have
> > > > it) we should kick it out, instead of making obstacles for 99%
> > > > good
> > people.
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > > > > Sent: Wednesday, September 18, 2013 10:33 AM
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: [PROPOSAL] Modularize Spring
> > > > >
> > > > > Right, component isn't a thing. I probably shouldn't use that term.
> > > > > I
> > > > want to
> > > > > standarize on the naming convention of plugin, module, and extension.
> > > > It is
> > > > > explained a bit on the wiki [1] but I'll try to do a little
> > > > > better job
> > > > here. So a
> > > > > plugin is basically a jar. A jar contains multiple modules. A
> > > > > modules
> > > > ends up
> > > > > being a spring application context that composes multiple
> > > > > configuration
> > > > files.
> > > > > Modules are assembled into a hierarchy at runtime. Extensions
> > > > > are implementations of interfaces that exist in a module. A
> > > > > maven project produces a jar, so a plugin ends up being a maven
> > > > > project
> > also.
> > > > >
> > > > > So currently today we don't have a strong definition of Plugin
> > > > > and I
> > > > hope to
> > > > > address that.
> > > > >
> > > > > Darren
> > > > >
> > > > > [1]
> > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> > > > > ins%2C+Modules%2C+and+Extensions
> > > > >
> > > > >
> > > > > On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
> > > > > <da...@gmail.com>wrote:
> > > > >
> > > > > > sounds great Darren,
> > > > > >
> > > > > > By component, you mean maven project or some larger chunk like
> > > > > > distribution package? (did i miss this definition somewhere or
> > > > > > do we define the components now?)
> > > > > >
> > > > > > regards,
> > > > > > Daan
> > > > > >
> > > > > > On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> > > > > > <da...@gmail.com> wrote:
> > > > > > > Currently ACS code is fairly modular in that you can add
> > > > > > > plug-ins to ACS
> > > > > > to
> > > > > > > extend most functionality. Unfortunately ACS is not
> > > > > > > packaged in a
> > > > > > modular
> > > > > > > way. It is still delivered essentially as one large unit.
> > > > > > > There are
> > > > > > many
> > > > > > > reason for this but one large barrier to modularizing ACS is
> > > > > > > that the Spring configuration is managed as one large unit.
> > > > > > >
> > > > > > > I propose that we break apart the Spring XML configuration
> > > > > > > such that each component contributes its own configuration.
> > > > > > > Additionally each component will be loaded into its own
> > > > > > > Spring ApplicationContext such that its beans will not
> > > > > > > conflict with the wiring of other beans in ACS. This change
> > > > > > will
> > > > > > > lay the foundation for a richer plugin model and
> > > > > > > additionally a more distributed architecture.
> > > > > > >
> > > > > > > The technical details for this proposal can be found on the
> > > > > > > wiki
> > [1].
> > > > > > >
> > > > > > > Darren
> > > > > > >
> > > > > > > [1]
> > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modular
> > > > > > ize+
> > > > > > Spri
> > > > > > ng
> > > > > >
> > > >
> >
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
If you want to see this all working you can just fetch the "no-at-db4"
branch at https://github.com/ibuildthecloud/cloudstack.git
Plugin composes multiple modules. If modules are siblings they can't
inject from each other. But a plugin can augment another module if it
chooses to. The reality is that the core cloudstack is a tangled mess of
dependencies such that most of the core code can't be modularized as it
stands. So there exists a context towards the top of the hierarchy called
"core" that a lot of jars contribute to it. Here is the full hierarchy
right now. I'll probably rename a bunch of things, but this gives you an
idea.
bootstrap
system
core
allocator
allocator-server
planner
api-planner
baremetal-planner
explicit-dedication
host-anti-affinity
implicit-dedication-planner
server-planner
user-concentrated-pod-planner
random-allocator
api
acl-static-role-based
rate-limit
server-api
user-authenticator-ldap
user-authenticator-md5
user-authenticator-plaintext
user-authenticator-sha256salted
backend
alert-adapter-server-backend
compute
alert-adapter-server-compute
baremetal-compute
fencer-server
investigator-server
kvm-compute
ovm-compute
server-compute
xenserver-compute
network
baremetal-network
elb
midonet
nvp
ovs
server-network
ssp
storage
alert-adapter-server-storage
allocator-storage
baremetal-storage
secondary-storage
server-storage
storage-image-default
storage-image-s3
storage-image-swift
storage-volume-default
storage-volume-solidfire
template-adapter-server
discoverer
baremetal-discoverer
discoverer-server
ovm-discoverer
xenserver-discoverer
If you look at the baremetal hypervisor plugin that is pretty cross cutting
to most of ACS. So the modules it contributes to are as follows
resources/META-INF/cloudstack/baremetal-storage/spring-context.xml
resources/META-INF/cloudstack/baremetal-compute/spring-context.xml
resources/META-INF/cloudstack/baremetal-discoverer/spring-context.xml
resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml
resources/META-INF/cloudstack/baremetal-planner/spring-context.xml
resources/META-INF/cloudstack/baremetal-network/spring-context.xml
So it creates child contextes of storage, compute, network, planner, and
discoverer to add its extensions where it needs to be. Additionally,
you'll notice, it adds some beans to the core context from the file
resources/META-INF/cloudstack/core/spring-baremetal-core-context.xml. This
is because it has some manager class that is used by multiple contexts.
Frank, I understand the scare that we are going too complex, but do you
have some other suggestion? I don't like the idea of one gigantic spring
context. So I feel I am making it as simple as I can while maintaining
some order. People just need to create one or more spring xml files and a
properties files that says where to put it in the hierarchy.
Additionally, by putting forcing people to put beans in certains modules it
forces them to think about what is the role of the code. For example,
today in ACS the *ManagerImpl classes are a huge mess. They implement too
many interfaces and the code crosses to many architectural boundaries. Its
about time we start splitting things up to be more maintainable.
If you have some time, please check out what I have on github.
Darren
On Wed, Sep 18, 2013 at 1:56 PM, Frank Zhang <Fr...@citrix.com> wrote:
> I am not against boundary, I am just against making things unnecessary
> complex to enable boundary.
> If we are going this way, I hope we can make it as much as transparent to
> developers. That means, as
> a developer, all a plugin I need to do is 1) provide my separate spring
> xml 2) inject beans I want (valid beans)
> in my bean and code business logic 3) compile to a jar and put to some
> place designated by CloudStack. That's it.
>
> I raise this topic because I have seen some projects to create boundary
> making things horrible complex. And
> sometimes developers are hurt by wrong boundaries, as a result, to
> overcome these limitations people write
> lots of ugly code which makes thing even worse.
>
> However, I am still worry about if we can make things so simpler.
> For example, we may have an orchestration context that contains major
> beans needed by almost every
> plugin, this context can be easily set as parent context for all plugin
> contexts when bootstrap. However, if a plugin A needs
> to access some bean defined in plugin B, given they are sibling, how
> plugin framework resolves the dependency ?
>
> > -----Original Message-----
> > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > Sent: Wednesday, September 18, 2013 11:53 AM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] Modularize Spring
> >
> > I'm not for OSGi either, but contexts are quite useful and will lead to
> better
> > things. First off, we don't want one gigantic spring XML config file
> like we have
> > today. I think we can all agree on that. So each plugin will have to
> supply its
> > own XML. So the obstacles you mention, will largely be just that for
> people.
> >
> > With Spring it is really simple to just inject dependencies and cross
> > architectural boundaries. Its currently everywhere in ACS. You can't
> just say
> > we should review code and make sure nobody doesn't do bad things. A
> little bit
> > of framework to enforce separation is a good thing. But I'm guessing
> you will
> > disagree with me there.
> >
> > Here are some random points on why contexts are good. Say I want to use
> > Spring AOP or Spring TX in my plugin. With your own context you can
> ensure
> > that you won't screw with anybody else code by accidentally having you
> > pointcut match their bean. You don't have to worry about bean name
> conflicts.
> > If two config files specify bean X, Spring will gladly just use the last
> one. I've
> > already found multiply defined beans in ACS, but things still just
> happen to work.
> > Having multiple contexts also defines initialization order. We can
> ensure that
> > the framework is loaded and ready before child context are loaded and
> started.
> > (we kind of do this today with ComponentLifeCycle, but its a hack in my
> mind).
> > Additionally, when things start you will know, loading context "crapping
> plug X".
> > If spring fails to initialize, the issue it there. Today, if spring
> fails to start, it
> > could be one of over 500 beans causing the weird problem. The list goes
> on
> > and on.
> >
> > Finally, this is the big one and why I really want contexts. I have
> some notes on
> > the wiki [1] that you might want to read through. Basically I want to
> get to a
> > more flexible deployment model that allows both a single monolithic JVM
> as
> > today and also a fully distributed system. Having contexts in hierarchy
> will
> > enable me to accomplish this. By selecting which contexts are loaded at
> > runtime will determine what role the JVM takes on. The contexts help
> people
> > better understand how the distributed architecture will work too, when
> we get
> > there.
> >
> > Frank, trust me, I hate complex things. I don't want OSGi, classloader
> magic,
> > etc. But I do like organization and a little bit of framework so that
> people don't
> > accidentally shoot themselves in the foot. I personally like knowing
> that I
> > couldn't have screwed something up, because the framework won't even
> allow
> > it. If we separate everything as I want today, and then tomorrow we say
> this is
> > way too much overhead, moving to a flat context is simple. Don't think
> we are
> > on some slippery slope to classloaders and dependency hell.
> >
> > Darren
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+her
> > e...#Nothingtoseehere...-DeploymentModels
> >
> >
> >
> > On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
> > <Fr...@citrix.com>wrote:
> >
> > > What's the point in using separate spring context per plugin?
> > > Separate class loader is the thing I hate most in OSGI, I am afraid we
> > > are on the same way.
> > > Frankly speaking, I never see benefits of this *separate* model, our
> > > project(or most projects) is not like Chrome which has to create
> > > sandbox for extensions in order to avoid bad plugin screws up the
> > > whole browser(however, I still see bad plugins screw up my Chrome
> > > well).
> > > Projects like CloudStack using plugin to decouple architecture should
> > > not introduce many isolations to plugin writer, the point preventing
> > > wrong use of some components is not making much sense to me. If a
> > > plugin not following guide(if we have
> > > it) we should kick it out, instead of making obstacles for 99% good
> people.
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > > > Sent: Wednesday, September 18, 2013 10:33 AM
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: [PROPOSAL] Modularize Spring
> > > >
> > > > Right, component isn't a thing. I probably shouldn't use that term.
> > > > I
> > > want to
> > > > standarize on the naming convention of plugin, module, and extension.
> > > It is
> > > > explained a bit on the wiki [1] but I'll try to do a little better
> > > > job
> > > here. So a
> > > > plugin is basically a jar. A jar contains multiple modules. A
> > > > modules
> > > ends up
> > > > being a spring application context that composes multiple
> > > > configuration
> > > files.
> > > > Modules are assembled into a hierarchy at runtime. Extensions are
> > > > implementations of interfaces that exist in a module. A maven
> > > > project produces a jar, so a plugin ends up being a maven project
> also.
> > > >
> > > > So currently today we don't have a strong definition of Plugin and I
> > > hope to
> > > > address that.
> > > >
> > > > Darren
> > > >
> > > > [1]
> > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> > > > ins%2C+Modules%2C+and+Extensions
> > > >
> > > >
> > > > On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
> > > > <da...@gmail.com>wrote:
> > > >
> > > > > sounds great Darren,
> > > > >
> > > > > By component, you mean maven project or some larger chunk like
> > > > > distribution package? (did i miss this definition somewhere or do
> > > > > we define the components now?)
> > > > >
> > > > > regards,
> > > > > Daan
> > > > >
> > > > > On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> > > > > <da...@gmail.com> wrote:
> > > > > > Currently ACS code is fairly modular in that you can add
> > > > > > plug-ins to ACS
> > > > > to
> > > > > > extend most functionality. Unfortunately ACS is not packaged in
> > > > > > a
> > > > > modular
> > > > > > way. It is still delivered essentially as one large unit.
> > > > > > There are
> > > > > many
> > > > > > reason for this but one large barrier to modularizing ACS is
> > > > > > that the Spring configuration is managed as one large unit.
> > > > > >
> > > > > > I propose that we break apart the Spring XML configuration such
> > > > > > that each component contributes its own configuration.
> > > > > > Additionally each component will be loaded into its own Spring
> > > > > > ApplicationContext such that its beans will not conflict with
> > > > > > the wiring of other beans in ACS. This change
> > > > > will
> > > > > > lay the foundation for a richer plugin model and additionally a
> > > > > > more distributed architecture.
> > > > > >
> > > > > > The technical details for this proposal can be found on the wiki
> [1].
> > > > > >
> > > > > > Darren
> > > > > >
> > > > > > [1]
> > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+
> > > > > Spri
> > > > > ng
> > > > >
> > >
>
RE: [PROPOSAL] Modularize Spring
Posted by Frank Zhang <Fr...@citrix.com>.
I am not against boundary, I am just against making things unnecessary complex to enable boundary.
If we are going this way, I hope we can make it as much as transparent to developers. That means, as
a developer, all a plugin I need to do is 1) provide my separate spring xml 2) inject beans I want (valid beans)
in my bean and code business logic 3) compile to a jar and put to some place designated by CloudStack. That's it.
I raise this topic because I have seen some projects to create boundary making things horrible complex. And
sometimes developers are hurt by wrong boundaries, as a result, to overcome these limitations people write
lots of ugly code which makes thing even worse.
However, I am still worry about if we can make things so simpler.
For example, we may have an orchestration context that contains major beans needed by almost every
plugin, this context can be easily set as parent context for all plugin contexts when bootstrap. However, if a plugin A needs
to access some bean defined in plugin B, given they are sibling, how plugin framework resolves the dependency ?
> -----Original Message-----
> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> Sent: Wednesday, September 18, 2013 11:53 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Modularize Spring
>
> I'm not for OSGi either, but contexts are quite useful and will lead to better
> things. First off, we don't want one gigantic spring XML config file like we have
> today. I think we can all agree on that. So each plugin will have to supply its
> own XML. So the obstacles you mention, will largely be just that for people.
>
> With Spring it is really simple to just inject dependencies and cross
> architectural boundaries. Its currently everywhere in ACS. You can't just say
> we should review code and make sure nobody doesn't do bad things. A little bit
> of framework to enforce separation is a good thing. But I'm guessing you will
> disagree with me there.
>
> Here are some random points on why contexts are good. Say I want to use
> Spring AOP or Spring TX in my plugin. With your own context you can ensure
> that you won't screw with anybody else code by accidentally having you
> pointcut match their bean. You don't have to worry about bean name conflicts.
> If two config files specify bean X, Spring will gladly just use the last one. I've
> already found multiply defined beans in ACS, but things still just happen to work.
> Having multiple contexts also defines initialization order. We can ensure that
> the framework is loaded and ready before child context are loaded and started.
> (we kind of do this today with ComponentLifeCycle, but its a hack in my mind).
> Additionally, when things start you will know, loading context "crapping plug X".
> If spring fails to initialize, the issue it there. Today, if spring fails to start, it
> could be one of over 500 beans causing the weird problem. The list goes on
> and on.
>
> Finally, this is the big one and why I really want contexts. I have some notes on
> the wiki [1] that you might want to read through. Basically I want to get to a
> more flexible deployment model that allows both a single monolithic JVM as
> today and also a fully distributed system. Having contexts in hierarchy will
> enable me to accomplish this. By selecting which contexts are loaded at
> runtime will determine what role the JVM takes on. The contexts help people
> better understand how the distributed architecture will work too, when we get
> there.
>
> Frank, trust me, I hate complex things. I don't want OSGi, classloader magic,
> etc. But I do like organization and a little bit of framework so that people don't
> accidentally shoot themselves in the foot. I personally like knowing that I
> couldn't have screwed something up, because the framework won't even allow
> it. If we separate everything as I want today, and then tomorrow we say this is
> way too much overhead, moving to a flat context is simple. Don't think we are
> on some slippery slope to classloaders and dependency hell.
>
> Darren
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+her
> e...#Nothingtoseehere...-DeploymentModels
>
>
>
> On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang
> <Fr...@citrix.com>wrote:
>
> > What's the point in using separate spring context per plugin?
> > Separate class loader is the thing I hate most in OSGI, I am afraid we
> > are on the same way.
> > Frankly speaking, I never see benefits of this *separate* model, our
> > project(or most projects) is not like Chrome which has to create
> > sandbox for extensions in order to avoid bad plugin screws up the
> > whole browser(however, I still see bad plugins screw up my Chrome
> > well).
> > Projects like CloudStack using plugin to decouple architecture should
> > not introduce many isolations to plugin writer, the point preventing
> > wrong use of some components is not making much sense to me. If a
> > plugin not following guide(if we have
> > it) we should kick it out, instead of making obstacles for 99% good people.
> >
> >
> >
> > > -----Original Message-----
> > > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > > Sent: Wednesday, September 18, 2013 10:33 AM
> > > To: dev@cloudstack.apache.org
> > > Subject: Re: [PROPOSAL] Modularize Spring
> > >
> > > Right, component isn't a thing. I probably shouldn't use that term.
> > > I
> > want to
> > > standarize on the naming convention of plugin, module, and extension.
> > It is
> > > explained a bit on the wiki [1] but I'll try to do a little better
> > > job
> > here. So a
> > > plugin is basically a jar. A jar contains multiple modules. A
> > > modules
> > ends up
> > > being a spring application context that composes multiple
> > > configuration
> > files.
> > > Modules are assembled into a hierarchy at runtime. Extensions are
> > > implementations of interfaces that exist in a module. A maven
> > > project produces a jar, so a plugin ends up being a maven project also.
> > >
> > > So currently today we don't have a strong definition of Plugin and I
> > hope to
> > > address that.
> > >
> > > Darren
> > >
> > > [1]
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> > > ins%2C+Modules%2C+and+Extensions
> > >
> > >
> > > On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
> > > <da...@gmail.com>wrote:
> > >
> > > > sounds great Darren,
> > > >
> > > > By component, you mean maven project or some larger chunk like
> > > > distribution package? (did i miss this definition somewhere or do
> > > > we define the components now?)
> > > >
> > > > regards,
> > > > Daan
> > > >
> > > > On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> > > > <da...@gmail.com> wrote:
> > > > > Currently ACS code is fairly modular in that you can add
> > > > > plug-ins to ACS
> > > > to
> > > > > extend most functionality. Unfortunately ACS is not packaged in
> > > > > a
> > > > modular
> > > > > way. It is still delivered essentially as one large unit.
> > > > > There are
> > > > many
> > > > > reason for this but one large barrier to modularizing ACS is
> > > > > that the Spring configuration is managed as one large unit.
> > > > >
> > > > > I propose that we break apart the Spring XML configuration such
> > > > > that each component contributes its own configuration.
> > > > > Additionally each component will be loaded into its own Spring
> > > > > ApplicationContext such that its beans will not conflict with
> > > > > the wiring of other beans in ACS. This change
> > > > will
> > > > > lay the foundation for a richer plugin model and additionally a
> > > > > more distributed architecture.
> > > > >
> > > > > The technical details for this proposal can be found on the wiki [1].
> > > > >
> > > > > Darren
> > > > >
> > > > > [1]
> > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+
> > > > Spri
> > > > ng
> > > >
> >
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
I'm not for OSGi either, but contexts are quite useful and will lead to
better things. First off, we don't want one gigantic spring XML config
file like we have today. I think we can all agree on that. So each plugin
will have to supply its own XML. So the obstacles you mention, will
largely be just that for people.
With Spring it is really simple to just inject dependencies and cross
architectural boundaries. Its currently everywhere in ACS. You can't just
say we should review code and make sure nobody doesn't do bad things. A
little bit of framework to enforce separation is a good thing. But I'm
guessing you will disagree with me there.
Here are some random points on why contexts are good. Say I want to use
Spring AOP or Spring TX in my plugin. With your own context you can ensure
that you won't screw with anybody else code by accidentally having you
pointcut match their bean. You don't have to worry about bean name
conflicts. If two config files specify bean X, Spring will gladly just use
the last one. I've already found multiply defined beans in ACS, but things
still just happen to work. Having multiple contexts also defines
initialization order. We can ensure that the framework is loaded and ready
before child context are loaded and started. (we kind of do this today
with ComponentLifeCycle, but its a hack in my mind). Additionally, when
things start you will know, loading context "crapping plug X". If spring
fails to initialize, the issue it there. Today, if spring fails to start,
it could be one of over 500 beans causing the weird problem. The list goes
on and on.
Finally, this is the big one and why I really want contexts. I have some
notes on the wiki [1] that you might want to read through. Basically I
want to get to a more flexible deployment model that allows both a single
monolithic JVM as today and also a fully distributed system. Having
contexts in hierarchy will enable me to accomplish this. By selecting
which contexts are loaded at runtime will determine what role the JVM takes
on. The contexts help people better understand how the distributed
architecture will work too, when we get there.
Frank, trust me, I hate complex things. I don't want OSGi, classloader
magic, etc. But I do like organization and a little bit of framework so
that people don't accidentally shoot themselves in the foot. I personally
like knowing that I couldn't have screwed something up, because the
framework won't even allow it. If we separate everything as I want today,
and then tomorrow we say this is way too much overhead, moving to a flat
context is simple. Don't think we are on some slippery slope to
classloaders and dependency hell.
Darren
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Nothing+to+see+here...#Nothingtoseehere...-DeploymentModels
On Wed, Sep 18, 2013 at 11:22 AM, Frank Zhang <Fr...@citrix.com>wrote:
> What's the point in using separate spring context per plugin?
> Separate class loader is the thing I hate most in OSGI, I am afraid we are
> on the same way.
> Frankly speaking, I never see benefits of this *separate* model, our
> project(or most projects) is not like Chrome which has to create sandbox
> for extensions
> in order to avoid bad plugin screws up the whole browser(however, I still
> see bad plugins screw up my Chrome well).
> Projects like CloudStack using plugin to decouple architecture should not
> introduce many isolations to plugin writer, the point preventing wrong use
> of some components
> is not making much sense to me. If a plugin not following guide(if we have
> it) we should kick it out, instead of making obstacles for 99% good people.
>
>
>
> > -----Original Message-----
> > From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> > Sent: Wednesday, September 18, 2013 10:33 AM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [PROPOSAL] Modularize Spring
> >
> > Right, component isn't a thing. I probably shouldn't use that term. I
> want to
> > standarize on the naming convention of plugin, module, and extension.
> It is
> > explained a bit on the wiki [1] but I'll try to do a little better job
> here. So a
> > plugin is basically a jar. A jar contains multiple modules. A modules
> ends up
> > being a spring application context that composes multiple configuration
> files.
> > Modules are assembled into a hierarchy at runtime. Extensions are
> > implementations of interfaces that exist in a module. A maven project
> > produces a jar, so a plugin ends up being a maven project also.
> >
> > So currently today we don't have a strong definition of Plugin and I
> hope to
> > address that.
> >
> > Darren
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> > ins%2C+Modules%2C+and+Extensions
> >
> >
> > On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
> > <da...@gmail.com>wrote:
> >
> > > sounds great Darren,
> > >
> > > By component, you mean maven project or some larger chunk like
> > > distribution package? (did i miss this definition somewhere or do we
> > > define the components now?)
> > >
> > > regards,
> > > Daan
> > >
> > > On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> > > <da...@gmail.com> wrote:
> > > > Currently ACS code is fairly modular in that you can add plug-ins to
> > > > ACS
> > > to
> > > > extend most functionality. Unfortunately ACS is not packaged in a
> > > modular
> > > > way. It is still delivered essentially as one large unit. There
> > > > are
> > > many
> > > > reason for this but one large barrier to modularizing ACS is that
> > > > the Spring configuration is managed as one large unit.
> > > >
> > > > I propose that we break apart the Spring XML configuration such that
> > > > each component contributes its own configuration. Additionally each
> > > > component will be loaded into its own Spring ApplicationContext such
> > > > that its beans will not conflict with the wiring of other beans in
> > > > ACS. This change
> > > will
> > > > lay the foundation for a richer plugin model and additionally a more
> > > > distributed architecture.
> > > >
> > > > The technical details for this proposal can be found on the wiki [1].
> > > >
> > > > Darren
> > > >
> > > > [1]
> > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Spri
> > > ng
> > >
>
RE: [PROPOSAL] Modularize Spring
Posted by Frank Zhang <Fr...@citrix.com>.
What's the point in using separate spring context per plugin?
Separate class loader is the thing I hate most in OSGI, I am afraid we are on the same way.
Frankly speaking, I never see benefits of this *separate* model, our project(or most projects) is not like Chrome which has to create sandbox for extensions
in order to avoid bad plugin screws up the whole browser(however, I still see bad plugins screw up my Chrome well).
Projects like CloudStack using plugin to decouple architecture should not introduce many isolations to plugin writer, the point preventing wrong use of some components
is not making much sense to me. If a plugin not following guide(if we have it) we should kick it out, instead of making obstacles for 99% good people.
> -----Original Message-----
> From: Darren Shepherd [mailto:darren.s.shepherd@gmail.com]
> Sent: Wednesday, September 18, 2013 10:33 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [PROPOSAL] Modularize Spring
>
> Right, component isn't a thing. I probably shouldn't use that term. I want to
> standarize on the naming convention of plugin, module, and extension. It is
> explained a bit on the wiki [1] but I'll try to do a little better job here. So a
> plugin is basically a jar. A jar contains multiple modules. A modules ends up
> being a spring application context that composes multiple configuration files.
> Modules are assembled into a hierarchy at runtime. Extensions are
> implementations of interfaces that exist in a module. A maven project
> produces a jar, so a plugin ends up being a maven project also.
>
> So currently today we don't have a strong definition of Plugin and I hope to
> address that.
>
> Darren
>
> [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-
> ins%2C+Modules%2C+and+Extensions
>
>
> On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland
> <da...@gmail.com>wrote:
>
> > sounds great Darren,
> >
> > By component, you mean maven project or some larger chunk like
> > distribution package? (did i miss this definition somewhere or do we
> > define the components now?)
> >
> > regards,
> > Daan
> >
> > On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> > <da...@gmail.com> wrote:
> > > Currently ACS code is fairly modular in that you can add plug-ins to
> > > ACS
> > to
> > > extend most functionality. Unfortunately ACS is not packaged in a
> > modular
> > > way. It is still delivered essentially as one large unit. There
> > > are
> > many
> > > reason for this but one large barrier to modularizing ACS is that
> > > the Spring configuration is managed as one large unit.
> > >
> > > I propose that we break apart the Spring XML configuration such that
> > > each component contributes its own configuration. Additionally each
> > > component will be loaded into its own Spring ApplicationContext such
> > > that its beans will not conflict with the wiring of other beans in
> > > ACS. This change
> > will
> > > lay the foundation for a richer plugin model and additionally a more
> > > distributed architecture.
> > >
> > > The technical details for this proposal can be found on the wiki [1].
> > >
> > > Darren
> > >
> > > [1]
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Spri
> > ng
> >
Re: [PROPOSAL] Modularize Spring
Posted by Darren Shepherd <da...@gmail.com>.
Right, component isn't a thing. I probably shouldn't use that term. I
want to standarize on the naming convention of plugin, module, and
extension. It is explained a bit on the wiki [1] but I'll try to do a
little better job here. So a plugin is basically a jar. A jar contains
multiple modules. A modules ends up being a spring application context
that composes multiple configuration files. Modules are assembled into a
hierarchy at runtime. Extensions are implementations of interfaces that
exist in a module. A maven project produces a jar, so a plugin ends up
being a maven project also.
So currently today we don't have a strong definition of Plugin and I hope
to address that.
Darren
[1]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Plug-ins%2C+Modules%2C+and+Extensions
On Wed, Sep 18, 2013 at 4:25 AM, Daan Hoogland <da...@gmail.com>wrote:
> sounds great Darren,
>
> By component, you mean maven project or some larger chunk like
> distribution package? (did i miss this definition somewhere or do we
> define the components now?)
>
> regards,
> Daan
>
> On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
> <da...@gmail.com> wrote:
> > Currently ACS code is fairly modular in that you can add plug-ins to ACS
> to
> > extend most functionality. Unfortunately ACS is not packaged in a
> modular
> > way. It is still delivered essentially as one large unit. There are
> many
> > reason for this but one large barrier to modularizing ACS is that the
> > Spring configuration is managed as one large unit.
> >
> > I propose that we break apart the Spring XML configuration such that each
> > component contributes its own configuration. Additionally each component
> > will be loaded into its own Spring ApplicationContext such that its beans
> > will not conflict with the wiring of other beans in ACS. This change
> will
> > lay the foundation for a richer plugin model and additionally a more
> > distributed architecture.
> >
> > The technical details for this proposal can be found on the wiki [1].
> >
> > Darren
> >
> > [1]
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Spring
>
Re: [PROPOSAL] Modularize Spring
Posted by Daan Hoogland <da...@gmail.com>.
sounds great Darren,
By component, you mean maven project or some larger chunk like
distribution package? (did i miss this definition somewhere or do we
define the components now?)
regards,
Daan
On Wed, Sep 18, 2013 at 12:10 AM, Darren Shepherd
<da...@gmail.com> wrote:
> Currently ACS code is fairly modular in that you can add plug-ins to ACS to
> extend most functionality. Unfortunately ACS is not packaged in a modular
> way. It is still delivered essentially as one large unit. There are many
> reason for this but one large barrier to modularizing ACS is that the
> Spring configuration is managed as one large unit.
>
> I propose that we break apart the Spring XML configuration such that each
> component contributes its own configuration. Additionally each component
> will be loaded into its own Spring ApplicationContext such that its beans
> will not conflict with the wiring of other beans in ACS. This change will
> lay the foundation for a richer plugin model and additionally a more
> distributed architecture.
>
> The technical details for this proposal can be found on the wiki [1].
>
> Darren
>
> [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/Modularize+Spring