[DISCUSS] Decouple your code by text file(e.g XML)

[Frank Zhang <Fr...@citrix.com>]

I have said too much about XML files recently, excuse me for raising this topic again, 
I really want to elaborate why text (e.g XML) configuration files is so important and how 
it decouples your project especially for project written by static language like java.

Let me cite our new ApiDiscoveryService as an example(This service is very good, I use it
as example just because I encounter a small problem and the problem is a good study case
for decoupling code using text configuration files).

The problem is: ApiDiscoveryService discovers API commands by invoking PluggableService. getCommands().
Each plugin service needs to implement getCommands() which returns a List<Class<?>> including classes of
API command the plugin service supports. This is good, lots of project use similar way in plugin system.

ManagementServerImpl is a special API service, it registers all orchestration API commands like *addHost*. Now
I have a special API called *addBaremetalHost* which inherits *addHost* adding one more field.  *addBaremetalHost*
is actually handled by the same orchestration code as *addHost*(but different HostDiscover which is another story),
unfortunately, this API belongs to my Baremetal plugin that means I can't add it into  ManagementServerImpl. getCommands()
because orchestration code should not have a dependency to plugin code.

The solution is simple, I use a BaremetalManager to register *addBaremetalHost* and inject/call orchestration code to deal with it.

However, this case shows us how compiling time dependency in static language makes your code tight coupling. 
Everybody programing in java/C/C++ knows the pain of planning your project skeleton to avoid circular dependency. 
One of solutions is to use text configuration file. Back to our example, ApiDiscoveryService, we can put a folder which
stores API configuration XML at a well-known place, every PluggableService has a XML file describing its API and Spring bean
name in the folder. for example:

<apiService>
	<api>
		<command> com.cloud.baremetal.manager.AddBaremetalHostCmd</command>
		<acl>some acl</acl>
		<owner>Spring_bean_name_of_baremetal_manager</owner>
  	</api>		
<apiService>

When ApiDiscoveryService starts, it scans the folder and parses XML files to discover every api/theirs properties/owner bean. By doing this, ApiDiscoveryService/
PluggableService/API command classes totally have no compiling time dependency, they can freely seat in any packages as long as these packages present in
classpath during runtime. To my case, I don't have to inject orchestration code in my plugin anymore,  I just use a XML file to tell ApiDiscoveryService that addBaremetalHost
is taken cared by ManagementServerImpl as well though ManagementServerImpl has no knowledge about it.

CloudStack used to have similar idea command.properties, unfortunately it's only used as ACL declaration now.

This is a trivial example,  but it reflects a truth that compiling dependency is naturally making your code tight. Many people criticize dynamic language is error-prone because it
lacks compiling time check, but as another side of coin, it's also easy to produce loose-coupling code. Thinking about OpenStack, if you were going to use Java to fork it, I am pretty
sure the compiling time dependency will drive you crazy.  For instance, java-nova and java-cinder both need to know Volume that represents vm disk, now where do you define it? in java-nova?
in java-cinder? Then who depends on who? Or totally don't define any data structure, using a HashMap to transmit primitive variables, then where is beauty of compiling check?

As we have adopted Spring that uses XML a lot in CloudStack, I suggest us taking more consideration of text configuration file when designing a new feature. it really works.

Re: [DISCUSS] Decouple your code by text file(e.g XML)

[Chip Childers <ch...@sungard.com>]

On Thu, Feb 14, 2013 at 04:24:48PM -0800, Frank Zhang wrote:
> I have said too much about XML files recently, excuse me for raising this topic again, 
> I really want to elaborate why text (e.g XML) configuration files is so important and how 
> it decouples your project especially for project written by static language like java.
> 
> Let me cite our new ApiDiscoveryService as an example(This service is very good, I use it
> as example just because I encounter a small problem and the problem is a good study case
> for decoupling code using text configuration files).
> 
> The problem is: ApiDiscoveryService discovers API commands by invoking PluggableService. getCommands().
> Each plugin service needs to implement getCommands() which returns a List<Class<?>> including classes of
> API command the plugin service supports. This is good, lots of project use similar way in plugin system.
> 
> ManagementServerImpl is a special API service, it registers all orchestration API commands like *addHost*. Now
> I have a special API called *addBaremetalHost* which inherits *addHost* adding one more field.  *addBaremetalHost*
> is actually handled by the same orchestration code as *addHost*(but different HostDiscover which is another story),
> unfortunately, this API belongs to my Baremetal plugin that means I can't add it into  ManagementServerImpl. getCommands()
> because orchestration code should not have a dependency to plugin code.
> 
> The solution is simple, I use a BaremetalManager to register *addBaremetalHost* and inject/call orchestration code to deal with it.
> 
> However, this case shows us how compiling time dependency in static language makes your code tight coupling. 
> Everybody programing in java/C/C++ knows the pain of planning your project skeleton to avoid circular dependency. 
> One of solutions is to use text configuration file. Back to our example, ApiDiscoveryService, we can put a folder which
> stores API configuration XML at a well-known place, every PluggableService has a XML file describing its API and Spring bean
> name in the folder. for example:
> 
> <apiService>
> 	<api>
> 		<command> com.cloud.baremetal.manager.AddBaremetalHostCmd</command>
> 		<acl>some acl</acl>
> 		<owner>Spring_bean_name_of_baremetal_manager</owner>
>   	</api>		
> <apiService>
> 
> When ApiDiscoveryService starts, it scans the folder and parses XML files to discover every api/theirs properties/owner bean. By doing this, ApiDiscoveryService/
> PluggableService/API command classes totally have no compiling time dependency, they can freely seat in any packages as long as these packages present in
> classpath during runtime. To my case, I don't have to inject orchestration code in my plugin anymore,  I just use a XML file to tell ApiDiscoveryService that addBaremetalHost
> is taken cared by ManagementServerImpl as well though ManagementServerImpl has no knowledge about it.
> 
> CloudStack used to have similar idea command.properties, unfortunately it's only used as ACL declaration now.
> 
> This is a trivial example,  but it reflects a truth that compiling dependency is naturally making your code tight. Many people criticize dynamic language is error-prone because it
> lacks compiling time check, but as another side of coin, it's also easy to produce loose-coupling code. Thinking about OpenStack, if you were going to use Java to fork it, I am pretty
> sure the compiling time dependency will drive you crazy.  For instance, java-nova and java-cinder both need to know Volume that represents vm disk, now where do you define it? in java-nova?
> in java-cinder? Then who depends on who? Or totally don't define any data structure, using a HashMap to transmit primitive variables, then where is beauty of compiling check?
> 
> As we have adopted Spring that uses XML a lot in CloudStack, I suggest us taking more consideration of text configuration file when designing a new feature. it really works.
> 
>

+1 to what Frank is saying.  Runtime configuration is much more flexible
than compile-time scenarios!

-chip

Re: [DISCUSS] Decouple your code by text file(e.g XML)

[Rohit Yadav <bh...@apache.org>]

On Fri, Feb 15, 2013 at 5:54 AM, Frank Zhang <Fr...@citrix.com> wrote:
> I have said too much about XML files recently, excuse me for raising this topic again,
> I really want to elaborate why text (e.g XML) configuration files is so important and how
> it decouples your project especially for project written by static language like java.
>
> Let me cite our new ApiDiscoveryService as an example(This service is very good, I use it
> as example just because I encounter a small problem and the problem is a good study case
> for decoupling code using text configuration files).
>
> The problem is: ApiDiscoveryService discovers API commands by invoking PluggableService. getCommands().
> Each plugin service needs to implement getCommands() which returns a List<Class<?>> including classes of
> API command the plugin service supports. This is good, lots of project use similar way in plugin system.
>
> ManagementServerImpl is a special API service, it registers all orchestration API commands like *addHost*. Now
> I have a special API called *addBaremetalHost* which inherits *addHost* adding one more field.  *addBaremetalHost*
> is actually handled by the same orchestration code as *addHost*(but different HostDiscover which is another story),
> unfortunately, this API belongs to my Baremetal plugin that means I can't add it into  ManagementServerImpl. getCommands()
> because orchestration code should not have a dependency to plugin code.

Hi Frank, sorry for a really late reply I've a lot of pending email to
reply to or comment on. So, actually you can in your case. Just
implement a manager in your baremetal plugin, the managerimpl would be
a spring component that implements PluggableService and therefore
implements/overrides getCommands() which returns list of cmd classes.
For example, the api discovery plugin itself is a pluggable service
that return listApisCmd.class

>
> The solution is simple, I use a BaremetalManager to register *addBaremetalHost* and inject/call orchestration code to deal with it.
>
> However, this case shows us how compiling time dependency in static language makes your code tight coupling.
> Everybody programing in java/C/C++ knows the pain of planning your project skeleton to avoid circular dependency.
> One of solutions is to use text configuration file. Back to our example, ApiDiscoveryService, we can put a folder which
> stores API configuration XML at a well-known place, every PluggableService has a XML file describing its API and Spring bean
> name in the folder. for example:
>
> <apiService>
>         <api>
>                 <command> com.cloud.baremetal.manager.AddBaremetalHostCmd</command>
>                 <acl>some acl</acl>
>                 <owner>Spring_bean_name_of_baremetal_manager</owner>
>         </api>
> <apiService>
>
> When ApiDiscoveryService starts, it scans the folder and parses XML files to discover every api/theirs properties/owner bean. By doing this, ApiDiscoveryService/
> PluggableService/API command classes totally have no compiling time dependency, they can freely seat in any packages as long as these packages present in
> classpath during runtime. To my case, I don't have to inject orchestration code in my plugin anymore,  I just use a XML file to tell ApiDiscoveryService that addBaremetalHost
> is taken cared by ManagementServerImpl as well though ManagementServerImpl has no knowledge about it.
>
> CloudStack used to have similar idea command.properties, unfortunately it's only used as ACL declaration now.

The XML based config may help, but I'm not sure. I see a problem with
maintaining acl/xmls for every plugin. The ACL is not a property of
the plugin itself, it's actually a property of an ACL plugin, or
CloudStack/API server.

We separated policy from mechanism with ACL refactoring (which is not
totally complete, we still have ACL annotations and preprocessing
pending), so every plugins returns a list of cmd classes, but we've
another plugin called StaticRoleBasedACL which enforces some ACL rule
(in this case a static role mask defined in it's command.properties
file, one can diy acl plugin) for all apis.

Having just one command.properties file makes it easier for the
sysadmin to apply acl rules, and also this file does not belong to
CloudStack or all its plugins. It's actually only belong to the static
ACL plugin.

Should we use XML, yes but we'll have to group them in such a way that
it makes life of a CloudStack user, a sysadmin easier and not
difficult.

Regards.


>
> This is a trivial example,  but it reflects a truth that compiling dependency is naturally making your code tight. Many people criticize dynamic language is error-prone because it
> lacks compiling time check, but as another side of coin, it's also easy to produce loose-coupling code. Thinking about OpenStack, if you were going to use Java to fork it, I am pretty
> sure the compiling time dependency will drive you crazy.  For instance, java-nova and java-cinder both need to know Volume that represents vm disk, now where do you define it? in java-nova?
> in java-cinder? Then who depends on who? Or totally don't define any data structure, using a HashMap to transmit primitive variables, then where is beauty of compiling check?
>
> As we have adopted Spring that uses XML a lot in CloudStack, I suggest us taking more consideration of text configuration file when designing a new feature. it really works.
>