[merlin] a view of a wizard's laboratory

[Leo Simons <le...@apache.org>]

A view of a running merlin system
---------------------------------

+-------------------------------------------------------+
| Merlin CLI (bootstrapping package)                    |
|                                                       |
| +---------------------------------------------------+ |
| | Merlin Kernel                                     | |
| |                                                   | |
| | +-----------------------------------------------+ | |
| | | Merlin root block                             | | |
| | |                                               | | |
| | | +-------------------------------------------+ | | |
| | | | Merlin root container                     | | | |
| | | |                                           | | | |
| | | | +------------------- +------------------+ | | | |
| | | | | Merlin appliance | | Merlin appliance | | | | |
| | | | |                  | |                  | | | | |
| | | | | -------------    | | -------------    | | | | |
| | | | | | Component |    | | | Component |    | | | | |
| | | | | -------------    | | -------------    | | | | |
| | | | +------------------+ +------------------+ | | | |
| | | |                                           | | | |
| | | +-------------------------------------------+ | | |
| | |                                               | | |
| | +-----------------------------------------------+ | |
| |                                                   | |
| +---------------------------------------------------+ |
|                                                       |
+-------------------------------------------------------+
                               (a typical simple running merlin system)


What happens on startup
-----------------------
the merlin-cli module use commons-cli to parse its commandline arguments 
and set up the stuff the merlin kernel needs. It bootstraps merlin-core. 
It uses a KernelLoader to create a new Kernel instance.

when DefaultKernelLoader calls initialize() on DefaultKernel, the first 
thing DefaultKernel does is registering a shutdown hook with the 
runtime. It then creates several things it needs for setting up the root 
block (a logger manager, a thread pool, an assembly engine, a block 
loader). Which implementation to use for those things is currently not 
configurable and the methods which create them are private. This is 
different from the phoenix kernel, where you can specify implementations 
in the kernel.xml file. The DefaultKernel then proceeds on to set up and 
initialize the assembly engine and then the root block loader. It 
figures out which block should be its root block, and assembles then 
installs this block into the root block loader. After that, 
DefaultKernel won't do anything till shutdown.

Control transfers to the root block. This is a big ugly class which does 
lots of things when a block is installed, including recursively building 
a graph of all child blocks. Which block is created and what exactly 
needs to be set up for it is declared in a block.xml file whose location 
gets passed in from the command line.

The root block will normally be an instance of DefaultBlock, which will 
manage n appliances (which appliances to manage is configured via more 
xml config). It uses the assembly engine (EngineClassLoader from the 
merlin-assembly package) for actually building and setting up those 
appliances. This is again a huge complicated class which does DAG 
building and walking in proper ordering.

The usual appliances which are set up are DefaultAppliance and/or 
DefaultBlock (yes, you can nest merlin inside itself to arbitrary 
depth), which are again complicated classes to wrap around a component. 
They work together with the assembly engine to dynamically setup and 
manage the component they host.

At some point in time, all graphs will have been traversed, and all 
components will hopefully have been properly set up and initialized. I'm 
not going to go into bigger detail on the assembly and appliance 
internals right now; attaching a debugger and using "go to declaration" 
will get you from place to place easily enough.



What happens on shutdown
------------------------
When you tell the java runtime to terminate, the shutdown hook of 
DefaultKernel kicks in and calls shutdown() on the kernel, which resuls 
in shutdown of the root block, which resuls in shutdown of the 
containers, appliances, blocks and components created during the above 
phases in reverse order.



A view on this setup
--------------------
Clearly, it is an order of magnitude more complex than what fortress or 
phoenix does. There is lots of complex, dynamic logic involved in the 
resolution of the multiple (possibly dozens) levels, with implementation 
spread out over several complex default implementations. Coupling 
between component implementations is at some points more tight than one 
would expect from the seperation of spi and implementation; clearly 
merlin needs at least a round or two of refactoring to extend the spi 
interfaces and reduce the coupling between its components.

Fortress is something you might wrap as an implementation of what is a 
"container" in the merlin context, then run inside one of the blocks 
hosted by the root block. Merlin would not typically want to host 
phoenix: it does most of the things phoenix is used for by itself.

The reverse very is possible: DefaultKernel is an avalon component which 
you should be able to run within phoenix without notable issues. Since 
all classes described above are indeed components, you can also embed 
only part of the merlin subsystem as a component inside another 
container, for example you could also choose to bypass DefaultKernel and 
DefaultBlock and host DefaultAppliance directly.



The Merlin metadata attribute system
------------------------------------
Is completely seperate from the runtime; you generate xml files (which 
are parsed as avalon configuration objects during startup) from javadoc 
tags. These xml files contain lots of declarations of what the system 
does and assumes, exactly.



An opinion
----------
In my opinion, merlin has too many layers, and the seperation between 
those layers is at some points blurry. While there are some very 
powerful features up this wizard's sleeve, they are hidden nested 
several levels deep into the code.

Digesting the organisation of its internals takes way more time than for 
Fortress (which I "got" nearly completely in an afternoon) or Phoenix 
(which I basically learned in a single night). I am guessing I've also 
misunderstood some things, probably partly due to my dislike of 
scrolling long sourcefiles.

Merlin is a cool project with cool features, but it will take most 
people more time than me (and it took me too long already IMO) to digest 
its internals before being able to get to work on it, and that is a huge 
(by far the biggest, actually) downside of the package.



Cross-container integration options
-----------------------------------

Merlin's Fortress

Given the nice self-containment of fortress, it is probably relatively 
easy to write a FortressContainer which implements the Container 
interface from merlin by wrapping around the fortress DefaultContainer. 
This would allow rather clean transparent integration between the two 
projects.

Fortified Merlin

Similarly, Merlin's DefaultKernel is a proper component which has no 
particularly heavy requirements and can likely be loaded into fortress 
rather easily.

Merlin's Phoenix

You don't want to embed phoenix inside merlin. Generally, you don't want 
to embed phoenix but do things the other way around (my opinion).

Foxy Merlin

DefaultKernel feels threadsafe alright, so you can easily run it as a 
component inside phoenix (possibly best to use a thin wrapper around it).

Merlin's Plexus

Haven't looked at plexus recently, but last time I checked is was pretty 
similar to fortress in scope and usage pattern. I think what I say about 
integration between fortress and merlin applies to integration between 
merlin and plexus as well.




hope this is of use,

cheers,

- Leo



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org

Re: [merlin] a view of a wizard's laboratory

[Stephen McConnell <mc...@apache.org>]

Leo Simons wrote:

> A view of a running merlin system
> --------------------------------- 


<snip ASCII art>

There is a better version here:

http://avalon.apache.org/sandbox/merlin/merlin/index.html

> What happens on startup
> -----------------------
> the merlin-cli module use commons-cli to parse its commandline 
> arguments and set up the stuff the merlin kernel needs. It bootstraps 
> merlin-core. It uses a KernelLoader to create a new Kernel instance.
>
> when DefaultKernelLoader calls initialize() on DefaultKernel, the 
> first thing DefaultKernel does is registering a shutdown hook with the 
> runtime. It then creates several things it needs for setting up the 
> root block (a logger manager, a thread pool, an assembly engine, a 
> block loader). Which implementation to use for those things is 
> currently not configurable and the methods which create them are 
> private. This is different from the phoenix kernel, where you can 
> specify implementations in the kernel.xml file. 


Under the current implementation the main conjuration concerns are the 
pool and logging manager.  These will be made configurable.  Engine 
related content in largely configurable under the current model.

> The DefaultKernel then proceeds on to set up and initialize the 
> assembly engine and then the root block loader. It figures out which 
> block should be its root block, and assembles then installs this block 
> into the root block loader. After that, DefaultKernel won't do 
> anything till shutdown.
>
> Control transfers to the root block. This is a big ugly class which 
> does lots of things when a block is installed, including recursively 
> building a graph of all child blocks. Which block is created and what 
> exactly needs to be set up for it is declared in a block.xml file 
> whose location gets passed in from the command line. 


The block loading scheme has been refractored since this email.  A lot 
of improvements have been put in place that primarily simply things at 
the implementation level.  You may want to take another look at the 
StandardBlock and StandardBlockLoader classes - I think you will find a 
significant improvement.

>
> The root block will normally be an instance of DefaultBlock, which 
> will manage n appliances (which appliances to manage is configured via 
> more xml config). It uses the assembly engine (EngineClassLoader from 
> the merlin-assembly package) for actually building and setting up 
> those appliances. This is again a huge complicated class which does 
> DAG building and walking in proper ordering. 


It's not so complicated.  Basically you have a bunch of stuff to do with 
(a) classloader handling and (b) type/profile/service repositories.  In 
the longer term I figure that the type/profile/service repositories can 
be moved out into a separate service. In the meantime there have been a 
bunch of renaming of things that should make things clearing in terms of 
what's-what.

>
> The usual appliances which are set up are DefaultAppliance and/or 
> DefaultBlock (yes, you can nest merlin inside itself to arbitrary 
> depth), which are again complicated classes to wrap around a 
> component. They work together with the assembly engine to dynamically 
> setup and manage the component they host.
>
> At some point in time, all graphs will have been traversed, and all 
> components will hopefully have been properly set up and initialized. 
> I'm not going to go into bigger detail on the assembly and appliance 
> internals right now; attaching a debugger and using "go to 
> declaration" will get you from place to place easily enough.


Attaching a debugger can be a little scary if you don't know what is 
happening.  Basically Merlin is using Merlin to build and deploy the 
components that make up the system that an instance of Merlin 
represents.  This means that what you see in a debug is the deployment 
by Merlin of appliance instances that build and deploy other appliances 
instances that may build and deploy additional appliances instances - 
which basically reflects the fact that just about anything in the Merlin 
system can be treated as a full Avalon component (lifecycle handler, 
contextization stage, component deployer, deployment factory, etc.).  In 
practice - this means that there is a cycle - the deployment of a 
component - that is totally central to the integrity of the Merlin system.

>
> What happens on shutdown
> ------------------------
> When you tell the java runtime to terminate, the shutdown hook of 
> DefaultKernel kicks in and calls shutdown() on the kernel, which 
> resuls in shutdown of the root block, which resuls in shutdown of the 
> containers, appliances, blocks and components created during the above 
> phases in reverse order. 


Shutdown order is based on the service/dependency graph. I.e.  if a if 
component X is subject to shutdown, all components (Y) that are 
dependent on X will be shutdown and then same process is applied to Y 
before shutdown of X is initiated.  This means that no component 
consuming a shutdown target is running before the shutdown of X is 
initiated.

>
>
> A view on this setup
> --------------------
> Clearly, it is an order of magnitude more complex than what fortress 
> or phoenix does. There is lots of complex, dynamic logic involved in 
> the resolution of the multiple (possibly dozens) levels, with 
> implementation spread out over several complex default 
> implementations. Coupling between component implementations is at some 
> points more tight than one would expect from the seperation of spi and 
> implementation; clearly merlin needs at least a round or two of 
> refactoring to extend the spi interfaces and reduce the coupling 
> between its components.


Agreed.

There are two area on my hit-list - the lifestyle and the lifecycle 
packages (fortunately both represent implementation details).


>
> Fortress is something you might wrap as an implementation of what is a 
> "container" in the merlin context, then run inside one of the blocks 
> hosted by the root block. 


There are some recent updates to Merlin that move "container" 
requirements to any component.  Placing Fortress into this context today 
is a feasible proposition (without any change to Fortress).


> Merlin would not typically want to host phoenix: it does most of the 
> things phoenix is used for by itself.
>
> The reverse very is possible: DefaultKernel is an avalon component 
> which you should be able to run within phoenix without notable issues. 
> Since all classes described above are indeed components, you can also 
> embed only part of the merlin subsystem as a component inside another 
> container, for example you could also choose to bypass DefaultKernel 
> and DefaultBlock and host DefaultAppliance directly.
>
>
>
> The Merlin metadata attribute system
> ------------------------------------
> Is completely seperate from the runtime; you generate xml files (which 
> are parsed as avalon configuration objects during startup) from 
> javadoc tags. These xml files contain lots of declarations of what the 
> system does and assumes, exactly.


Just a note on this - the meta generation needs a little more smarts 
concerning handling of "users" defined declarations.  Initially this 
will include skipping generation if a user-defined variant exists - and 
later on I would like this to validate the user-defined version as well.

> An opinion
> ----------
> In my opinion, merlin has too many layers, and the seperation between 
> those layers is at some points blurry. While there are some very 
> powerful features up this wizard's sleeve, they are hidden nested 
> several levels deep into the code. 


There is a lot of flexibility - which begs the question - "is there too 
much"?  In some cases - maybe (in particular I'm thinking about the 
multiple lifecycle strategies.  On the other hand when dealing with 
non-Avalon components - this sort of stuff is basic requirements.  But 
yes - its nested - however - there is a bunch of stuff coming together 
which I hope will (a) document this, and (b) serve as examples, and (c) 
function as operation test cases.


>
> Merlin is a cool project with cool features


Something important happened earlier today concerning "cool" stuff.  
It's basically the completion of the "dynamic components" solution.  
Using block specifications in XML you can now declare in XML new 
components that export services based on the block XML implementation 
directives.  This is the thing I've been aiming towards for ages - the 
ability to declare a component model at the assembly level - and deploy 
it and have it behave like a component to the components that consume 
it.  A demonstration and test case are now available in CVS and the 
documentation on this will be up on the Avalon site ASAP.

Cheers, Steve.

-- 

Stephen J. McConnell
mailto:mcconnell@apache.org
http://www.osm.net

Sent via James running under Merlin as an NT service.



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@avalon.apache.org
For additional commands, e-mail: dev-help@avalon.apache.org