OSGi Deps was [Re: Separating out RMIClassLoader dependencies]

[Peter Firmstone <ji...@zeus.net.au>]

Thanks Nic for your comments, pls see reply below.

Niclas Hedhman wrote:
 > A big chunk of your bundle analysis already exist in a widely used OSGi
 > tool, named BND. Most OSGi developers use it, since it is near 
impossible to
 > create the metadata by hand, especially the "uses" constraint.
 >  

Definitely agree and quite useful for eliminating syntax errors.

 > But before that we spoke of figuring out how a particular implementation
 > wire up a random set of bundles. That is not predictable, nor does 
the OSGi
 > spec provides an API to find out. Well, kind of... If you have the loaded
 > class, you can ask from which bundle it came, but unfortunately the
 > StackElement doesn't give you the Class, only the class name, which isn't
 > enough.
 >  

Bytecode analysis, can identify all method signatures, all fields, the
level of access, public, package private, protected or private.

It can also identify the serialVersionUID for classes that implement
Serializable.

In addition if a bundle is used, the Metadata in the bundle indicates
packages to be exported, which defines the publicly accessible API.  
Classloaders can be designed to not export private packages.

Metadata and API can be stored in Static Mirror Objects, for each
bundle, package and class.  Public and Protected Public Class
API signatures and serialVersionUID's, stored in class mirror objects,
would be compared to determine API compatibility.

Since Static Bytecode Anaysis enables identification of API
signatures, most importantly a Packages Public API, Protected API and
Class serialVersionUID's, it allows comparison of API between different
package versions. - Package API.

Special consideration has to be given to Package API that is shared
between multiple packages, lazy loading requires some thought too. 
Failed resolution would require a JVM restart, with current state,
persisted for a retry.

Additional support for the Java Product Versioning Specification may be 
advantageous, both as an alternative for developers and where maximum 
sharing of bytecode is required by allowing one Classloader per package, 
instead of per bundle.  In cases where maximum sharing of classes is 
desired, eg globally visible classes, it increases probability of 
successful Package API matches, between cooperating entities over time, 
where no individual entity has control over the available codebase. 

Compatible Package Implementations, with the latest version ID, would be 
preferred.  Considering the size and availability of OSGi bundle 
repository's, the likelyhood of version conflict isn't great, but it may 
be useful where new software is undergoing rapid development and API 
revision or where shared classes need to be persisted over time.

I think one limitation of OSGi bundles for a long lived distributed 
environment, not controlled by any one single entity is, compatibility 
is determined at creation time, as software ages, some components may go 
without updates, the original dependencies may become locked in, 
preventing cooperation between disparate remote systems that depend on 
utilizing compatibility of common packages.   Package authors may want 
to break backward compatibility at some later date by dropping 
depreciated methods or classes.  It may be possible to do this while 
preserving a form of Serialization compatibility to transition stateful 
objects to new class versions.

I've commented previously about handling of versioning conflicts using 
exceptions and JVM restarts.  This is still my intent, due to the JVM's 
inability to reload class definitions and my desire to reuse existing 
software.

Perhaps I should name the Mirror Interface, in Reflective Terms,
StaticMirror - to indicate no change, or ImmutableMirror, with 
implementations such as ClassImmutableMirror, PackageImmutableMirror and 
BundleImmutableMirror, Mirror objects produced by analysis could be 
stored in a hash, packages referred by name.

For dependency resolution, a PackageDependencyMirror Object,  for 
resolving package dependencies, generated with static bytecode analysis 
and used at runtime by ClassLoaders for submission to codebase services 
for resolution of the latest compatible provider packages.   In this 
case the local jvm implementation can learn of new compatible packages 
as they become available rather than from bundle metadata alone. If the 
dependency forms part of the exported Package API, then it becomes a 
special case where all dependants must also be checked for dependency on 
that provider also.

Package version metadata is important, without it, we cannot identify 
which is the later package and which is the former.  But combined with 
Static Analysis, opens the door to new possibilities.  All with regular 
jar files.

The responsibility of Immutable Mirrors belongs to the Classloader.

Marshalled Objects will need to be marshalled with their 
ClassImmutableMirror object, which is linked to the 
PackageImmutableMirror object, both must be very minimalistic to reduce 
added overheads, this might be added using bytecode weaving, adding a 
private field, or as part of the Marshalled Object Instance, allowing 
serialized objects to be unmarshalled using compatible class bytecode.  
Suggestions are welcome.

Best Regards,

Peter.



 > But I agree with you that with Jini tech and OSGi's well-defined
 > classloading semantics one can do really cool stuff. Including 
getting rid
 > of some of the more troubling Jini aspects (codebase).
 >
 > -- Niclas
 >
 > On Oct 16, 2009 8:21 PM, "Peter Firmstone" <ji...@zeus.net.au> wrote:
 >
 > Hmm, that sounds like opportunity.
 >
 > A dedicated codebase service has all the time in the world to burn 
processor
 > cycles on analysis, as its main task is simply serving up jar files over
 > networks.
 >
 > Bytecode analysis, identifies class, package and possibly module API, 
which
 > can be stored with the harvested metadata in mirror objects, one for each
 > class, package and module.  Optional Package metadata could be a 
potential
 > source of information too.
 >
 > Bundles, depend upon and export packages.  Once the API is identified 
using
 > bytecode analysis, fast comparison using mirror objects could identify
 > compatibility.
 >
 > This could be utilised in two ways:
 >
 > 1. As a check of backward compatibility for modules / packages over
 > different release versions, for substituting later compatible bundle
 > versions, if desired.
 > 2. When a dependency on a package only utilises a subset of that package,
 > the actual API requirements may still be satisfied, even though different
 > release versions of that package may not usually be interchangeable 
or fully
 > compatible.
 >
 > A module or bundle would exist within its own classloader, in the 
local jvm.
 >   The packages or modules upon which it depends could be made 
available from
 > other classloaders.
 >
 > Bundles could be uploaded to codebase services or a codebase service 
could
 > retrieve the bundles from designated repository's, perform analysis, then
 > make the bundles available in a location independent manner, to prevent
 > codebase loss and allow for redundant codebase services.
 >
 > Then all one need do is to upload application bundles to the codebase 
server
 > and register a service, the service and bytecode could be provided
 > independently, the codebase service can provide entire application 
bytecode
 > built on Jini services and other third party libraries.
 >
 > Have a look on springsource, there are many OSGi jar bundles available,
 > these are simply jar files with Metadata.
 >
 > Someone's done so much hard work already, why not ride the wave?  
There are
 > already support tools available to create application bundle manifests.
 >
 > The reason I'm considering bundles, is it reduces the number of 
classloaders
 > required, one per bundle as opposed to one per package.  One cannot 
rely on
 > standard java Package meta data to exist in jar files.
 >
 > ********This doesn't require an OSGi framework.*********
 >
 > http://blog.springsource.com/2008/02/18/creating-osgi-bundles/
 >
 > http://www.springsource.com/repository/app/bundle?query=A
 >
 > http://www.springsource.com/repository/app/faq
 >
 > >From the website:
 >
 >
 >     What is the SpringSource Enterprise Bundle Repository?
 >
 > The SpringSource Enterprise Bundle Repository is a collection of open 
source
 > libraries commonly used for developing enterprise Java applications 
with the
 > Spring Framework. The repository contains jar files (bundles) and library
 > definition (".libd") files. A library defines a collection of bundles 
that
 > are often used together for some purpose (e.g. the "Spring Framework"
 > library). There are hundreds of bundles contained in the repository.
 >
 > The repository meets the following criteria:
 >
 >   * Every jar file in the repository is a valid OSGi bundle. Any jar
 >     you download from the repository can be deployed as-is into an
 >     OSGi Service Platform and the SpringSource dm Server. It can also
 >     be used as a regular jar file outside of OSGi.
 >   * Every bundle and library has full version information associated
 >     with it. The package export information for a bundle contains
 >     version information, and the package import information for a
 >     bundle contains full version range compatibility information.
 >   * The repository is transitively complete. The mandatory
 >     dependencies of any bundle are guaranteed to also be in the
 >     repository. Most of the optional dependencies of any bundle in the
 >     repository will also be present. The bundles listed in any library
 >     definition are guaranteed to be in the repository.
 >   * The repository is self-consistent. Before any artefact is uploaded
 >     to the repository, we verify that it can be installed, resolved,
 >     and started in an OSGi Service Platform (using the same profile as
 >     the SpringSource dm Server) alongside all of the other bundles in
 >     the repository.
 >   * The repository can be used from Ivy and Maven based builds.
 >
 > Cheers,
 >
 > Peter.
 >
 > Niclas Hedhman wrote: > > On Fri, Oct 16, 2009 at 1:04 PM, Niclas 
Hedhman <
 > niclas@hedhman.org> wro...
 >
 >