Cooperative Distributed computing - Versioning over time - Container Metadata.

[Peter Firmstone <ji...@zeus.net.au>]

Let me put it another way:

I want to work toward a cooperative distributed computing platform using 
Java & Jini / River, continuing from  where Sun left, using lessons learned.

   1. I'd like to set up a class loader framework building on
      classworlds, to segregate incompatible class versions with the
      same name, while allowing compatible implementations to
      interoperate, based on Class loader visibility.
   2. I'd like to utilise the ASM bytecode library to perform static
      analysis on bytecode to identify Package & Class API to determine
      compatibility.
   3. I'd like to solve the lost code base problem using codebase
      services combined with the suggested Class loading framework.
   4. I'd like to use codebase servcies to provide codebase proxy's for
      often disconnected clients.
   5. I'd like to allow bytecode to evolve organically by taking
      advantage of the flexibility of binary compatibility.
   6. I'd like some metadata to identify at the minimum ,Package
      Versions, I could generate checksums based on Package API, however
      this wouldn't identify which version is the latest, only which
      Package implementations can substitute another's API  where those
      packages share the same name. Or identify packages with the same
      name that are incompatible and segregate these in separate
      Classloaders.
   7. If possible due to concerns about the performance impact of many
      Classloaders, group some packages together when the author doesn't
      intend those packages to evolve separately.  Eg libraries.
   8. I'd like to make it possible to transfer class version and package
      version information with serialized object instances for
      compatible re-instantiation with the latest evolution of
      compatible classes.  All suggestions are welcome.

Does anyone have any suggestions for suitable existing formats for 
Containers with Version Metadata that might be utilised?   

If so what issues are there to be aware of?  What are the arguments for 
and Against?

Note, I'm aware of the existing Java Version Spec that applies to packages.

I believe I made the mistake of suggesting a solution without properly 
formulating or posing the problem, my apologies for any confusion this 
might have caused, I should have raised a new thread / topic,   In the 
original thread, Gregg is looking at classloader dependency issues 
implementing a plugin for Netbeans to make River more accessible to IDE 
developers, please feel free to make suggestions, we need your help.

I made a suggestion earlier to utilise the OSGi bundle metadata 
specification, these are ordinary jar files with package version, 
exports and dependency metadata.  They are otherwise ordinary plain old 
jar archives,  quite useable as a jar file containing java class files 
that can be utilised without  the OSGi framework, which is what I was 
suggesting. I attempted to imply that the exported packages could have 
their API identified and shared in a compatible manner with other 
packages, using Class loader isolation perhaps this is not a suitable 
solution due to the confusion that it would cause with developers 
expecting an OSGi implementation, utilising OSGi service interfaces.  If 
this format was to be utilised, it might have to be renamed, to River 
Codebase Bundles (for uploading to codebases) whilst sharing the same 
format to avoid confusion.   It might be a subset of the OSGi bundle 
format, such that OSGi bundles, like those on the Spring Source 
Repository could be utilised.  It might also be a component in an 
eventual Container spec, if we can sort that out.  Gregg has asked for 
container suggestions more than once, I'd like to see that conversation 
continued.

If this solution was adopted, the dependency version metadata (Import 
Package) in the bundle would be noted, considered a minimum version as 
per the OSGi spec, but a later version would be utilized if it were 
found API compatible.  However additional protection by static analysis 
would guarantee that a later version that wasn't API compatible would 
not be utilised. The reason I'm considering  this compromise is that a 
bundle author cannot know in advance that later import packages will be 
compatible, hence Static Analysis can verify API compatibility.  It is 
still possible for identical named classes to be loaded in the same jvm, 
in another classloader, this might occur if an earlier package version 
already exists but doesn't satisfy the Import Package version criteria; 
it wont be used as it is earlier than the minimum required version.  The 
Classloader responsible for the bundle would keep a copy of the  
metadata available while it remains loaded, it would not utilise the 
OSGi interfaces to access the metadata.  I envision Immutable Mirror 
Objects that can be transferred with marshalled instances and utilised 
for API compatibility checking.  The Mirror objects would be created by 
a codebase service and stored in memory by the Classloader that download 
the class bytecode (River Bundles) being Mirrored,  the mirror objects 
would contain the metadata from the bundle as well as the API, the 
mirror objects would be used by the classloader to determine which 
packages to export as well as check for import package compatibility, 
and by other Classloader's for compatibility.  If no compatible packages 
are found loaded then the codebase services would be consulted, the 
import package mirror object would be submitted to the codebase services 
to locate a compatible package or bundle.

In principle, if this is still objectionable, I'm happy to create an 
implementation utilising another container or bundle format that 
satisfies the requirements, provided that one is recommended.  This list 
does have a history of heated debate regarding OSGi and Jini, I'd like 
to avoid any such unproductive arguments if possible.

Regards,

Peter.


Niclas Hedhman wrote:
> On Mon, Oct 19, 2009 at 7:40 AM, Peter Firmstone <ji...@zeus.net.au> wrote:
>   
>>> Thanks Nic for your comments, pls see reply below.
>>>       
>
> I think we have diverged and discussing different things.
>
> Some time ago, it was said that the classloader of a class could be
> retrieved by calling a static method. I have tried to point out that
> this is fairly "impossible" in OSGi without collaboration with the
> OSGi framework implementation, as multiple (even fully compatible, or
> even exact same versions) instance of the same classes can be present
> in different class spaces within the OSGi framework, and which package
> got resolved to which bundle (and hence the classloader) is an
> implementation detail in OSGi. There is no theory around that fact. I
> am speaking of "locating a classloader of an existing class already
> present", and not how to locate one externally to the JVM and OSGi
> framework.
>
> See the difference?
>
> So if ClassA wants to load an implementation of InterfaceB, then that
> implementation must be chosen from the class space that ClassA and
> InterfaceB belongs to, and not only that it implements
> InterfaceB+constraints.
>
>
> Cheers
>   

Re: Cooperative Distributed computing - Versioning over time - Container Metadata.

[Peter Firmstone <ji...@zeus.net.au>]

Calum Shaw-Mackay wrote:
>> Interesting.
>>
>> Perhaps Marshalled object instances should also be decorated.
>>     
>
> Yeah you'd need somewhere to start, and as I mentioned below about
> transparency, you'd possibly need a
> DEFAULT or CURRENT version, because if you're bringing in an object
> foreign to the VM, and you're using
> static analysis, you'd need to have the bytecodes available to
> actually compare it against, having an initial
> version identifier would give it a head start in terms of performance,
> and you don't want to brute-force
> hammer the network bringing in multiple dl jars, for a class that has
> not been seen in any version in this JVM session
>
> I think the more 'hints' you could provide to the classloader, through
> either compatible version ranges or soft links
> the better.
>
>   
>>> Whichever way you want to tackle this, you'll need it to be as
>>> transparent a mechanism as possible so that service/bundles/clients
>>> that don;t specify versions work as they always have,
>>>
>>>       
>> Thanks Calum,  good point, when version metadata doesn't exist, the results
>> from Static Analysis alone could be utilised to find API compatible import
>> packages.  I guess classes lacking version metadata could be treated as
>> version 0.  If someone later decides to add version metadata, perhaps after
>> correcting a bug, the higher versioned package would be preferred, provided
>> it remains API compatible and shares the same fully qualified package name.
>>     
>
> Yeah I'd say a DEFAULT (i.e. 0, although that itself may go through
> changes in time) and perhaps symlink-type versioning including a
> CURRENT.
>   
I could use the class file modification dates from the jar files, so 
where more than one package of the same name exists, where both have an 
API that satisfy the dependency requirements, where version metadata 
isn't provided, the later modified file is preferred.
> In a way, what you don't want is to break compatibility unnecessarily.
>
> Calum
>
>   

Re: Cooperative Distributed computing - Versioning over time - Container Metadata.

[Peter Firmstone <ji...@zeus.net.au>]

Gregg Wonderly wrote:
> My changes to reggie to provide "never preferred classes" changed 
> MarshalledWrapper and EntryRep to implement
>
> public interface MarshalledDataAccess<T> {
>     public T getUnmarshalledData() throws IOException, 
> InstantiationException,
>             ClassNotFoundException, IllegalAccessException ;
>     public List<String> getClassNames();
> }
>
> There is still some other work that I need to finalize in these 
> changes to get service objects to always have this information (there 
> are complex threads of execution and MarshalledObject use).
>
> I made EntryRep work, and that was the big thing that I needed, so I 
> stopped at that point.
>
> Creating an interface, such as the above, and putting it into play 
> would be helpful I think.
>
> I also still believe that we really should change lookup to use 
> unmarshalled data objects (these changes are visible at 
> http://reef.dev.java.net) so that we can then plug in unmarshalling 
> contexts and metadata transport pieces to make services have enough 
> plug-ability that we can provide for registration and discovery of 
> other types of "enviroments" than just what DiscoveryV1 and V2 provide.
>
> Gregg Wonderly
You know what, your right.  Reggie is implementation specific.  We need 
to change that.

We need a place on svn for River 3.0 where we can make these changes and 
experiment, perhaps using a minimal subset of the current codebase for 
clarity while we experiment.

What components we should initially include?

The AR2 codebase is ready for release apart from some minor changes to 
testing that I haven't quite gotten around to yet, no good holding up 
productivity of fertile minds in the mean time...  Its safe to branch.

Cheers,

Peter.

>
> Calum Shaw-Mackay wrote:
>>> Interesting.
>>>
>>> Perhaps Marshalled object instances should also be decorated.
>>
>> Yeah you'd need somewhere to start, and as I mentioned below about
>> transparency, you'd possibly need a
>> DEFAULT or CURRENT version, because if you're bringing in an object
>> foreign to the VM, and you're using
>> static analysis, you'd need to have the bytecodes available to
>> actually compare it against, having an initial
>> version identifier would give it a head start in terms of performance,
>> and you don't want to brute-force
>> hammer the network bringing in multiple dl jars, for a class that has
>> not been seen in any version in this JVM session
>>
>> I think the more 'hints' you could provide to the classloader, through
>> either compatible version ranges or soft links
>> the better.
>>
>>>> Whichever way you want to tackle this, you'll need it to be as
>>>> transparent a mechanism as possible so that service/bundles/clients
>>>> that don;t specify versions work as they always have,
>>>>
>>> Thanks Calum,  good point, when version metadata doesn't exist, the 
>>> results
>>> from Static Analysis alone could be utilised to find API compatible 
>>> import
>>> packages.  I guess classes lacking version metadata could be treated as
>>> version 0.  If someone later decides to add version metadata, 
>>> perhaps after
>>> correcting a bug, the higher versioned package would be preferred, 
>>> provided
>>> it remains API compatible and shares the same fully qualified 
>>> package name.
>>
>> Yeah I'd say a DEFAULT (i.e. 0, although that itself may go through
>> changes in time) and perhaps symlink-type versioning including a
>> CURRENT.
>>
>> In a way, what you don't want is to break compatibility unnecessarily.
>>
>> Calum
>>
>
>

Re: Cooperative Distributed computing - Versioning over time - Container Metadata.

[Gregg Wonderly <ge...@cox.net>]

My changes to reggie to provide "never preferred classes" changed 
MarshalledWrapper and EntryRep to implement

public interface MarshalledDataAccess<T> {
	public T getUnmarshalledData() throws IOException, InstantiationException,
     		ClassNotFoundException, IllegalAccessException ;
	public List<String> getClassNames();
}

There is still some other work that I need to finalize in these changes to get 
service objects to always have this information (there are complex threads of 
execution and MarshalledObject use).

I made EntryRep work, and that was the big thing that I needed, so I stopped at 
that point.

Creating an interface, such as the above, and putting it into play would be 
helpful I think.

I also still believe that we really should change lookup to use unmarshalled 
data objects (these changes are visible at http://reef.dev.java.net) so that we 
can then plug in unmarshalling contexts and metadata transport pieces to make 
services have enough plug-ability that we can provide for registration and 
discovery of other types of "enviroments" than just what DiscoveryV1 and V2 provide.

Gregg Wonderly

Calum Shaw-Mackay wrote:
>> Interesting.
>>
>> Perhaps Marshalled object instances should also be decorated.
> 
> Yeah you'd need somewhere to start, and as I mentioned below about
> transparency, you'd possibly need a
> DEFAULT or CURRENT version, because if you're bringing in an object
> foreign to the VM, and you're using
> static analysis, you'd need to have the bytecodes available to
> actually compare it against, having an initial
> version identifier would give it a head start in terms of performance,
> and you don't want to brute-force
> hammer the network bringing in multiple dl jars, for a class that has
> not been seen in any version in this JVM session
> 
> I think the more 'hints' you could provide to the classloader, through
> either compatible version ranges or soft links
> the better.
> 
>>> Whichever way you want to tackle this, you'll need it to be as
>>> transparent a mechanism as possible so that service/bundles/clients
>>> that don;t specify versions work as they always have,
>>>
>> Thanks Calum,  good point, when version metadata doesn't exist, the results
>> from Static Analysis alone could be utilised to find API compatible import
>> packages.  I guess classes lacking version metadata could be treated as
>> version 0.  If someone later decides to add version metadata, perhaps after
>> correcting a bug, the higher versioned package would be preferred, provided
>> it remains API compatible and shares the same fully qualified package name.
> 
> Yeah I'd say a DEFAULT (i.e. 0, although that itself may go through
> changes in time) and perhaps symlink-type versioning including a
> CURRENT.
> 
> In a way, what you don't want is to break compatibility unnecessarily.
> 
> Calum
> 

Re: Cooperative Distributed computing - Versioning over time - Container Metadata.

[Calum Shaw-Mackay <ca...@gmail.com>]

>
> Interesting.
>
> Perhaps Marshalled object instances should also be decorated.

Yeah you'd need somewhere to start, and as I mentioned below about
transparency, you'd possibly need a
DEFAULT or CURRENT version, because if you're bringing in an object
foreign to the VM, and you're using
static analysis, you'd need to have the bytecodes available to
actually compare it against, having an initial
version identifier would give it a head start in terms of performance,
and you don't want to brute-force
hammer the network bringing in multiple dl jars, for a class that has
not been seen in any version in this JVM session

I think the more 'hints' you could provide to the classloader, through
either compatible version ranges or soft links
the better.

>>
>> Whichever way you want to tackle this, you'll need it to be as
>> transparent a mechanism as possible so that service/bundles/clients
>> that don;t specify versions work as they always have,
>>
>
> Thanks Calum,  good point, when version metadata doesn't exist, the results
> from Static Analysis alone could be utilised to find API compatible import
> packages.  I guess classes lacking version metadata could be treated as
> version 0.  If someone later decides to add version metadata, perhaps after
> correcting a bug, the higher versioned package would be preferred, provided
> it remains API compatible and shares the same fully qualified package name.

Yeah I'd say a DEFAULT (i.e. 0, although that itself may go through
changes in time) and perhaps symlink-type versioning including a
CURRENT.

In a way, what you don't want is to break compatibility unnecessarily.

Calum

Re: Cooperative Distributed computing - Versioning over time - Container Metadata.

[Peter Firmstone <ji...@zeus.net.au>]

Calum Shaw-Mackay wrote:
> This sounds very like Newton......
>
> Another way might be to provide a new URL Handler and having a Version
> ClassLoader higher up the classloader stack.
>   
I'd like to make the traditional http codebase obsolete, or at a minimum 
have a codebase service act as an intermediatory on its behalf.  
Traditional http codebases are a source of pain. I want something 
simpler.  This would allow utilisation of byte code analysis to build 
class mirrors that the classloading infrastructure could use to resolve 
compatible packages, probably with a version classloader higher up the 
stack as you suggest, which keeps track of where all currently loaded 
packages, like classworlds.
> Although this doesn't really help in terms of attaching the version
> required by a caller when trying to find a particular version, what
> you might need is a decorated
> classloader, that is built on the metadata supplied by the
> service/client, with a version classloader acting as it's parent, so
> when a class is requested by the service/client, it calls the
> decorated classloader, which contains the metadata, which in turn
> requests loadClass(name, version) from the Versioned ClassLoader.
>   
Interesting.

Perhaps Marshalled object instances should also be decorated.
> Whichever way you want to tackle this, you'll need it to be as
> transparent a mechanism as possible so that service/bundles/clients
> that don;t specify versions work as they always have,
>   
Thanks Calum,  good point, when version metadata doesn't exist, the 
results from Static Analysis alone could be utilised to find API 
compatible import packages.  I guess classes lacking version metadata 
could be treated as version 0.  If someone later decides to add version 
metadata, perhaps after correcting a bug, the higher versioned package 
would be preferred, provided it remains API compatible and shares the 
same fully qualified package name. 
> --Calum
>
> 2009/10/20 Peter Firmstone <ji...@zeus.net.au>:
>   
>> N.B.
>>
>> If we were able to utilise OSGi bundles, I can't then see why there couldn't
>> be a way of setting up some kind of Bridge or Facade pattern to interoperate
>> with OSGi.  I'm only hypothesising, I don't have any particular
>> implementation in mind. The advantage here would be the possibility of using
>> common bundle code for both environments.
>>
>> Peter Firmstone wrote:
>>     
>>> Let me put it another way:
>>>
>>> I want to work toward a cooperative distributed computing platform using
>>> Java & Jini / River, continuing from  where Sun left, using lessons learned.
>>>
>>>  1. I'd like to set up a class loader framework building on
>>>     classworlds, to segregate incompatible class versions with the
>>>     same name, while allowing compatible implementations to
>>>     interoperate, based on Class loader visibility.
>>>  2. I'd like to utilise the ASM bytecode library to perform static
>>>     analysis on bytecode to identify Package & Class API to determine
>>>     compatibility.
>>>  3. I'd like to solve the lost code base problem using codebase
>>>     services combined with the suggested Class loading framework.
>>>  4. I'd like to use codebase servcies to provide codebase proxy's for
>>>     often disconnected clients.
>>>  5. I'd like to allow bytecode to evolve organically by taking
>>>     advantage of the flexibility of binary compatibility.
>>>  6. I'd like some metadata to identify at the minimum ,Package
>>>     Versions, I could generate checksums based on Package API, however
>>>     this wouldn't identify which version is the latest, only which
>>>     Package implementations can substitute another's API  where those
>>>     packages share the same name. Or identify packages with the same
>>>     name that are incompatible and segregate these in separate
>>>     Classloaders.
>>>  7. If possible due to concerns about the performance impact of many
>>>     Classloaders, group some packages together when the author doesn't
>>>     intend those packages to evolve separately.  Eg libraries.
>>>  8. I'd like to make it possible to transfer class version and package
>>>     version information with serialized object instances for
>>>     compatible re-instantiation with the latest evolution of
>>>     compatible classes.  All suggestions are welcome.
>>>
>>> Does anyone have any suggestions for suitable existing formats for
>>> Containers with Version Metadata that might be utilised?  If so what issues
>>> are there to be aware of?  What are the arguments for and Against?
>>>
>>> Note, I'm aware of the existing Java Version Spec that applies to
>>> packages.
>>>
>>> I believe I made the mistake of suggesting a solution without properly
>>> formulating or posing the problem, my apologies for any confusion this might
>>> have caused, I should have raised a new thread / topic,   In the original
>>> thread, Gregg is looking at classloader dependency issues implementing a
>>> plugin for Netbeans to make River more accessible to IDE developers, please
>>> feel free to make suggestions, we need your help.
>>>
>>> I made a suggestion earlier to utilise the OSGi bundle metadata
>>> specification, these are ordinary jar files with package version, exports
>>> and dependency metadata.  They are otherwise ordinary plain old jar
>>> archives,  quite useable as a jar file containing java class files that can
>>> be utilised without  the OSGi framework, which is what I was suggesting. I
>>> attempted to imply that the exported packages could have their API
>>> identified and shared in a compatible manner with other packages, using
>>> Class loader isolation perhaps this is not a suitable solution due to the
>>> confusion that it would cause with developers expecting an OSGi
>>> implementation, utilising OSGi service interfaces.  If this format was to be
>>> utilised, it might have to be renamed, to River Codebase Bundles (for
>>> uploading to codebases) whilst sharing the same format to avoid confusion.
>>> It might be a subset of the OSGi bundle format, such that OSGi bundles, like
>>> those on the Spring Source Repository could be utilised.  It might also be a
>>> component in an eventual Container spec, if we can sort that out.  Gregg has
>>> asked for container suggestions more than once, I'd like to see that
>>> conversation continued.
>>>
>>> If this solution was adopted, the dependency version metadata (Import
>>> Package) in the bundle would be noted, considered a minimum version as per
>>> the OSGi spec, but a later version would be utilized if it were found API
>>> compatible.  However additional protection by static analysis would
>>> guarantee that a later version that wasn't API compatible would not be
>>> utilised. The reason I'm considering  this compromise is that a bundle
>>> author cannot know in advance that later import packages will be compatible,
>>> hence Static Analysis can verify API compatibility.  It is still possible
>>> for identical named classes to be loaded in the same jvm, in another
>>> classloader, this might occur if an earlier package version already exists
>>> but doesn't satisfy the Import Package version criteria; it wont be used as
>>> it is earlier than the minimum required version.  The Classloader
>>> responsible for the bundle would keep a copy of the  metadata available
>>> while it remains loaded, it would not utilise the OSGi interfaces to access
>>> the metadata.  I envision Immutable Mirror Objects that can be transferred
>>> with marshalled instances and utilised for API compatibility checking.  The
>>> Mirror objects would be created by a codebase service and stored in memory
>>> by the Classloader that download the class bytecode (River Bundles) being
>>> Mirrored,  the mirror objects would contain the metadata from the bundle as
>>> well as the API, the mirror objects would be used by the classloader to
>>> determine which packages to export as well as check for import package
>>> compatibility, and by other Classloader's for compatibility.  If no
>>> compatible packages are found loaded then the codebase services would be
>>> consulted, the import package mirror object would be submitted to the
>>> codebase services to locate a compatible package or bundle.
>>>
>>> In principle, if this is still objectionable, I'm happy to create an
>>> implementation utilising another container or bundle format that satisfies
>>> the requirements, provided that one is recommended.  This list does have a
>>> history of heated debate regarding OSGi and Jini, I'd like to avoid any such
>>> unproductive arguments if possible.
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>>
>>> Niclas Hedhman wrote:
>>>       
>>>> On Mon, Oct 19, 2009 at 7:40 AM, Peter Firmstone <ji...@zeus.net.au>
>>>> wrote:
>>>>
>>>>         
>>>>>> Thanks Nic for your comments, pls see reply below.
>>>>>>
>>>>>>             
>>>> I think we have diverged and discussing different things.
>>>>
>>>> Some time ago, it was said that the classloader of a class could be
>>>> retrieved by calling a static method. I have tried to point out that
>>>> this is fairly "impossible" in OSGi without collaboration with the
>>>> OSGi framework implementation, as multiple (even fully compatible, or
>>>> even exact same versions) instance of the same classes can be present
>>>> in different class spaces within the OSGi framework, and which package
>>>> got resolved to which bundle (and hence the classloader) is an
>>>> implementation detail in OSGi. There is no theory around that fact. I
>>>> am speaking of "locating a classloader of an existing class already
>>>> present", and not how to locate one externally to the JVM and OSGi
>>>> framework.
>>>>
>>>> See the difference?
>>>>
>>>> So if ClassA wants to load an implementation of InterfaceB, then that
>>>> implementation must be chosen from the class space that ClassA and
>>>> InterfaceB belongs to, and not only that it implements
>>>> InterfaceB+constraints.
>>>>
>>>>
>>>> Cheers
>>>>
>>>>         
>>>       
>>     
>
>   

Re: Cooperative Distributed computing - Versioning over time - Container Metadata.

[Calum Shaw-Mackay <ca...@gmail.com>]

This sounds very like Newton......

Another way might be to provide a new URL Handler and having a Version
ClassLoader higher up the classloader stack.

Although this doesn't really help in terms of attaching the version
required by a caller when trying to find a particular version, what
you might need is a decorated
classloader, that is built on the metadata supplied by the
service/client, with a version classloader acting as it's parent, so
when a class is requested by the service/client, it calls the
decorated classloader, which contains the metadata, which in turn
requests loadClass(name, version) from the Versioned ClassLoader.

Whichever way you want to tackle this, you'll need it to be as
transparent a mechanism as possible so that service/bundles/clients
that don;t specify versions work as they always have,

--Calum

2009/10/20 Peter Firmstone <ji...@zeus.net.au>:
> N.B.
>
> If we were able to utilise OSGi bundles, I can't then see why there couldn't
> be a way of setting up some kind of Bridge or Facade pattern to interoperate
> with OSGi.  I'm only hypothesising, I don't have any particular
> implementation in mind. The advantage here would be the possibility of using
> common bundle code for both environments.
>
> Peter Firmstone wrote:
>>
>> Let me put it another way:
>>
>> I want to work toward a cooperative distributed computing platform using
>> Java & Jini / River, continuing from  where Sun left, using lessons learned.
>>
>>  1. I'd like to set up a class loader framework building on
>>     classworlds, to segregate incompatible class versions with the
>>     same name, while allowing compatible implementations to
>>     interoperate, based on Class loader visibility.
>>  2. I'd like to utilise the ASM bytecode library to perform static
>>     analysis on bytecode to identify Package & Class API to determine
>>     compatibility.
>>  3. I'd like to solve the lost code base problem using codebase
>>     services combined with the suggested Class loading framework.
>>  4. I'd like to use codebase servcies to provide codebase proxy's for
>>     often disconnected clients.
>>  5. I'd like to allow bytecode to evolve organically by taking
>>     advantage of the flexibility of binary compatibility.
>>  6. I'd like some metadata to identify at the minimum ,Package
>>     Versions, I could generate checksums based on Package API, however
>>     this wouldn't identify which version is the latest, only which
>>     Package implementations can substitute another's API  where those
>>     packages share the same name. Or identify packages with the same
>>     name that are incompatible and segregate these in separate
>>     Classloaders.
>>  7. If possible due to concerns about the performance impact of many
>>     Classloaders, group some packages together when the author doesn't
>>     intend those packages to evolve separately.  Eg libraries.
>>  8. I'd like to make it possible to transfer class version and package
>>     version information with serialized object instances for
>>     compatible re-instantiation with the latest evolution of
>>     compatible classes.  All suggestions are welcome.
>>
>> Does anyone have any suggestions for suitable existing formats for
>> Containers with Version Metadata that might be utilised?  If so what issues
>> are there to be aware of?  What are the arguments for and Against?
>>
>> Note, I'm aware of the existing Java Version Spec that applies to
>> packages.
>>
>> I believe I made the mistake of suggesting a solution without properly
>> formulating or posing the problem, my apologies for any confusion this might
>> have caused, I should have raised a new thread / topic,   In the original
>> thread, Gregg is looking at classloader dependency issues implementing a
>> plugin for Netbeans to make River more accessible to IDE developers, please
>> feel free to make suggestions, we need your help.
>>
>> I made a suggestion earlier to utilise the OSGi bundle metadata
>> specification, these are ordinary jar files with package version, exports
>> and dependency metadata.  They are otherwise ordinary plain old jar
>> archives,  quite useable as a jar file containing java class files that can
>> be utilised without  the OSGi framework, which is what I was suggesting. I
>> attempted to imply that the exported packages could have their API
>> identified and shared in a compatible manner with other packages, using
>> Class loader isolation perhaps this is not a suitable solution due to the
>> confusion that it would cause with developers expecting an OSGi
>> implementation, utilising OSGi service interfaces.  If this format was to be
>> utilised, it might have to be renamed, to River Codebase Bundles (for
>> uploading to codebases) whilst sharing the same format to avoid confusion.
>> It might be a subset of the OSGi bundle format, such that OSGi bundles, like
>> those on the Spring Source Repository could be utilised.  It might also be a
>> component in an eventual Container spec, if we can sort that out.  Gregg has
>> asked for container suggestions more than once, I'd like to see that
>> conversation continued.
>>
>> If this solution was adopted, the dependency version metadata (Import
>> Package) in the bundle would be noted, considered a minimum version as per
>> the OSGi spec, but a later version would be utilized if it were found API
>> compatible.  However additional protection by static analysis would
>> guarantee that a later version that wasn't API compatible would not be
>> utilised. The reason I'm considering  this compromise is that a bundle
>> author cannot know in advance that later import packages will be compatible,
>> hence Static Analysis can verify API compatibility.  It is still possible
>> for identical named classes to be loaded in the same jvm, in another
>> classloader, this might occur if an earlier package version already exists
>> but doesn't satisfy the Import Package version criteria; it wont be used as
>> it is earlier than the minimum required version.  The Classloader
>> responsible for the bundle would keep a copy of the  metadata available
>> while it remains loaded, it would not utilise the OSGi interfaces to access
>> the metadata.  I envision Immutable Mirror Objects that can be transferred
>> with marshalled instances and utilised for API compatibility checking.  The
>> Mirror objects would be created by a codebase service and stored in memory
>> by the Classloader that download the class bytecode (River Bundles) being
>> Mirrored,  the mirror objects would contain the metadata from the bundle as
>> well as the API, the mirror objects would be used by the classloader to
>> determine which packages to export as well as check for import package
>> compatibility, and by other Classloader's for compatibility.  If no
>> compatible packages are found loaded then the codebase services would be
>> consulted, the import package mirror object would be submitted to the
>> codebase services to locate a compatible package or bundle.
>>
>> In principle, if this is still objectionable, I'm happy to create an
>> implementation utilising another container or bundle format that satisfies
>> the requirements, provided that one is recommended.  This list does have a
>> history of heated debate regarding OSGi and Jini, I'd like to avoid any such
>> unproductive arguments if possible.
>>
>> Regards,
>>
>> Peter.
>>
>>
>> Niclas Hedhman wrote:
>>>
>>> On Mon, Oct 19, 2009 at 7:40 AM, Peter Firmstone <ji...@zeus.net.au>
>>> wrote:
>>>
>>>>>
>>>>> Thanks Nic for your comments, pls see reply below.
>>>>>
>>>
>>> I think we have diverged and discussing different things.
>>>
>>> Some time ago, it was said that the classloader of a class could be
>>> retrieved by calling a static method. I have tried to point out that
>>> this is fairly "impossible" in OSGi without collaboration with the
>>> OSGi framework implementation, as multiple (even fully compatible, or
>>> even exact same versions) instance of the same classes can be present
>>> in different class spaces within the OSGi framework, and which package
>>> got resolved to which bundle (and hence the classloader) is an
>>> implementation detail in OSGi. There is no theory around that fact. I
>>> am speaking of "locating a classloader of an existing class already
>>> present", and not how to locate one externally to the JVM and OSGi
>>> framework.
>>>
>>> See the difference?
>>>
>>> So if ClassA wants to load an implementation of InterfaceB, then that
>>> implementation must be chosen from the class space that ClassA and
>>> InterfaceB belongs to, and not only that it implements
>>> InterfaceB+constraints.
>>>
>>>
>>> Cheers
>>>
>>
>>
>
>

Re: Cooperative Distributed computing - Versioning over time - Container Metadata.

[Peter Firmstone <ji...@zeus.net.au>]

N.B.

If we were able to utilise OSGi bundles, I can't then see why there 
couldn't be a way of setting up some kind of Bridge or Facade pattern to 
interoperate with OSGi.  I'm only hypothesising, I don't have any 
particular implementation in mind. The advantage here would be the 
possibility of using common bundle code for both environments.

Peter Firmstone wrote:
> Let me put it another way:
>
> I want to work toward a cooperative distributed computing platform 
> using Java & Jini / River, continuing from  where Sun left, using 
> lessons learned.
>
>   1. I'd like to set up a class loader framework building on
>      classworlds, to segregate incompatible class versions with the
>      same name, while allowing compatible implementations to
>      interoperate, based on Class loader visibility.
>   2. I'd like to utilise the ASM bytecode library to perform static
>      analysis on bytecode to identify Package & Class API to determine
>      compatibility.
>   3. I'd like to solve the lost code base problem using codebase
>      services combined with the suggested Class loading framework.
>   4. I'd like to use codebase servcies to provide codebase proxy's for
>      often disconnected clients.
>   5. I'd like to allow bytecode to evolve organically by taking
>      advantage of the flexibility of binary compatibility.
>   6. I'd like some metadata to identify at the minimum ,Package
>      Versions, I could generate checksums based on Package API, however
>      this wouldn't identify which version is the latest, only which
>      Package implementations can substitute another's API  where those
>      packages share the same name. Or identify packages with the same
>      name that are incompatible and segregate these in separate
>      Classloaders.
>   7. If possible due to concerns about the performance impact of many
>      Classloaders, group some packages together when the author doesn't
>      intend those packages to evolve separately.  Eg libraries.
>   8. I'd like to make it possible to transfer class version and package
>      version information with serialized object instances for
>      compatible re-instantiation with the latest evolution of
>      compatible classes.  All suggestions are welcome.
>
> Does anyone have any suggestions for suitable existing formats for 
> Containers with Version Metadata that might be utilised?  
> If so what issues are there to be aware of?  What are the arguments 
> for and Against?
>
> Note, I'm aware of the existing Java Version Spec that applies to 
> packages.
>
> I believe I made the mistake of suggesting a solution without properly 
> formulating or posing the problem, my apologies for any confusion this 
> might have caused, I should have raised a new thread / topic,   In the 
> original thread, Gregg is looking at classloader dependency issues 
> implementing a plugin for Netbeans to make River more accessible to 
> IDE developers, please feel free to make suggestions, we need your help.
>
> I made a suggestion earlier to utilise the OSGi bundle metadata 
> specification, these are ordinary jar files with package version, 
> exports and dependency metadata.  They are otherwise ordinary plain 
> old jar archives,  quite useable as a jar file containing java class 
> files that can be utilised without  the OSGi framework, which is what 
> I was suggesting. I attempted to imply that the exported packages 
> could have their API identified and shared in a compatible manner with 
> other packages, using Class loader isolation perhaps this is not a 
> suitable solution due to the confusion that it would cause with 
> developers expecting an OSGi implementation, utilising OSGi service 
> interfaces.  If this format was to be utilised, it might have to be 
> renamed, to River Codebase Bundles (for uploading to codebases) whilst 
> sharing the same format to avoid confusion.   It might be a subset of 
> the OSGi bundle format, such that OSGi bundles, like those on the 
> Spring Source Repository could be utilised.  It might also be a 
> component in an eventual Container spec, if we can sort that out.  
> Gregg has asked for container suggestions more than once, I'd like to 
> see that conversation continued.
>
> If this solution was adopted, the dependency version metadata (Import 
> Package) in the bundle would be noted, considered a minimum version as 
> per the OSGi spec, but a later version would be utilized if it were 
> found API compatible.  However additional protection by static 
> analysis would guarantee that a later version that wasn't API 
> compatible would not be utilised. The reason I'm considering  this 
> compromise is that a bundle author cannot know in advance that later 
> import packages will be compatible, hence Static Analysis can verify 
> API compatibility.  It is still possible for identical named classes 
> to be loaded in the same jvm, in another classloader, this might occur 
> if an earlier package version already exists but doesn't satisfy the 
> Import Package version criteria; it wont be used as it is earlier than 
> the minimum required version.  The Classloader responsible for the 
> bundle would keep a copy of the  metadata available while it remains 
> loaded, it would not utilise the OSGi interfaces to access the 
> metadata.  I envision Immutable Mirror Objects that can be transferred 
> with marshalled instances and utilised for API compatibility 
> checking.  The Mirror objects would be created by a codebase service 
> and stored in memory by the Classloader that download the class 
> bytecode (River Bundles) being Mirrored,  the mirror objects would 
> contain the metadata from the bundle as well as the API, the mirror 
> objects would be used by the classloader to determine which packages 
> to export as well as check for import package compatibility, and by 
> other Classloader's for compatibility.  If no compatible packages are 
> found loaded then the codebase services would be consulted, the import 
> package mirror object would be submitted to the codebase services to 
> locate a compatible package or bundle.
>
> In principle, if this is still objectionable, I'm happy to create an 
> implementation utilising another container or bundle format that 
> satisfies the requirements, provided that one is recommended.  This 
> list does have a history of heated debate regarding OSGi and Jini, I'd 
> like to avoid any such unproductive arguments if possible.
>
> Regards,
>
> Peter.
>
>
> Niclas Hedhman wrote:
>> On Mon, Oct 19, 2009 at 7:40 AM, Peter Firmstone <ji...@zeus.net.au> 
>> wrote:
>>  
>>>> Thanks Nic for your comments, pls see reply below.
>>>>       
>>
>> I think we have diverged and discussing different things.
>>
>> Some time ago, it was said that the classloader of a class could be
>> retrieved by calling a static method. I have tried to point out that
>> this is fairly "impossible" in OSGi without collaboration with the
>> OSGi framework implementation, as multiple (even fully compatible, or
>> even exact same versions) instance of the same classes can be present
>> in different class spaces within the OSGi framework, and which package
>> got resolved to which bundle (and hence the classloader) is an
>> implementation detail in OSGi. There is no theory around that fact. I
>> am speaking of "locating a classloader of an existing class already
>> present", and not how to locate one externally to the JVM and OSGi
>> framework.
>>
>> See the difference?
>>
>> So if ClassA wants to load an implementation of InterfaceB, then that
>> implementation must be chosen from the class space that ClassA and
>> InterfaceB belongs to, and not only that it implements
>> InterfaceB+constraints.
>>
>>
>> Cheers
>>   
>
>