Trying to export packages from the Class Path

[Jeremy Wales <je...@gmail.com>]

Hi,

I'm trying to come up with a way to expose all the packages which happen to
be available on the Class Path as proper OSGi package exports. Inspired by
Chapter 9 Migration in the JSR 277 Early Draft, which defines a
java.classpath module so modules can import from legacy JARs, I thought
something similar should be available in OSGi, but that in OSGi it could
actually be much better.

The beauty of exporting packages from the Class Path would be that client
bundles of legacy JARs could use them the proper way, via package imports,
and then when the legacy JARs finally get a proper OSGi manifest they can be
used in exactly the same way, with no refactoring of the client bundles.
This contrasts to the JSR 277 proposal which would let you use legacy JARs
via a java.classpath module but then you have to refactor all the client
modules when a legacy JAR becomes a proper module. Felix Commons seems to
plug a similar gap (for OSGi) but requires explicitly wrapping each legacy
JAR via Maven at build time whereas I (and I'd imagine many others) would
love the option to have similar functionality provided transparently at
runtime.

So, working out what packages are available on the Class Path, and the other
packages each uses, isn't hard (I've got something working and it seems to
be fast enough) but then they need to be exported somehow. Reading the OSGi
spec, it seems there should be several possible approaches:

1)      Add these packages to the
org.osgi.framework.system.packagesframework property.

This has the draw back that it relies on the property being set as part of
the framework bootstrapping process which is always proprietary because it's
not covered by the spec. I would like a solution to be implementation
agnostic.

Another disadvantage is that the packages are not substitutable via an
equivalent import definition (as per 3.5.6 Exporting and Importing a Package
of the OSGi spec).

2)      Install and start a bundle which then installs a framework extension
bundle which just has a manifest with the Class Path export package
definition included.

This has the same draw back that an extension bundle cannot have imports,
which I would want for substitutability.

It also has the disadvantage that it doesn't work in version 1.0.0 of Felix!
When debugging in Felix I found that, although an extension bundle is
identified as such so that its exports are added to the ExtensionManager,
when the R4SearchPolicyCore is invoked during the resolution of other
bundles it doesn't consult the ExtensionManager and the exports added to the
ExtensionManager are not added to the ones searched by R4SearchPolicyCore.
So bundles that require packages exported only by an extension bundle fail
to be resolved. This looks to me like a bug in Felix but I don't know Felix
well enough to be sure (but exactly the same thing happens to work in
Equinox).

3)      Install and start a bundle which installs a fragment bundle, with
the Class Path exports in its manifest, which has system.bundle as its
fragment host but don't make it an extension bundle, just a normal fragment
bundle.

This would have the advantage that it should allow imports as well as
exports. I can make the import list mirror the export list and then remove
any imports that conflict with those of system.bundle (including its
currently attached fragments) and then as the framework resolves the new
fragment it can decide if it should wire in any existing exports from other
bundles as substitutes for those from the Class Path.

This works brilliantly throughout the resolution process in Felix! I
actually wasn't expecting resolution to work at all because the outstanding
http://issues.apache.org/jira/browse/FELIX-29 seems to indicate that
fragments aren't supported at all? However, even though it appears to work,
it does fall over when a client bundle of one of the Class Path exports
actually tries to load a Class from a wire to system.bundle. It seems that
in Felix the special ability of the system bundle to export packages from
its parent ClassLoader is actually restricted to those exports specified by
the system bundle JAR itself, the
org.osgi.framework.system.packagesproperty and any framework extension
bundles. For a non-extension fragment
bundle attached to system.bundle Felix doesn't delegate the wire to the
parent ClassLoader but instead only searches in the system.bundle JAR itself
and therefore doesn't find the Class. This isn't the behaviour I would
expect from reading the OSGi spec (and it's not the behaviour of Equinox
either, as it can delegate any system.bundle exports to the parent
ClassLoader no matter where they come from… it can't however handle
additional imports (even optionals) to an already resolved bundle like Felix
seems to… which I'll have to post to their dev list).

So, I'm wondering, would it be possible for Felix to change this behaviour
so that this use case works?

Re: Trying to export packages from the Class Path

["Richard S. Hall" <he...@ungoverned.org>]

Jeremy Wales wrote:
> Thanks for your response, Richard.
>
>
>   
>> So, working out what packages are available on the Class Path, and the
>> other
>>     
>>> packages each uses, isn't hard (I've got something working and it seems
>>>       
>> to
>>     
>>> be fast enough) but then they need to be exported somehow. Reading the
>>>       
>> OSGi
>>     
>>> spec, it seems there should be several possible approaches:
>>>
>>> 1)      Add these packages to the
>>> org.osgi.framework.system.packagesframework property.
>>>
>>> This has the draw back that it relies on the property being set as part
>>>       
>> of
>>     
>>> the framework bootstrapping process which is always proprietary because
>>>       
>> it's
>>     
>>> not covered by the spec. I would like a solution to be implementation
>>> agnostic.
>>>
>>>       
>> This might change in the next spec...
>>     
>
>
>  OK, I'll watch that with interest, but for now...
>
>   
>> Another disadvantage is that the packages are not substitutable via an
>>     
>>> equivalent import definition (as per 3.5.6 Exporting and Importing a
>>>       
>> Package
>>     
>>> of the OSGi spec).
>>>
>>>       
>> I am not sure what you mean here, it is substitutable for consumers. The
>> framework doesn't use these packages, so it doesn't matter if it is
>> substitutable from the framework's perspective.
>>     
>
>
> What I mean (and I may be misunderstanding how things work) is that if (for
> example) I make the system bundle export packages P and Q and P is said to use
> Q, then unless the system bundle also imports package Q then, for any
> importer of both packages P and Q, if the framework wires P from the system
> bundle it will also have to wire Q from the system bundle, even if there
> might be another, "better" export of Q already in use by other bundles.
> However, if the system bundle can be made to import package Q, then it can
> be wired with the Q package from the "better" export already in use and
> export that, thus creating a greater chance of class space consistency. If
> that understanding is correct (and feel free to correct me if it isn't!)
> then being able to add imports at the same time as exports to the system
> bundle would be desirable.
>   

If you are talking about "uses" constraints, then yes if the system 
bundle exports have "uses" constraints then importers of those packages 
will be constrained by them. However, the system bundle still never 
imports anything and even if it did, it is resolved before any other 
bundles are installed so you'd still not get what you expect, I would 
imagine.

>> 2)      Install and start a bundle which then installs a framework
>> extension
>>     
>>> bundle which just has a manifest with the Class Path export package
>>> definition included.
>>>
>>> This has the same draw back that an extension bundle cannot have
>>>       
>> imports,
>>     
>>> which I would want for substitutability.
>>>
>>> It also has the disadvantage that it doesn't work in version 1.0.0 of
>>>       
>> Felix!
>>     
>>> When debugging in Felix I found that, although an extension bundle is
>>> identified as such so that its exports are added to the
>>>       
>> ExtensionManager,
>>     
>>> when the R4SearchPolicyCore is invoked during the resolution of other
>>> bundles it doesn't consult the ExtensionManager and the exports added to
>>>       
>> the
>>     
>>> ExtensionManager are not added to the ones searched by
>>>       
>> R4SearchPolicyCore.
>>     
>>> So bundles that require packages exported only by an extension bundle
>>>       
>> fail
>>     
>>> to be resolved. This looks to me like a bug in Felix but I don't know
>>>       
>> Felix
>>     
>>> well enough to be sure (but exactly the same thing happens to work in
>>> Equinox).
>>>
>>>       
>> Yes, this sounds like a shortcoming in the current non-complete
>> implementation of this feature.
>>     
>
>
> Fair enough. Would it be helpful to have a JIRA raised for this or will it
> already be in the spotlight for future work on extension bundles?
>   

I think a JIRA issue linked to the original would be a good idea...

>> 3)      Install and start a bundle which installs a fragment bundle, with
>>     
>>> the Class Path exports in its manifest, which has system.bundle as its
>>> fragment host but don't make it an extension bundle, just a normal
>>>       
>> fragment
>>     
>>> bundle.
>>>
>>> This would have the advantage that it should allow imports as well as
>>> exports. I can make the import list mirror the export list and then
>>>       
>> remove
>>     
>>> any imports that conflict with those of system.bundle (including its
>>> currently attached fragments) and then as the framework resolves the new
>>> fragment it can decide if it should wire in any existing exports from
>>>       
>> other
>>     
>>> bundles as substitutes for those from the Class Path.
>>>
>>> This works brilliantly throughout the resolution process in Felix! I
>>> actually wasn't expecting resolution to work at all because the
>>>       
>> outstanding
>>     
>>> http://issues.apache.org/jira/browse/FELIX-29 seems to indicate that
>>> fragments aren't supported at all? However, even though it appears to
>>>       
>> work,
>>     
>>> it does fall over when a client bundle of one of the Class Path exports
>>> actually tries to load a Class from a wire to system.bundle. It seems
>>>       
>> that
>>     
>>> in Felix the special ability of the system bundle to export packages
>>>       
>> from
>>     
>>> its parent ClassLoader is actually restricted to those exports specified
>>>       
>> by
>>     
>>> the system bundle JAR itself, the
>>> org.osgi.framework.system.packagesproperty and any framework extension
>>> bundles. For a non-extension fragment
>>> bundle attached to system.bundle Felix doesn't delegate the wire to the
>>> parent ClassLoader but instead only searches in the system.bundle JAR
>>>       
>> itself
>>     
>>> and therefore doesn't find the Class. This isn't the behaviour I would
>>> expect from reading the OSGi spec (and it's not the behaviour of Equinox
>>> either, as it can delegate any system.bundle exports to the parent
>>> ClassLoader no matter where they come from… it can't however handle
>>> additional imports (even optionals) to an already resolved bundle like
>>>       
>> Felix
>>     
>>> seems to… which I'll have to post to their dev list).
>>>
>>>       
>> I was under the impression that Felix currently supports fragments that
>> add to the system bundle exports, however, this is not a widely tested
>> feature. If you want to create a JIRA issue with a simple example to
>> demonstrate the issue, we can look into it.
>>     
>
>
> My experience is that it certainly lets me add to the system bundle exports,
> and those exports are utilised in resolving other bundles. However, when
> Bundle#loadClass(Class) is called for a Class in one of those exported
> packages wired from the system bundle, nothing is found. I'll try to write a
> minimal test to expose this (or expose that it's something else I'm doing
> wrong!) and raise a JIRA if it's really a problem.
>   

Ok. Thanks. I think we had some of this working at one time, but perhaps 
it stopped working when we made changes to the resolver to cache some 
data for performance reasons.

-> richard

Re: Trying to export packages from the Class Path

[Jeremy Wales <je...@gmail.com>]

Thanks for your response, Richard.


> So, working out what packages are available on the Class Path, and the
> other
> > packages each uses, isn't hard (I've got something working and it seems
> to
> > be fast enough) but then they need to be exported somehow. Reading the
> OSGi
> > spec, it seems there should be several possible approaches:
> >
> > 1)      Add these packages to the
> > org.osgi.framework.system.packagesframework property.
> >
> > This has the draw back that it relies on the property being set as part
> of
> > the framework bootstrapping process which is always proprietary because
> it's
> > not covered by the spec. I would like a solution to be implementation
> > agnostic.
> >
>
> This might change in the next spec...


 OK, I'll watch that with interest, but for now...

> Another disadvantage is that the packages are not substitutable via an
> > equivalent import definition (as per 3.5.6 Exporting and Importing a
> Package
> > of the OSGi spec).
> >
>
> I am not sure what you mean here, it is substitutable for consumers. The
> framework doesn't use these packages, so it doesn't matter if it is
> substitutable from the framework's perspective.


What I mean (and I may be misunderstanding how things work) is that if (for
example) I make the system bundle export packages P and Q and P is said to use
Q, then unless the system bundle also imports package Q then, for any
importer of both packages P and Q, if the framework wires P from the system
bundle it will also have to wire Q from the system bundle, even if there
might be another, "better" export of Q already in use by other bundles.
However, if the system bundle can be made to import package Q, then it can
be wired with the Q package from the "better" export already in use and
export that, thus creating a greater chance of class space consistency. If
that understanding is correct (and feel free to correct me if it isn't!)
then being able to add imports at the same time as exports to the system
bundle would be desirable.

> 2)      Install and start a bundle which then installs a framework
> extension
> > bundle which just has a manifest with the Class Path export package
> > definition included.
> >
> > This has the same draw back that an extension bundle cannot have
> imports,
> > which I would want for substitutability.
> >
> > It also has the disadvantage that it doesn't work in version 1.0.0 of
> Felix!
> > When debugging in Felix I found that, although an extension bundle is
> > identified as such so that its exports are added to the
> ExtensionManager,
> > when the R4SearchPolicyCore is invoked during the resolution of other
> > bundles it doesn't consult the ExtensionManager and the exports added to
> the
> > ExtensionManager are not added to the ones searched by
> R4SearchPolicyCore.
> > So bundles that require packages exported only by an extension bundle
> fail
> > to be resolved. This looks to me like a bug in Felix but I don't know
> Felix
> > well enough to be sure (but exactly the same thing happens to work in
> > Equinox).
> >
>
> Yes, this sounds like a shortcoming in the current non-complete
> implementation of this feature.


Fair enough. Would it be helpful to have a JIRA raised for this or will it
already be in the spotlight for future work on extension bundles?

> 3)      Install and start a bundle which installs a fragment bundle, with
> > the Class Path exports in its manifest, which has system.bundle as its
> > fragment host but don't make it an extension bundle, just a normal
> fragment
> > bundle.
> >
> > This would have the advantage that it should allow imports as well as
> > exports. I can make the import list mirror the export list and then
> remove
> > any imports that conflict with those of system.bundle (including its
> > currently attached fragments) and then as the framework resolves the new
> > fragment it can decide if it should wire in any existing exports from
> other
> > bundles as substitutes for those from the Class Path.
> >
> > This works brilliantly throughout the resolution process in Felix! I
> > actually wasn't expecting resolution to work at all because the
> outstanding
> > http://issues.apache.org/jira/browse/FELIX-29 seems to indicate that
> > fragments aren't supported at all? However, even though it appears to
> work,
> > it does fall over when a client bundle of one of the Class Path exports
> > actually tries to load a Class from a wire to system.bundle. It seems
> that
> > in Felix the special ability of the system bundle to export packages
> from
> > its parent ClassLoader is actually restricted to those exports specified
> by
> > the system bundle JAR itself, the
> > org.osgi.framework.system.packagesproperty and any framework extension
> > bundles. For a non-extension fragment
> > bundle attached to system.bundle Felix doesn't delegate the wire to the
> > parent ClassLoader but instead only searches in the system.bundle JAR
> itself
> > and therefore doesn't find the Class. This isn't the behaviour I would
> > expect from reading the OSGi spec (and it's not the behaviour of Equinox
> > either, as it can delegate any system.bundle exports to the parent
> > ClassLoader no matter where they come from… it can't however handle
> > additional imports (even optionals) to an already resolved bundle like
> Felix
> > seems to… which I'll have to post to their dev list).
> >
>
> I was under the impression that Felix currently supports fragments that
> add to the system bundle exports, however, this is not a widely tested
> feature. If you want to create a JIRA issue with a simple example to
> demonstrate the issue, we can look into it.


My experience is that it certainly lets me add to the system bundle exports,
and those exports are utilised in resolving other bundles. However, when
Bundle#loadClass(Class) is called for a Class in one of those exported
packages wired from the system bundle, nothing is found. I'll try to write a
minimal test to expose this (or expose that it's something else I'm doing
wrong!) and raise a JIRA if it's really a problem.


Jeremy

Re: Trying to export packages from the Class Path

["Richard S. Hall" <he...@ungoverned.org>]

Jeremy Wales wrote:
> Hi,
>
> I'm trying to come up with a way to expose all the packages which happen to
> be available on the Class Path as proper OSGi package exports. Inspired by
> Chapter 9 Migration in the JSR 277 Early Draft, which defines a
> java.classpath module so modules can import from legacy JARs, I thought
> something similar should be available in OSGi, but that in OSGi it could
> actually be much better.
>
> The beauty of exporting packages from the Class Path would be that client
> bundles of legacy JARs could use them the proper way, via package imports,
> and then when the legacy JARs finally get a proper OSGi manifest they can be
> used in exactly the same way, with no refactoring of the client bundles.
> This contrasts to the JSR 277 proposal which would let you use legacy JARs
> via a java.classpath module but then you have to refactor all the client
> modules when a legacy JAR becomes a proper module. Felix Commons seems to
> plug a similar gap (for OSGi) but requires explicitly wrapping each legacy
> JAR via Maven at build time whereas I (and I'd imagine many others) would
> love the option to have similar functionality provided transparently at
> runtime.
>
> So, working out what packages are available on the Class Path, and the other
> packages each uses, isn't hard (I've got something working and it seems to
> be fast enough) but then they need to be exported somehow. Reading the OSGi
> spec, it seems there should be several possible approaches:
>
> 1)      Add these packages to the
> org.osgi.framework.system.packagesframework property.
>
> This has the draw back that it relies on the property being set as part of
> the framework bootstrapping process which is always proprietary because it's
> not covered by the spec. I would like a solution to be implementation
> agnostic.
>   

This might change in the next spec...

> Another disadvantage is that the packages are not substitutable via an
> equivalent import definition (as per 3.5.6 Exporting and Importing a Package
> of the OSGi spec).
>   

I am not sure what you mean here, it is substitutable for consumers. The 
framework doesn't use these packages, so it doesn't matter if it is 
substitutable from the framework's perspective.

> 2)      Install and start a bundle which then installs a framework extension
> bundle which just has a manifest with the Class Path export package
> definition included.
>
> This has the same draw back that an extension bundle cannot have imports,
> which I would want for substitutability.
>
> It also has the disadvantage that it doesn't work in version 1.0.0 of Felix!
> When debugging in Felix I found that, although an extension bundle is
> identified as such so that its exports are added to the ExtensionManager,
> when the R4SearchPolicyCore is invoked during the resolution of other
> bundles it doesn't consult the ExtensionManager and the exports added to the
> ExtensionManager are not added to the ones searched by R4SearchPolicyCore.
> So bundles that require packages exported only by an extension bundle fail
> to be resolved. This looks to me like a bug in Felix but I don't know Felix
> well enough to be sure (but exactly the same thing happens to work in
> Equinox).
>   

Yes, this sounds like a shortcoming in the current non-complete 
implementation of this feature.

> 3)      Install and start a bundle which installs a fragment bundle, with
> the Class Path exports in its manifest, which has system.bundle as its
> fragment host but don't make it an extension bundle, just a normal fragment
> bundle.
>
> This would have the advantage that it should allow imports as well as
> exports. I can make the import list mirror the export list and then remove
> any imports that conflict with those of system.bundle (including its
> currently attached fragments) and then as the framework resolves the new
> fragment it can decide if it should wire in any existing exports from other
> bundles as substitutes for those from the Class Path.
>
> This works brilliantly throughout the resolution process in Felix! I
> actually wasn't expecting resolution to work at all because the outstanding
> http://issues.apache.org/jira/browse/FELIX-29 seems to indicate that
> fragments aren't supported at all? However, even though it appears to work,
> it does fall over when a client bundle of one of the Class Path exports
> actually tries to load a Class from a wire to system.bundle. It seems that
> in Felix the special ability of the system bundle to export packages from
> its parent ClassLoader is actually restricted to those exports specified by
> the system bundle JAR itself, the
> org.osgi.framework.system.packagesproperty and any framework extension
> bundles. For a non-extension fragment
> bundle attached to system.bundle Felix doesn't delegate the wire to the
> parent ClassLoader but instead only searches in the system.bundle JAR itself
> and therefore doesn't find the Class. This isn't the behaviour I would
> expect from reading the OSGi spec (and it's not the behaviour of Equinox
> either, as it can delegate any system.bundle exports to the parent
> ClassLoader no matter where they come from… it can't however handle
> additional imports (even optionals) to an already resolved bundle like Felix
> seems to… which I'll have to post to their dev list).
>   

I was under the impression that Felix currently supports fragments that 
add to the system bundle exports, however, this is not a widely tested 
feature. If you want to create a JIRA issue with a simple example to 
demonstrate the issue, we can look into it.

Thanks.

-> richard