Non-File based Framework

[Felix Meschberger <fm...@gmail.com>]

Hi all,

I am new to this list and project, so let me introduce quickly: My
name is Felix Meschberger and I work for Day Management AG in Basel.
As an employee of Day, I am a commiter to the Jackrabbit
(http://jackrabbit.apache.org) and contributed the JCR based class
loader and primitive extension framework as well some contributions to
the RMI library. Currently, I am involved in redesigning our flagship
product an am considering multiple options to implement
modularization.

For our product's requirements the OSGi model perfectly fits and the
Felix project is a nice playground. Actually, I currently have two
minor issues with Felix, which I want to post for discussion:

(1) Our product is running as a web application inside any customer
supplied servlet container (Tomcat, WebSphere, WebLogic, Caucho).
Also, we want to execute the product with as minimal OS filesystem
requirement as possible accessing code and content only from a JCR
repository. Unfortunately, out of the box, the Felix framework
requires an OS filesystem. Please find my proposal below.

(2) Bundles are not completely "closed" on framework shutdown. I will
elaborate this matter in a separate mail to keep issues separated.

Regarding Filesystems: I found, that direct filesystem access in the
current Felix project trunk is isolated in only few places.
Additionally, JAR files seem to be accessed through a JarFileX class,
which extends JarFile. So what I did to remove the reliance on the
filesystem, is to define a resource interface (I call it OSGiResource
laking a better name) and a resource factory abstract class (I call it
OSGiResourceFactory for the same reason). The factory provides a
static method to create resource instances from a string path or
location URL. The resource factory implementation to use is defined by
a configuration property. Resources are either created by calling the
resource factory method or creating the resource instance as child of
a another resource.

Probably, it would also make sense to expose the resource factory as a
service for other bundles to be able to use it.

Of course, as a consequence of using a resource class instead of the
java.io.File class, the BundleContext.getDataFile(String) method might
return null, if the required bundle resource cannot be accessed in the
file system.

I would be very interested in any feedback on this idea and whether
you would be interested in a patch.

Thanks and regards
Felix

Re: Non-File based Framework

[Felix Meschberger <Fe...@day.com>]

Hi Richard,

Thanks for the feedback. I am looking forward to your comments.

Enjoy ApacheCon!

Regards
Felix

On 6/27/06, Richard S. Hall <he...@ungoverned.org> wrote:
> Hey Felix,
>
> FYI: I am at ApacheCon right now and will take a look at this next week
> after I return...just didn't want you to think that I was ignoring it.
>
> -> richard
>
>
> Felix Meschberger wrote:
> > HI again,
> >
> > Based upon your recommendations, I tried to implement the abstract
> > BundleCache and BundleArchive strategy as a prototype.
> >
> > I have not cleaned it up properly and there is in fact one problem
> > with it: The BundleArchive constructors initiliaze the instances. When
> > extending the BundleArchive for storage dependent implementations, the
> > constructors of the base class do not yet know anything about the
> > storage, but initialization depends on the storage. My solution is to
> > introduce an init() method, which initializes and ist called by the
> > BundleCache.
> >
> > I do not like this implementation very much as it kind of transfers
> > responsibility for additional intialization to the caller. But in the
> > short time, I did not come up with a better solution.
> >
> > I attach the patch for this prototype along with the File based
> > implementation.
> >
> > What do you think: Could this be a way to go ? Should I post a Jira
> > issue ?
> >
> > Regards and Thanks
> > Felix, the person :-)
> >
> > On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
> >> Felix Meschberger wrote:
> >> > On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
> >> >> In general, any solution for doing stuff like you have suggested, I
> >> >> would hope, should concentrate on using/improving these existing
> >> >> mechanisms rather than creating new ones.
> >> >
> >> > I definitely agree. Yet not being able to ammend BundleCache and
> >> > BundleArchive, I am still required to have file system space - this
> >> > sort of worries me.
> >>
> >> This is the sort of stuff that we can improve upon then. I am not
> >> against making it once again possible to be able configure which
> >> implementation of BundleCache to use. It was just removed because I had
> >> no valid use case. We just need to discuss what is needed and how to
> >> do it.
> >>
> >> -> richard
> >>
>

Re: Non-File based Framework

["Richard S. Hall" <he...@ungoverned.org>]

Hey Felix,

FYI: I am at ApacheCon right now and will take a look at this next week 
after I return...just didn't want you to think that I was ignoring it.

-> richard


Felix Meschberger wrote:
> HI again,
>
> Based upon your recommendations, I tried to implement the abstract
> BundleCache and BundleArchive strategy as a prototype.
>
> I have not cleaned it up properly and there is in fact one problem
> with it: The BundleArchive constructors initiliaze the instances. When
> extending the BundleArchive for storage dependent implementations, the
> constructors of the base class do not yet know anything about the
> storage, but initialization depends on the storage. My solution is to
> introduce an init() method, which initializes and ist called by the
> BundleCache.
>
> I do not like this implementation very much as it kind of transfers
> responsibility for additional intialization to the caller. But in the
> short time, I did not come up with a better solution.
>
> I attach the patch for this prototype along with the File based 
> implementation.
>
> What do you think: Could this be a way to go ? Should I post a Jira 
> issue ?
>
> Regards and Thanks
> Felix, the person :-)
>
> On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
>> Felix Meschberger wrote:
>> > On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
>> >> In general, any solution for doing stuff like you have suggested, I
>> >> would hope, should concentrate on using/improving these existing
>> >> mechanisms rather than creating new ones.
>> >
>> > I definitely agree. Yet not being able to ammend BundleCache and
>> > BundleArchive, I am still required to have file system space - this
>> > sort of worries me.
>>
>> This is the sort of stuff that we can improve upon then. I am not
>> against making it once again possible to be able configure which
>> implementation of BundleCache to use. It was just removed because I had
>> no valid use case. We just need to discuss what is needed and how to 
>> do it.
>>
>> -> richard
>>

Re: Non-File based Framework

[Felix Meschberger <fm...@gmail.com>]

HI again,

Based upon your recommendations, I tried to implement the abstract
BundleCache and BundleArchive strategy as a prototype.

I have not cleaned it up properly and there is in fact one problem
with it: The BundleArchive constructors initiliaze the instances. When
extending the BundleArchive for storage dependent implementations, the
constructors of the base class do not yet know anything about the
storage, but initialization depends on the storage. My solution is to
introduce an init() method, which initializes and ist called by the
BundleCache.

I do not like this implementation very much as it kind of transfers
responsibility for additional intialization to the caller. But in the
short time, I did not come up with a better solution.

I attach the patch for this prototype along with the File based implementation.

What do you think: Could this be a way to go ? Should I post a Jira issue ?

Regards and Thanks
Felix, the person :-)

On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
> Felix Meschberger wrote:
> > On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
> >> In general, any solution for doing stuff like you have suggested, I
> >> would hope, should concentrate on using/improving these existing
> >> mechanisms rather than creating new ones.
> >
> > I definitely agree. Yet not being able to ammend BundleCache and
> > BundleArchive, I am still required to have file system space - this
> > sort of worries me.
>
> This is the sort of stuff that we can improve upon then. I am not
> against making it once again possible to be able configure which
> implementation of BundleCache to use. It was just removed because I had
> no valid use case. We just need to discuss what is needed and how to do it.
>
> -> richard
>

Re: Non-File based Framework

["Richard S. Hall" <he...@ungoverned.org>]

Felix Meschberger wrote:
> On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
>> In general, any solution for doing stuff like you have suggested, I
>> would hope, should concentrate on using/improving these existing
>> mechanisms rather than creating new ones.
>
> I definitely agree. Yet not being able to ammend BundleCache and
> BundleArchive, I am still required to have file system space - this
> sort of worries me.

This is the sort of stuff that we can improve upon then. I am not 
against making it once again possible to be able configure which 
implementation of BundleCache to use. It was just removed because I had 
no valid use case. We just need to discuss what is needed and how to do it.

-> richard

Re: Non-File based Framework

[Felix Meschberger <Fe...@day.com>]

On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
> In general, any solution for doing stuff like you have suggested, I
> would hope, should concentrate on using/improving these existing
> mechanisms rather than creating new ones.

I definitely agree. Yet not being able to ammend BundleCache and
BundleArchive, I am still required to have file system space - this
sort of worries me.

> > Maybe to illustrate my solution, I will just provide the patch to the
> > mailing list in minute.

Here we go...

Regards
Felix

Re: Non-File based Framework

["Richard S. Hall" <he...@ungoverned.org>]

Felix Meschberger wrote:
> On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
>> I cannot say that I totally understand your solution.
>
> Maybe I just too many words again :-)
>
> * The Felix class has an instance field m_resourceFactory of type
> OSGiResourceFactory which is assigned in the Felix.start() method.
> * All resource accesses go through instance method
> OSGiResourceFactory.createResource(String)
> * A resource can also be acquired through OSGiResource.getChild(String 
> relPath)
>
> This solution has no static global - except the somewhat static global
> in the property resolver.

Ok, that is good.

> This sounds promising and also allows me to solve yet another issue:
> In my approach the storage has to be present outside the framework. If
> I would be able to plugin into the creation of the BundleRevision
> instance, this would be great as I could create a bundle which just
> acquires the storage and provides functionality to access bundles in
> that storage
>
> I do not think hooking into the creation of the IContent instances is
> a top priority, as those are created inside the BundleRevisions, right
> ?

That is correct.

> But going that way, would probably also require touching the
> BundleCache and BundleArchive classes, as they currenty are File
> based.

Yes, that is correct.

> In fact the Felix.start method is documented to support a
> felix.cache.class property which names the BundleCache class to use -
> yet it seems to not be implemented like this.

This area was refactored at the beginning of this year, during which 
time the capability to provide a new cache implementation was removed, 
since most of the reasons for doing so could be emulated with creating 
new revision or content implementations.

However, this area is not completely baked, so we are open to improving it.

In general, any solution for doing stuff like you have suggested, I 
would hope, should concentrate on using/improving these existing 
mechanisms rather than creating new ones.

> Maybe to illustrate my solution, I will just provide the patch to the
> mailing list in minute.

Ok.

-> richard

Re: Non-File based Framework

[Jeff McAffer <Je...@ca.ibm.com>]

you might want to look at the Equinox Adaptor Hook mechanism.  Basically 
under Equinox is an "adaptor" layer that allows you to replace or hook 
into many of the implementation details of the framework (classloading, 
storage strategies, ...).  Different framework fragment bundles can hook 
different parts of the adaptor and add or replace function.  We use this, 
in particular, to completely change how bundles are laid out on disk 
(e.g., for Flash file systems vs normal file systems) adding caching etc. 
Seems relevant here.

Jeff




"Felix Meschberger" <Fe...@day.com> 
Sent by: fmeschbe@gmail.com
06/20/2006 05:20 AM
Please respond to
felix-dev@incubator.apache.org


To
felix-dev@incubator.apache.org
cc

Subject
Re: Non-File based Framework






On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
> I cannot say that I totally understand your solution.

Maybe I just too many words again :-)

* The Felix class has an instance field m_resourceFactory of type
OSGiResourceFactory which is assigned in the Felix.start() method.
* All resource accesses go through instance method
OSGiResourceFactory.createResource(String)
* A resource can also be acquired through OSGiResource.getChild(String 
relPath)

This solution has no static global - except the somewhat static global
in the property resolver.

> In general, I try
> to avoid the use of static globals since they decrease flexibility in

I absolutely agree. In fact the only static method (perhaps described
incorrectly is OSGiResourceFactory.createInstance(PropertyResolver)
which creates a factory from the configuration properties and which is
simply sort of a convenience method.

> how Felix instances are created and embedded (e.g., some users embed
> Felix inside of bundles, which themselves can have Felix inside of their
> bundles).
>
> In general, Felix has two abstractions that relate to the file system:
>
>     * org.apache.felix.framework.cache.BundleRevision - which represents
>       how bundles are saved in the bundle cache.
>     * org.apache.felix.module.IContent - which represents access to the
>       module content itself.

This sounds promising and also allows me to solve yet another issue:
In my approach the storage has to be present outside the framework. If
I would be able to plugin into the creation of the BundleRevision
instance, this would be great as I could create a bundle which just
acquires the storage and provides functionality to access bundles in
that storage

I do not think hooking into the creation of the IContent instances is
a top priority, as those are created inside the BundleRevisions, right
?

But going that way, would probably also require touching the
BundleCache and BundleArchive classes, as they currenty are File
based. In fact the Felix.start method is documented to support a
felix.cache.class property which names the BundleCache class to use -
yet it seems to not be implemented like this.

Maybe to illustrate my solution, I will just provide the patch to the
mailing list in minute.

Regards
Felix

Re: Non-File based Framework

[Felix Meschberger <Fe...@day.com>]

On 6/20/06, Richard S. Hall <he...@ungoverned.org> wrote:
> I cannot say that I totally understand your solution.

Maybe I just too many words again :-)

* The Felix class has an instance field m_resourceFactory of type
OSGiResourceFactory which is assigned in the Felix.start() method.
* All resource accesses go through instance method
OSGiResourceFactory.createResource(String)
* A resource can also be acquired through OSGiResource.getChild(String relPath)

This solution has no static global - except the somewhat static global
in the property resolver.

> In general, I try
> to avoid the use of static globals since they decrease flexibility in

I absolutely agree. In fact the only static method (perhaps described
incorrectly is OSGiResourceFactory.createInstance(PropertyResolver)
which creates a factory from the configuration properties and which is
simply sort of a convenience method.

> how Felix instances are created and embedded (e.g., some users embed
> Felix inside of bundles, which themselves can have Felix inside of their
> bundles).
>
> In general, Felix has two abstractions that relate to the file system:
>
>     * org.apache.felix.framework.cache.BundleRevision - which represents
>       how bundles are saved in the bundle cache.
>     * org.apache.felix.module.IContent - which represents access to the
>       module content itself.

This sounds promising and also allows me to solve yet another issue:
In my approach the storage has to be present outside the framework. If
I would be able to plugin into the creation of the BundleRevision
instance, this would be great as I could create a bundle which just
acquires the storage and provides functionality to access bundles in
that storage

I do not think hooking into the creation of the IContent instances is
a top priority, as those are created inside the BundleRevisions, right
?

But going that way, would probably also require touching the
BundleCache and BundleArchive classes, as they currenty are File
based. In fact the Felix.start method is documented to support a
felix.cache.class property which names the BundleCache class to use -
yet it seems to not be implemented like this.

Maybe to illustrate my solution, I will just provide the patch to the
mailing list in minute.

Regards
Felix

Re: Non-File based Framework

["Richard S. Hall" <he...@ungoverned.org>]

Felix Meschberger wrote:
> (1) Our product is running as a web application inside any customer
> supplied servlet container (Tomcat, WebSphere, WebLogic, Caucho).
> Also, we want to execute the product with as minimal OS filesystem
> requirement as possible accessing code and content only from a JCR
> repository. Unfortunately, out of the box, the Felix framework
> requires an OS filesystem. Please find my proposal below.
>
> Regarding Filesystems: I found, that direct filesystem access in the
> current Felix project trunk is isolated in only few places.
> Additionally, JAR files seem to be accessed through a JarFileX class,
> which extends JarFile. So what I did to remove the reliance on the
> filesystem, is to define a resource interface (I call it OSGiResource
> laking a better name) and a resource factory abstract class (I call it
> OSGiResourceFactory for the same reason). The factory provides a
> static method to create resource instances from a string path or
> location URL. The resource factory implementation to use is defined by
> a configuration property. Resources are either created by calling the
> resource factory method or creating the resource instance as child of
> a another resource.

I cannot say that I totally understand your solution. In general, I try 
to avoid the use of static globals since they decrease flexibility in 
how Felix instances are created and embedded (e.g., some users embed 
Felix inside of bundles, which themselves can have Felix inside of their 
bundles).

In general, Felix has two abstractions that relate to the file system:

    * org.apache.felix.framework.cache.BundleRevision - which represents
      how bundles are saved in the bundle cache.
    * org.apache.felix.module.IContent - which represents access to the
      module content itself.

There are implementations of these interfaces for JarRevision and 
JarContent (as well as others). Why is it not sufficient to create 
different implementations of these interfaces to suit your needs? The 
purpose of these abstractions is to be able to do what you ask (i.e., 
change storage approach), so if they are not sufficient for that, then 
we should consider making it so. If the main thing is that there is no 
external way to add and configure new implementations, then that is 
something we could work on.

> Of course, as a consequence of using a resource class instead of the
> java.io.File class, the BundleContext.getDataFile(String) method might
> return null, if the required bundle resource cannot be accessed in the
> file system.

Not a problem, since the spec allows that.

> I would be very interested in any feedback on this idea and whether
> you would be interested in a patch.

We would definitely be interested in understanding what is need and 
coming up with and/or adopting a solution for you.

> Felix

This could get confusing...  ;-)

-> richard

Re: Non-File based Framework

[Marcel Offermans <ma...@luminis.nl>]

Hello Felix,

I'd like to briefly respond to your first proposal, about abstracting  
from the filesystem. Personally I am intested in this, as we've  
already encountered a real-life situation where it would be very  
helpful if we could have more control over which files go where.

On Jun 20, 2006, at 9:56 , Felix Meschberger wrote:

> Probably, it would also make sense to expose the resource factory as a
> service for other bundles to be able to use it.

Agreed. Recently there has been some discussion of writing a "store"  
interface that would allow us to write multiple back-ends for  
standard OSGi services like User Admin, Preferences and Configuration  
Admin.

> Of course, as a consequence of using a resource class instead of the
> java.io.File class, the BundleContext.getDataFile(String) method might
> return null, if the required bundle resource cannot be accessed in the
> file system.

I haven't given this much thought yet, but is it possible to subclass  
File and implement it using a back-end store of some sort?

Greetings, Marcel