[API] Experimenting OSGI fragments to solve the extensibility issue

[Pierre-Arnaud Marcelot <pa...@marcelot.net>]

Hi Dev,

Following our efforts on the OSGI side, I took some time to think about (and experiment) how we could solve our extensibility issue on the LDAP API, where we want our users to be able to provide their own custom implementation of various schema objects like comparators, normalizers, etc.

After a quick discussion with Emmanuel and Guillaume Nodet, it turned out using OSGI fragments bundles could probably be our best solution.

Here's a definition of what an OSGI fragment bundle is (more information available at [1]):
"An OSGi fragment is a Java archive file with specific manifest headers that enable it to attach to a specified host bundle or specified host bundles in order to function.
Fragments are treated as part of the host bundles. Relevant definitions of the fragment are merged with the host bundles definitions before the host is resolved, as long as the information does not conflict.
Fragment dependencies are resolved if possible. If the fragment dependencies can not be resolved, the fragment does not attach to the host bundle.
A fragment can not have its own class loader or bundle activator. It can not override the information present in the host bundles. Fragments extend bundles with resources, classes, and permitted headers enabling you to customize your bundles."

The great thing about fragments is that they *share* the same class loader as their host bundle. Which was pretty much the kind of issue we were having with classloaders being differents from one bundle to the other.

The other great thing I see with this approach, it that it would also work great outside of an OSGI container application (which is a strong requirement for the LDAP API). A fragment bundle is nothing else than a regular bundle with a specific OSGI directive (Fragment-Host) added to its MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when it's not included in an OSGI container. Thus, it would allow us to support third parties extensions.

I have done a small experiment in my Eclipse workspace with two bundles (one being the host and the other being the fragment ) and I was able to classload, without any classloader issue, a class defined in the fragment bundle from another class inside the host bundle.

I'd like to go a step further and experiment this on the API itself.

Ideally, I'd like to use the 'shared-ldap-model' module ('org.apache.directory.shared.ldap.model' bundle) as a host for all our schema objects implementations and move these implementations into a specific fragment bundle containing them all.
Users would have to do the same to include their extensions. A simple fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as host and it works.

One other thing that should be done, is to let the 'shared-ldap-model' module do the instanciation of the schema elements. At the moment, two classes from the 'shared-ldap-schema-data' module are responsible for this, 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We would need to move these classes to the 'shared-ldap-model' module, for them to have access to the right class loader.

All in all, I think that's the only things we need to do to get the extensibility we wanted inside and outside an OSGI container.

What do you think of this plan?

Regards,
Pierre-Arnaud

[1] - http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Emmanuel Lecharny <el...@gmail.com>]

On 10/19/11 9:27 PM, Göktürk Gezer wrote:
> On Wed, Oct 19, 2011 at 10:13 PM, Pierre-Arnaud Marcelot
> <pa...@gmail.com>wrote:
>
>> Hi Göktürk,
>>
>> Le 19 oct. 2011 à 19:05, "Göktürk Gezer"<go...@gmail.com>  a
>> écrit :
>>
>> Hi Pierre,
>>
>> On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot<pa...@marcelot.net>wrote:
>>
>>> Hi Dev,
>>>
>>> Following our efforts on the OSGI side, I took some time to think about
>>> (and experiment) how we could solve our extensibility issue on the LDAP API,
>>> where we want our users to be able to provide their own custom
>>> implementation of various schema objects like comparators, normalizers, etc.
>>>
>>> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out
>>> using OSGI fragments bundles could probably be our best solution.
>>>
>> Hmm, may not be the best, see below.
>>
>>> Here's a definition of what an OSGI fragment bundle is (more information
>>> available at [1]):
>>> "An OSGi fragment is a Java archive file with specific manifest headers
>>> that enable it to attach to a specified host bundle or specified host
>>> bundles in order to function.
>>> Fragments are treated as part of the host bundles. Relevant definitions of
>>> the fragment are merged with the host bundles definitions before the host is
>>> resolved, as long as the information does not conflict.
>>> Fragment dependencies are resolved if possible. If the fragment
>>> dependencies can not be resolved, the fragment does not attach to the host
>>> bundle.
>>> A fragment can not have its own class loader or bundle activator. It can
>>> not override the information present in the host bundles. Fragments extend
>>> bundles with resources, classes, and permitted headers enabling you to
>>> customize your bundles."
>>>
>> Yes, fragments are merged with their host and they left up in the framework
>> as just resolved, no active state at all. Everything it imports/exports
>> become host's imports/exports.
>>
>>> The great thing about fragments is that they *share* the same class loader
>>> as their host bundle. Which was pretty much the kind of issue we were having
>>> with classloaders being differents from one bundle to the other.
>>>
>>> The other great thing I see with this approach, it that it would also work
>>> great outside of an OSGI container application (which is a strong
>>> requirement for the LDAP API). A fragment bundle is nothing else than a
>>> regular bundle with a specific OSGI directive (Fragment-Host) added to its
>>> MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when
>>> it's not included in an OSGI container. Thus, it would allow us to support
>>> third parties extensions.
>>>
>> It would only allow 3th parties to support their own extensions. Why?.
>> Because being able to classload the 3th party classes is only half of the
>> story. We must also know that they've arrived and we must know what has been
>> arrived. Otherwise we have to hard code their name, ant it is not case in
>> 3th party scenario. Once the fragment is merged with the host, we can class
>> load them inside host and some other bundle having access to that fragments
>> specific package? But how do we enforce that package names and class names
>> across 3th parties?
>>
>>
>> Actually, that's exactly how it works at the moment. The fully qualified
>> name of the class is available and read from the schema ldif files. That's
>> how we get the other half of the story.
>> If someone wants to extend the schema with a new comparator, for example,
>> it needs to provide the class and edit the schema to declare the new
>> comparator.
>>
> Hmm. yes i see now. Actually this is a problem of Studio more than DS.
> Because it is used to create schemas. So when some 3th party provide custom
> elements, its Studio's responsibility to find and list it under
> corresponding GUI element. After that, DS can load it using that class name
> either with classloading(with fragments) or OSGI.
>
>> I have done a small experiment in my Eclipse workspace with two bundles
>>> (one being the host and the other being the fragment ) and I was able to
>>> classload, without any classloader issue, a class defined in the fragment
>>> bundle from another class inside the host bundle.
>>>
>> Did you do it while knowing the class name on the fragment bundle, or not
>> knowing the class name at all?
>>
>>
>> Yep, indeed.
>>
>>   I'd like to go a step further and experiment this on the API itself.
>>> Ideally, I'd like to use the 'shared-ldap-model' module
>>> ('org.apache.directory.shared.ldap.model' bundle) as a host for all our
>>> schema objects implementations and move these implementations into a
>>> specific fragment bundle containing them all.
>>> Users would have to do the same to include their extensions. A simple
>>> fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as
>>> host and it works.
>>>
>> Yes it works for us, but for some body writing his own classes, we must
>> dynamically know they've arrived.
>>
>>>
>>> One other thing that should be done, is to let the 'shared-ldap-model'
>>> module do the instanciation of the schema elements. At the moment, two
>>> classes from the 'shared-ldap-schema-data' module are responsible for this,
>>> 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and
>>> 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We
>>> would need to move these classes to the 'shared-ldap-model' module, for them
>>> to have access to the right class loader.
>>>
>>> All in all, I think that's the only things we need to do to get the
>>> extensibility we wanted inside and outside an OSGI container.
>>>
>> It may bring us to somewhere but not even close to real goals IMO. We can
>> use them to provide in-house extensions, but when it comes to 3th parties,
>> it won't be so easy to use that fragments comnig from them inside Shared or
>> ApacheDS.
>>
>>
>> Sure, this is some very basic extensibility but it matches perfectly our
>> needs for the API. On the ApacheDS side, it's another story and we might
>> probably need something better and more complex.
>>
> I was already scared to play with shared anyway, now my fear tripled :) But
> i have some good news. I made my use of IPojo and OSGI to not broke existing
> shared clients. But i've got one more issue: the OIDs that's got assigned by
> constructors, are they static? or is it possible to lets say
> BooleanComparator to be instantiated with two different oids. As I see its
> not the case and you're maintaining standart OID assigments by hand. What is
> the case here?

OIDs are static, and one object can only have one single OID (Object 
IDentifier...)

At some point, the guys who invented the OIDs wanted to attribue one OID 
for everything on earth...

I have no clear idea about the other part of the mail, but you have to 
know that a schema extension (we currently have 3 kind of schema 
extensions : Comparator, SyntaxCheckers and Normalizers) can be used in 
three different contexts :
- in ApacheDS, so in a fully compliant OSGi environment with an OGSi 
container (this is what you are working on)
- in Studio, which is based on Eclipse, so it also uses a container
- in the ldap API, which can perfectly be used without any OSGi 
container, and wit h users expecting to be able to classload the 
extensions after having put the classes in their classpath.

We must cover those three kind of usage.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Emmanuel Lécharny <el...@apache.org>]

On 10/20/11 3:48 AM, Göktürk Gezer wrote:
> Hi Emmanuel,Pierre
>
> I just attached my current work on Comparators to #DIRSERVER-1672. So here
> is the explanation of how it works and how we can improve.
>
> Because both issues i asked are ambigious. I decided to keep them both. I
> mean Comparators are not shared and their OID is also assignable.

Just some insight about why Compartor's OID can be assigned : they are 
associated with the MatchingRules, which have a unique OID. But the 
thing is that a Comparator can be used by more than one MatchingRule. In 
this case, the Comparator will inherit its OID from the MatchingRule it 
is associated with.


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Göktürk Gezer <go...@gmail.com>]

BTW i forgot to mention about 3th party developers issue.

3th parties using OSGI can extend our schema elements with any way they
want. IPojo,Blueprint, OSGI Services. IPojo is currently supported out of
the box for Comparators(implementing for the other two is piece of cake from
now on.) For the 3th parties not using OSGI, they can provide their classes
with the Fragment bundles. Actually it is OSGI feature too. To be able get
the class names of 3th parties that are not using Services we can introduce
some extender bundle for watching bundles that are installed into framework
those having some unique signature. Like
<ApacheDS-Comparator>org.foo.BarComparator<ApacheDS-Comparator>. So when
that extender sees that fragment being installed into framework, it can
notify the shared to load that new class name to its internal list.


Regards,
Gokturk

On Thu, Oct 20, 2011 at 4:48 AM, Göktürk Gezer <go...@gmail.com>wrote:

> Hi Emmanuel,Pierre
>
> I just attached my current work on Comparators to #DIRSERVER-1672. So here
> is the explanation of how it works and how we can improve.
>
> Because both issues i asked are ambigious. I decided to keep them both. I
> mean Comparators are not shared and their OID is also assignable. First of
> all, existing clients are ok with these changes. Object code manipulation
> did not brake the nonOSGI clients now, and i keep the old semantic of
> monolithic code exactly the same. Clients can continue using and
> instantiating Comparator classes without OSGI. IPojo related problems on
> that are resolved. keeping OID assignments in OSGI was so hard because
> constructor(String)s of Comparators was preventing default parameterless
> constructors to be created, which is IPojo's default way of creating
> components. After struggling with IPojo's source code a little, i found a
> solution to that too, which is constructor argument injection feature.
>
> Comparator class definitions are changed like the example below:
>
> @Component
> @Provides
> class BooleanComparator implements LdapComparator<String>
> {
>     @Property(name="ads.comp.type",value="comparator")
>     public String compType;
>
>     ...................
>     ..............
>     public BooleanComparator( @Property(name="ads.comp.comparator.oid")
> String oid )
>     {
>      ......................
>     }
> }
>
> The first two annotations are for making IPojo publishing a factory for
> BooleanComparator class. It's not an instance, the first thing that
> published is a factory which can be used to create instances of it, and i'm
> using  it later to create an instances of that type. It is normally as easy
> as putting @Instantiate annotation additionaly, but it does not fit in our
> case, so i had to access factories manually.
>
> The first @Property annotation is for publishing component type of a
> component. This property is not being used right now, and if we decide to go
> on with that, i will replace it with custom annotation on top of class like
> @AdsComparator. The second @Property is for constructor parameter. While
> creating instance using factory we 'll provide a configuration propery by
> name "asd.comp.comparator.oid" which will be injected to constructor as
> parameter and the component's underlying object will be created with that
> constructor. So the either way (OSGI and normal) creating the component will
> go through the same constructor.
>
> In order to be able create these IPojo instances i created,
> shared-ipojo-manager project with some helpers and main SchemaElementManager
> class which is for loading intended schema element (only Comparator is
> implemented by now, but the other two are just copy and modify job). 2 new
> versions of SchemaElementFactory class' two method, getLdapComparator() and
> classLoadComparator(), is created to allow using
> ipojo-manager.SchemaElementsManager in order to get class references through
> OSGI. Just additions at that level, no deletions. Only thing i changed that
> brokes the standalone ApacheDS is in DefaultSchemaManager.addComparators()
> method. I changed it to use SchemaElementFactory's OSGI versioned methods
> instead of the old ones. Actually if i move those changes above from
> DefaultSchemaManager the ApacheDS can still run as standalone application(if
> it makes any sense)
>
> I did it as demonstration, so sorry about not putting so much comments at
> the code, But the information i provided here will help you review the code
> mostly.
>
> BTW, FYI i will be mostly MIA for next 5 days. Have very heavy exam
> schedule. But i'll keep up with the mails for further discussions.
>
>
> Regards,
> Göktürk
>

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Göktürk Gezer <go...@gmail.com>]

Hi Emmanuel,Pierre

I just attached my current work on Comparators to #DIRSERVER-1672. So here
is the explanation of how it works and how we can improve.

Because both issues i asked are ambigious. I decided to keep them both. I
mean Comparators are not shared and their OID is also assignable. First of
all, existing clients are ok with these changes. Object code manipulation
did not brake the nonOSGI clients now, and i keep the old semantic of
monolithic code exactly the same. Clients can continue using and
instantiating Comparator classes without OSGI. IPojo related problems on
that are resolved. keeping OID assignments in OSGI was so hard because
constructor(String)s of Comparators was preventing default parameterless
constructors to be created, which is IPojo's default way of creating
components. After struggling with IPojo's source code a little, i found a
solution to that too, which is constructor argument injection feature.

Comparator class definitions are changed like the example below:

@Component
@Provides
class BooleanComparator implements LdapComparator<String>
{
    @Property(name="ads.comp.type",value="comparator")
    public String compType;

    ...................
    ..............
    public BooleanComparator( @Property(name="ads.comp.comparator.oid")
String oid )
    {
     ......................
    }
}

The first two annotations are for making IPojo publishing a factory for
BooleanComparator class. It's not an instance, the first thing that
published is a factory which can be used to create instances of it, and i'm
using  it later to create an instances of that type. It is normally as easy
as putting @Instantiate annotation additionaly, but it does not fit in our
case, so i had to access factories manually.

The first @Property annotation is for publishing component type of a
component. This property is not being used right now, and if we decide to go
on with that, i will replace it with custom annotation on top of class like
@AdsComparator. The second @Property is for constructor parameter. While
creating instance using factory we 'll provide a configuration propery by
name "asd.comp.comparator.oid" which will be injected to constructor as
parameter and the component's underlying object will be created with that
constructor. So the either way (OSGI and normal) creating the component will
go through the same constructor.

In order to be able create these IPojo instances i created,
shared-ipojo-manager project with some helpers and main SchemaElementManager
class which is for loading intended schema element (only Comparator is
implemented by now, but the other two are just copy and modify job). 2 new
versions of SchemaElementFactory class' two method, getLdapComparator() and
classLoadComparator(), is created to allow using
ipojo-manager.SchemaElementsManager in order to get class references through
OSGI. Just additions at that level, no deletions. Only thing i changed that
brokes the standalone ApacheDS is in DefaultSchemaManager.addComparators()
method. I changed it to use SchemaElementFactory's OSGI versioned methods
instead of the old ones. Actually if i move those changes above from
DefaultSchemaManager the ApacheDS can still run as standalone application(if
it makes any sense)

I did it as demonstration, so sorry about not putting so much comments at
the code, But the information i provided here will help you review the code
mostly.

BTW, FYI i will be mostly MIA for next 5 days. Have very heavy exam
schedule. But i'll keep up with the mails for further discussions.


Regards,
Göktürk

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Emmanuel Lecharny <el...@gmail.com>]

Forgot to mention that there are two ways to inject a new Schema element 
into the API :
- creating entries with the FQCN of the associated class
- or simply inject the bytes of the serialized class into the entry.

In both case, we must be able to class-load the schema element...


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Pierre-Arnaud Marcelot <pa...@marcelot.net>]

On 19 oct. 2011, at 21:27, Göktürk Gezer wrote:
> On Wed, Oct 19, 2011 at 10:13 PM, Pierre-Arnaud Marcelot <pa...@gmail.com> wrote:
> Hi Göktürk,
> 
> Le 19 oct. 2011 à 19:05, "Göktürk Gezer" <go...@gmail.com> a écrit :
> 
>> Hi Pierre,
>> 
>> On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net> wrote:
>> Hi Dev,
>> 
>> Following our efforts on the OSGI side, I took some time to think about (and experiment) how we could solve our extensibility issue on the LDAP API, where we want our users to be able to provide their own custom implementation of various schema objects like comparators, normalizers, etc.
>> 
>> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out using OSGI fragments bundles could probably be our best solution.
>> Hmm, may not be the best, see below. 
>> 
>> Here's a definition of what an OSGI fragment bundle is (more information available at [1]):
>> "An OSGi fragment is a Java archive file with specific manifest headers that enable it to attach to a specified host bundle or specified host bundles in order to function.
>> Fragments are treated as part of the host bundles. Relevant definitions of the fragment are merged with the host bundles definitions before the host is resolved, as long as the information does not conflict.
>> Fragment dependencies are resolved if possible. If the fragment dependencies can not be resolved, the fragment does not attach to the host bundle.
>> A fragment can not have its own class loader or bundle activator. It can not override the information present in the host bundles. Fragments extend bundles with resources, classes, and permitted headers enabling you to customize your bundles."
>> Yes, fragments are merged with their host and they left up in the framework as just resolved, no active state at all. Everything it imports/exports become host's imports/exports.
>> 
>> The great thing about fragments is that they *share* the same class loader as their host bundle. Which was pretty much the kind of issue we were having with classloaders being differents from one bundle to the other. 
>>  
>> 
>> The other great thing I see with this approach, it that it would also work great outside of an OSGI container application (which is a strong requirement for the LDAP API). A fragment bundle is nothing else than a regular bundle with a specific OSGI directive (Fragment-Host) added to its MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when it's not included in an OSGI container. Thus, it would allow us to support third parties extensions.
>> It would only allow 3th parties to support their own extensions. Why?. Because being able to classload the 3th party classes is only half of the story. We must also know that they've arrived and we must know what has been arrived. Otherwise we have to hard code their name, ant it is not case in 3th party scenario. Once the fragment is merged with the host, we can class load them inside host and some other bundle having access to that fragments specific package? But how do we enforce that package names and class names across 3th parties?
> 
> Actually, that's exactly how it works at the moment. The fully qualified name of the class is available and read from the schema ldif files. That's how we get the other half of the story. 
> If someone wants to extend the schema with a new comparator, for example, it needs to provide the class and edit the schema to declare the new comparator. 
> Hmm. yes i see now. Actually this is a problem of Studio more than DS. Because it is used to create schemas. So when some 3th party provide custom elements, its Studio's responsibility to find and list it under corresponding GUI element. After that, DS can load it using that class name either with classloading(with fragments) or OSGI.

Actually what I was describing above has very few to do with Studio. This is the way the schema is handled in the LDAP API (which is used inside Studio) and within ApacheDS too.

>> I have done a small experiment in my Eclipse workspace with two bundles (one being the host and the other being the fragment ) and I was able to classload, without any classloader issue, a class defined in the fragment bundle from another class inside the host bundle.
>> Did you do it while knowing the class name on the fragment bundle, or not knowing the class name at all? 
> 
> Yep, indeed. 
> 
>> I'd like to go a step further and experiment this on the API itself.
>> 
>> Ideally, I'd like to use the 'shared-ldap-model' module ('org.apache.directory.shared.ldap.model' bundle) as a host for all our schema objects implementations and move these implementations into a specific fragment bundle containing them all.
>> Users would have to do the same to include their extensions. A simple fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as host and it works.
>> Yes it works for us, but for some body writing his own classes, we must dynamically know they've arrived. 
>>  
>> 
>> One other thing that should be done, is to let the 'shared-ldap-model' module do the instanciation of the schema elements. At the moment, two classes from the 'shared-ldap-schema-data' module are responsible for this, 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We would need to move these classes to the 'shared-ldap-model' module, for them to have access to the right class loader.
>> 
>> All in all, I think that's the only things we need to do to get the extensibility we wanted inside and outside an OSGI container.
>> It may bring us to somewhere but not even close to real goals IMO. We can use them to provide in-house extensions, but when it comes to 3th parties, it won't be so easy to use that fragments comnig from them inside Shared or ApacheDS. 
> 
> Sure, this is some very basic extensibility but it matches perfectly our needs for the API. On the ApacheDS side, it's another story and we might probably need something better and more complex. 
> I was already scared to play with shared anyway, now my fear tripled :) But i have some good news. I made my use of IPojo and OSGI to not broke existing shared clients.

Cool. I'm looking for to seeing that. ;)
Could you share a little bit more how you're doing it?

> But i've got one more issue: the OIDs that's got assigned by constructors, are they static? or is it possible to lets say BooleanComparator to be instantiated with two different oids. As I see its not the case and you're maintaining standart OID assigments by hand. What is the case here?

I guess it's possible to have a BooleanComparator instantiated with two different OIDs, as soon as the API is open to users (they always manage to find corner cases :)). Especially in the case where multiple instances are involved. The schema could be very different from one instance to the other.

Regards,
Pierre-Arnaud

> 
> Regards,
> Pierre-Armaud
> 
>> What do you think of this plan?
>> 
>> Regards,
>> Pierre-Arnaud
>> 
>> [1] - http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html
>> 
>> Regards,
>> Gokturk
> Regards,
> Gokturk

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Göktürk Gezer <go...@gmail.com>]

On Wed, Oct 19, 2011 at 10:13 PM, Pierre-Arnaud Marcelot
<pa...@gmail.com>wrote:

> Hi Göktürk,
>
> Le 19 oct. 2011 à 19:05, "Göktürk Gezer" <go...@gmail.com> a
> écrit :
>
> Hi Pierre,
>
> On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net>wrote:
>
>> Hi Dev,
>>
>> Following our efforts on the OSGI side, I took some time to think about
>> (and experiment) how we could solve our extensibility issue on the LDAP API,
>> where we want our users to be able to provide their own custom
>> implementation of various schema objects like comparators, normalizers, etc.
>>
>> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out
>> using OSGI fragments bundles could probably be our best solution.
>>
> Hmm, may not be the best, see below.
>
>>
>> Here's a definition of what an OSGI fragment bundle is (more information
>> available at [1]):
>> "An OSGi fragment is a Java archive file with specific manifest headers
>> that enable it to attach to a specified host bundle or specified host
>> bundles in order to function.
>> Fragments are treated as part of the host bundles. Relevant definitions of
>> the fragment are merged with the host bundles definitions before the host is
>> resolved, as long as the information does not conflict.
>> Fragment dependencies are resolved if possible. If the fragment
>> dependencies can not be resolved, the fragment does not attach to the host
>> bundle.
>> A fragment can not have its own class loader or bundle activator. It can
>> not override the information present in the host bundles. Fragments extend
>> bundles with resources, classes, and permitted headers enabling you to
>> customize your bundles."
>>
> Yes, fragments are merged with their host and they left up in the framework
> as just resolved, no active state at all. Everything it imports/exports
> become host's imports/exports.
>
>>
>> The great thing about fragments is that they *share* the same class loader
>> as their host bundle. Which was pretty much the kind of issue we were having
>> with classloaders being differents from one bundle to the other.
>>
>
>>
>
>> The other great thing I see with this approach, it that it would also work
>> great outside of an OSGI container application (which is a strong
>> requirement for the LDAP API). A fragment bundle is nothing else than a
>> regular bundle with a specific OSGI directive (Fragment-Host) added to its
>> MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when
>> it's not included in an OSGI container. Thus, it would allow us to support
>> third parties extensions.
>>
> It would only allow 3th parties to support their own extensions. Why?.
> Because being able to classload the 3th party classes is only half of the
> story. We must also know that they've arrived and we must know what has been
> arrived. Otherwise we have to hard code their name, ant it is not case in
> 3th party scenario. Once the fragment is merged with the host, we can class
> load them inside host and some other bundle having access to that fragments
> specific package? But how do we enforce that package names and class names
> across 3th parties?
>
>
> Actually, that's exactly how it works at the moment. The fully qualified
> name of the class is available and read from the schema ldif files. That's
> how we get the other half of the story.
> If someone wants to extend the schema with a new comparator, for example,
> it needs to provide the class and edit the schema to declare the new
> comparator.
>
Hmm. yes i see now. Actually this is a problem of Studio more than DS.
Because it is used to create schemas. So when some 3th party provide custom
elements, its Studio's responsibility to find and list it under
corresponding GUI element. After that, DS can load it using that class name
either with classloading(with fragments) or OSGI.

>
> I have done a small experiment in my Eclipse workspace with two bundles
>> (one being the host and the other being the fragment ) and I was able to
>> classload, without any classloader issue, a class defined in the fragment
>> bundle from another class inside the host bundle.
>>
> Did you do it while knowing the class name on the fragment bundle, or not
> knowing the class name at all?
>
>
> Yep, indeed.
>
>  I'd like to go a step further and experiment this on the API itself.
>>
>> Ideally, I'd like to use the 'shared-ldap-model' module
>> ('org.apache.directory.shared.ldap.model' bundle) as a host for all our
>> schema objects implementations and move these implementations into a
>> specific fragment bundle containing them all.
>> Users would have to do the same to include their extensions. A simple
>> fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as
>> host and it works.
>>
> Yes it works for us, but for some body writing his own classes, we must
> dynamically know they've arrived.
>
>>
>>
>
>> One other thing that should be done, is to let the 'shared-ldap-model'
>> module do the instanciation of the schema elements. At the moment, two
>> classes from the 'shared-ldap-schema-data' module are responsible for this,
>> 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and
>> 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We
>> would need to move these classes to the 'shared-ldap-model' module, for them
>> to have access to the right class loader.
>>
>> All in all, I think that's the only things we need to do to get the
>> extensibility we wanted inside and outside an OSGI container.
>>
> It may bring us to somewhere but not even close to real goals IMO. We can
> use them to provide in-house extensions, but when it comes to 3th parties,
> it won't be so easy to use that fragments comnig from them inside Shared or
> ApacheDS.
>
>
> Sure, this is some very basic extensibility but it matches perfectly our
> needs for the API. On the ApacheDS side, it's another story and we might
> probably need something better and more complex.
>
I was already scared to play with shared anyway, now my fear tripled :) But
i have some good news. I made my use of IPojo and OSGI to not broke existing
shared clients. But i've got one more issue: the OIDs that's got assigned by
constructors, are they static? or is it possible to lets say
BooleanComparator to be instantiated with two different oids. As I see its
not the case and you're maintaining standart OID assigments by hand. What is
the case here?

>
> Regards,
> Pierre-Armaud
>
> What do you think of this plan?
>>
>> Regards,
>> Pierre-Arnaud
>>
>> [1] -
>> http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html
>>
>
> Regards,
> Gokturk
>
> Regards,
Gokturk

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Pierre-Arnaud Marcelot <pa...@gmail.com>]

Hi Göktürk,

Le 19 oct. 2011 à 19:05, "Göktürk Gezer" <go...@gmail.com> a écrit :

Hi Pierre,

On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net>wrote:

> Hi Dev,
>
> Following our efforts on the OSGI side, I took some time to think about
> (and experiment) how we could solve our extensibility issue on the LDAP API,
> where we want our users to be able to provide their own custom
> implementation of various schema objects like comparators, normalizers, etc.
>
> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out
> using OSGI fragments bundles could probably be our best solution.
>
Hmm, may not be the best, see below.

>
> Here's a definition of what an OSGI fragment bundle is (more information
> available at [1]):
> "An OSGi fragment is a Java archive file with specific manifest headers
> that enable it to attach to a specified host bundle or specified host
> bundles in order to function.
> Fragments are treated as part of the host bundles. Relevant definitions of
> the fragment are merged with the host bundles definitions before the host is
> resolved, as long as the information does not conflict.
> Fragment dependencies are resolved if possible. If the fragment
> dependencies can not be resolved, the fragment does not attach to the host
> bundle.
> A fragment can not have its own class loader or bundle activator. It can
> not override the information present in the host bundles. Fragments extend
> bundles with resources, classes, and permitted headers enabling you to
> customize your bundles."
>
Yes, fragments are merged with their host and they left up in the framework
as just resolved, no active state at all. Everything it imports/exports
become host's imports/exports.

>
> The great thing about fragments is that they *share* the same class loader
> as their host bundle. Which was pretty much the kind of issue we were having
> with classloaders being differents from one bundle to the other.
>

>

> The other great thing I see with this approach, it that it would also work
> great outside of an OSGI container application (which is a strong
> requirement for the LDAP API). A fragment bundle is nothing else than a
> regular bundle with a specific OSGI directive (Fragment-Host) added to its
> MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when
> it's not included in an OSGI container. Thus, it would allow us to support
> third parties extensions.
>
It would only allow 3th parties to support their own extensions. Why?.
Because being able to classload the 3th party classes is only half of the
story. We must also know that they've arrived and we must know what has been
arrived. Otherwise we have to hard code their name, ant it is not case in
3th party scenario. Once the fragment is merged with the host, we can class
load them inside host and some other bundle having access to that fragments
specific package? But how do we enforce that package names and class names
across 3th parties?


Actually, that's exactly how it works at the moment. The fully qualified
name of the class is available and read from the schema ldif files. That's
how we get the other half of the story.
If someone wants to extend the schema with a new comparator, for example, it
needs to provide the class and edit the schema to declare the new
comparator.

I have done a small experiment in my Eclipse workspace with two bundles (one
> being the host and the other being the fragment ) and I was able to
> classload, without any classloader issue, a class defined in the fragment
> bundle from another class inside the host bundle.
>
Did you do it while knowing the class name on the fragment bundle, or not
knowing the class name at all?


Yep, indeed.

I'd like to go a step further and experiment this on the API itself.
>
> Ideally, I'd like to use the 'shared-ldap-model' module
> ('org.apache.directory.shared.ldap.model' bundle) as a host for all our
> schema objects implementations and move these implementations into a
> specific fragment bundle containing them all.
> Users would have to do the same to include their extensions. A simple
> fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as
> host and it works.
>
Yes it works for us, but for some body writing his own classes, we must
dynamically know they've arrived.

>
>

> One other thing that should be done, is to let the 'shared-ldap-model'
> module do the instanciation of the schema elements. At the moment, two
> classes from the 'shared-ldap-schema-data' module are responsible for this,
> 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and
> 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We
> would need to move these classes to the 'shared-ldap-model' module, for them
> to have access to the right class loader.
>
> All in all, I think that's the only things we need to do to get the
> extensibility we wanted inside and outside an OSGI container.
>
It may bring us to somewhere but not even close to real goals IMO. We can
use them to provide in-house extensions, but when it comes to 3th parties,
it won't be so easy to use that fragments comnig from them inside Shared or
ApacheDS.


Sure, this is some very basic extensibility but it matches perfectly our
needs for the API. On the ApacheDS side, it's another story and we might
probably need something better and more complex.

Regards,
Pierre-Armaud

What do you think of this plan?
>
> Regards,
> Pierre-Arnaud
>
> [1] -
> http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html
>

Regards,
Gokturk

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Göktürk Gezer <go...@gmail.com>]

Hi Pierre,

On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net>wrote:

> Hi Dev,
>
> Following our efforts on the OSGI side, I took some time to think about
> (and experiment) how we could solve our extensibility issue on the LDAP API,
> where we want our users to be able to provide their own custom
> implementation of various schema objects like comparators, normalizers, etc.
>
> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out
> using OSGI fragments bundles could probably be our best solution.
>
Hmm, may not be the best, see below.

>
> Here's a definition of what an OSGI fragment bundle is (more information
> available at [1]):
> "An OSGi fragment is a Java archive file with specific manifest headers
> that enable it to attach to a specified host bundle or specified host
> bundles in order to function.
> Fragments are treated as part of the host bundles. Relevant definitions of
> the fragment are merged with the host bundles definitions before the host is
> resolved, as long as the information does not conflict.
> Fragment dependencies are resolved if possible. If the fragment
> dependencies can not be resolved, the fragment does not attach to the host
> bundle.
> A fragment can not have its own class loader or bundle activator. It can
> not override the information present in the host bundles. Fragments extend
> bundles with resources, classes, and permitted headers enabling you to
> customize your bundles."
>
Yes, fragments are merged with their host and they left up in the framework
as just resolved, no active state at all. Everything it imports/exports
become host's imports/exports.

>
> The great thing about fragments is that they *share* the same class loader
> as their host bundle. Which was pretty much the kind of issue we were having
> with classloaders being differents from one bundle to the other.
>

>

> The other great thing I see with this approach, it that it would also work
> great outside of an OSGI container application (which is a strong
> requirement for the LDAP API). A fragment bundle is nothing else than a
> regular bundle with a specific OSGI directive (Fragment-Host) added to its
> MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when
> it's not included in an OSGI container. Thus, it would allow us to support
> third parties extensions.
>
It would only allow 3th parties to support their own extensions. Why?.
Because being able to classload the 3th party classes is only half of the
story. We must also know that they've arrived and we must know what has been
arrived. Otherwise we have to hard code their name, ant it is not case in
3th party scenario. Once the fragment is merged with the host, we can class
load them inside host and some other bundle having access to that fragments
specific package? But how do we enforce that package names and class names
across 3th parties?

>
>

> I have done a small experiment in my Eclipse workspace with two bundles
> (one being the host and the other being the fragment ) and I was able to
> classload, without any classloader issue, a class defined in the fragment
> bundle from another class inside the host bundle.
>
Did you do it while knowing the class name on the fragment bundle, or not
knowing the class name at all?

>
> I'd like to go a step further and experiment this on the API itself.
>
> Ideally, I'd like to use the 'shared-ldap-model' module
> ('org.apache.directory.shared.ldap.model' bundle) as a host for all our
> schema objects implementations and move these implementations into a
> specific fragment bundle containing them all.
> Users would have to do the same to include their extensions. A simple
> fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as
> host and it works.
>
Yes it works for us, but for some body writing his own classes, we must
dynamically know they've arrived.

>
>

> One other thing that should be done, is to let the 'shared-ldap-model'
> module do the instanciation of the schema elements. At the moment, two
> classes from the 'shared-ldap-schema-data' module are responsible for this,
> 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and
> 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We
> would need to move these classes to the 'shared-ldap-model' module, for them
> to have access to the right class loader.
>
> All in all, I think that's the only things we need to do to get the
> extensibility we wanted inside and outside an OSGI container.
>
It may bring us to somewhere but not even close to real goals IMO. We can
use them to provide in-house extensions, but when it comes to 3th parties,
it won't be so easy to use that fragments comnig from them inside Shared or
ApacheDS.

>
> What do you think of this plan?
>
> Regards,
> Pierre-Arnaud
>
> [1] -
> http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html
>

Regards,
Gokturk

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Pierre-Arnaud Marcelot <pa...@marcelot.net>]

On 20 oct. 2011, at 12:33, Alex Karasulu wrote:

> On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 10/20/11 12:06 PM, Alex Karasulu wrote:
> On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>  wrote:
> 
> I think you first need to define your constraints and what you want to
> focus on.
> 
> Excellent point so let's recap a little. In the API we have certain
> functionality extension points:
> 
> 1). Codec Extensions
> 
> These extension points are for adding support LDAP extended operation and
> LDAP control support so the API can handle them explicitly as structured
> elements rather than opaque unstructured data. This is pretty well defined
> and understood by the Directory Team I think and we opted for using a
> non-OSGi based ClassLoader mechanism. If this has changed please correct me.
>  
> Those codec are not supposed to be written by external users.
> 
> I disagree with the this statement above. If codec extensions are not to be written by non-API committers then why are we going to all this trouble class loading them and making them pluggable?
> 
> Our users might want to implement a control using our API for an LDAP server which exposes controls or extended operations which we have not supported. This provides a means for them to extend our API and achieve their goals.

+1.
Controls and extended operations should be extensible by users of the API.
Same thing for Schema elements (like comparators, normalizers, syntax checkers, etc.)

>  This is just fine to keep them as they are. We can even make them plain OSGi services at some point. In any case, this is a non issue.
> 
> 
> This I agree with because it allows extension as-is. It's a clean straight forward solution for now.
>  
> 
> 2). LDAP Schema
> 
> Now from what I understand enabling new schema on clients via the API is the
> main issue introducing additional constraints requiring us to consider the
> use of fragments.
> 
> Is this correct?
> Correct, sir !
> 
> 
> :-D
>  
> 
> Another part of the problem is making this work in both a non-OSGi client
> side API environment in addition to an OSGi based environment in both
> ApacheDS and Studio.
> 
> Is this correct?
> Aye aye, sir !
> 
> 
> Since when was I knighted by the Queen :-D.
>  
> 
> Also we had some conversations in the past of not actually using OSGi to
> load schema into the API. Sorry I don't have a email reference to the past
> thread. Did our position change on this topic in the recent past?
> Nope, this is exactly the problem we have : be able to class-load those schema extensions the ol' Java way (ie, class.forName() )
> 
> 
> Sorry to beat a dead horse here but OK so we are not trying to use OSGi for loading schema in all environments: client API, apacheds, and studio?

We need to have a solution which works with or without an OSGI container.

> 
> If you want someone to be able to write a jar to extend the api which will
> work in a non osgi environment and (with minimal changes) in an osgi
> environment, go for fragments.
> 
> Forgive me, but the fragment approach seems like a hack. Fragments were
> created in OSGi to handle intractable problems with split packages where you
> had no way to merge the split. Now we're using it as a main stream feature
> to work around these hairy issues. Please note that I am not saying we
> should abandon it, just playing devil's advocate here. We obviously need
> more thought on the constraints which you've made very apparent here in your
> post. Thanks for it.
> Seems currently to be the only path we can use. We may ask on Felix's mailing list to see if there is some other way, but I trust Guillaume here.
> 
> 
> This is not a trust issue with anyone. It's a matter of exploration and lack of being up to date on my part. Hope no one misinterprets my questions as a lack of trust. I'm finding discrepancies in what I learned in the past with the proposed direction today and asking about it. 

Given my small experiment, it seems to do the trick (I've tested it on smaller use-case/environment).
But, it may not be the best solution or there may be other solutions to achieve this.


> If you want to go a cleaner osgi way, go for services, but forget about
> class names in the schema directly, and you need to define two different
> ways, one for osgi and one in a non osgi environment (as you won't have osgi
> services at all in a flat classloader).
> 
> It's all about trade-offs.
> 
> 
> Right.
> Yeah...
> 
> 
> 
> 
> When talking with Emmanuel this week, it seems to me that for the api,
> extending the api was not a very common operation and did not really require
> the osgi dynamism.  Fragments are perfect for those simple use cases.
> 
> Yes but the extension should not happen in the same package. Hence we're
> using this feature for something it was not intended for. Again just
> pointing this out, not saying it's not ideal. Perhaps this is fact is not at
> all that important in the end.
> 
> can you be a bit more explicit here, for people like me who ae semi-OSGi-agnostic ?
> (damn, I *knew* I should have jumped into the OSGi wagon one year ago, at least... That's the problem when beig lazzy :/ )
> 
> 
> Well I jumped on the wagon several times and fell off repeatedly without concrete application opportunities to solidify this knowledge. Guillaume is perhaps the most experienced having used OSGi practically for years now.
> 
> From my limited understanding the fragments feature was created to handle situations when you had split packages but could not merge those packages because you did not have access to the code or some sh*t like that.
> 
> These extensions should not be in the same package space. Those adding schema or other extensions should not be adding them into the same packages used by the API. In that case my logic leads me to the question: why use fragments? 

Guillaume can correct me here if I'm wrong but, there should be absolutely no issue with package names.
One of the examples for the use of fragments is localization.
You could have a specific fragment for each language containing the properties file with the proper string translations.
Each fragment is independent and can be added without requiring a release of the original host bundle (which contains the code).
Everything being under the same package (of course splitted into the different bundles).

> But for sure, if you ask an OSGi purist, the recommandation will be to not
> use fragments.
> 
> 
> Right and maybe we should not be purists ;).
> 
> Purity is good for virgins. We aren't...
> 
> 
> Heh, yeah but sometimes non-virgins like to look like virgins :-D.

I won't add anything to that part of the discussion... ;) :p

Regards,
Pierre-Arnaud


> 
> -- 
> Best Regards,
> -- Alex
> 

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Guillaume Nodet <gn...@gmail.com>]

I think the original point of fragments is more extending a bundle rather
than solving split package issues (though, that's a common use case now, but
more as a hack when porting non osgi libraries to osgi), see [1].  So imho,
fragments are not hacks, but the real use cases are not very common.

The api use case is a good one for fragments imho, especially if you want
the api to be easily used in a non osgi environment, such as web apps.
It's a good use case because what you really want is just to make sure the
needed classes are available to the classloader, you have no real service
dependencies.   I'm sure that could be modelled using services, but that
looks like a non intuitive solution to me, especially if your entry point is
the class name.

The main problem I see with using services is really that you will need two
different logics (one when inside the osgi container and one outside), which
means writing those extensions will be more complicated too if you need to
support both environment.   Also, my experience in writing such dual support
is that you sometime find problems when people start embedding those
libraries in a war and deploy them in osgi: you're in a mix situation where
the jars are deployed in osgi, but not as real bundles.

[1]
http://books.google.fr/books?id=ROGxuH3ffVsC&pg=PA153&lpg=PA153&dq=fragment+purpose+osgi&source=bl&ots=WuIZ5WKU7q&sig=DMkOX8Z34DW5c4Lm7FnIrwEn2gQ&hl=fr&ei=vP6fTqvZH43n-gaVmJykBQ&sa=X&oi=book_result&ct=result&resnum=10&ved=0CH8Q6AEwCQ#v=onepage&q=fragment%20purpose%20osgi&f=false

On Thu, Oct 20, 2011 at 12:33, Alex Karasulu <ak...@apache.org> wrote:

> On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny <el...@gmail.com>wrote:
>
>> On 10/20/11 12:06 PM, Alex Karasulu wrote:
>>
>>> On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>
>>>  wrote:
>>>
>>>  I think you first need to define your constraints and what you want to
>>>> focus on.
>>>>
>>>>  Excellent point so let's recap a little. In the API we have certain
>>> functionality extension points:
>>>
>>> 1). Codec Extensions
>>>
>>> These extension points are for adding support LDAP extended operation and
>>> LDAP control support so the API can handle them explicitly as structured
>>> elements rather than opaque unstructured data. This is pretty well
>>> defined
>>> and understood by the Directory Team I think and we opted for using a
>>> non-OSGi based ClassLoader mechanism. If this has changed please correct
>>> me.
>>>
>>
>
>>  Those codec are not supposed to be written by external users.
>
>
> I disagree with the this statement above. If codec extensions are not to be
> written by non-API committers then why are we going to all this trouble
> class loading them and making them pluggable?
>
> Our users might want to implement a control using our API for an LDAP
> server which exposes controls or extended operations which we have not
> supported. This provides a means for them to extend our API and achieve
> their goals.
>
>
>> This is just fine to keep them as they are. We can even make them plain
>> OSGi services at some point. In any case, this is a non issue.
>>
>>
> This I agree with because it allows extension as-is. It's a clean straight
> forward solution for now.
>
>
>>
>>> 2). LDAP Schema
>>>
>>> Now from what I understand enabling new schema on clients via the API is
>>> the
>>> main issue introducing additional constraints requiring us to consider
>>> the
>>> use of fragments.
>>>
>>> Is this correct?
>>>
>> Correct, sir !
>>
>>
> :-D
>
>
>>
>>> Another part of the problem is making this work in both a non-OSGi client
>>> side API environment in addition to an OSGi based environment in both
>>> ApacheDS and Studio.
>>>
>>> Is this correct?
>>>
>> Aye aye, sir !
>>
>>
> Since when was I knighted by the Queen :-D.
>
>
>>
>>> Also we had some conversations in the past of not actually using OSGi to
>>> load schema into the API. Sorry I don't have a email reference to the
>>> past
>>> thread. Did our position change on this topic in the recent past?
>>>
>> Nope, this is exactly the problem we have : be able to class-load those
>> schema extensions the ol' Java way (ie, class.forName() )
>>
>>
> Sorry to beat a dead horse here but OK so we are not trying to use OSGi for
> loading schema in all environments: client API, apacheds, and studio?
>
>
>>
>>> If you want someone to be able to write a jar to extend the api which
>>> will
>>>
>>>> work in a non osgi environment and (with minimal changes) in an osgi
>>>> environment, go for fragments.
>>>>
>>>>  Forgive me, but the fragment approach seems like a hack. Fragments were
>>> created in OSGi to handle intractable problems with split packages where
>>> you
>>> had no way to merge the split. Now we're using it as a main stream
>>> feature
>>> to work around these hairy issues. Please note that I am not saying we
>>> should abandon it, just playing devil's advocate here. We obviously need
>>> more thought on the constraints which you've made very apparent here in
>>> your
>>> post. Thanks for it.
>>>
>> Seems currently to be the only path we can use. We may ask on Felix's
>> mailing list to see if there is some other way, but I trust Guillaume here.
>>
>>
> This is not a trust issue with anyone. It's a matter of exploration and
> lack of being up to date on my part. Hope no one misinterprets my questions
> as a lack of trust. I'm finding discrepancies in what I learned in the past
> with the proposed direction today and asking about it.
>
>
>>
>>> If you want to go a cleaner osgi way, go for services, but forget about
>>>
>>>> class names in the schema directly, and you need to define two different
>>>> ways, one for osgi and one in a non osgi environment (as you won't have
>>>> osgi
>>>> services at all in a flat classloader).
>>>>
>>>> It's all about trade-offs.
>>>>
>>>>
>>>>  Right.
>>>
>> Yeah...
>>
>>
>>
>>>
>>>  When talking with Emmanuel this week, it seems to me that for the api,
>>>> extending the api was not a very common operation and did not really
>>>> require
>>>> the osgi dynamism.  Fragments are perfect for those simple use cases.
>>>>
>>>>  Yes but the extension should not happen in the same package. Hence
>>> we're
>>> using this feature for something it was not intended for. Again just
>>> pointing this out, not saying it's not ideal. Perhaps this is fact is not
>>> at
>>> all that important in the end.
>>>
>>
>> can you be a bit more explicit here, for people like me who ae
>> semi-OSGi-agnostic ?
>> (damn, I *knew* I should have jumped into the OSGi wagon one year ago, at
>> least... That's the problem when beig lazzy :/ )
>>
>>
> Well I jumped on the wagon several times and fell off repeatedly without
> concrete application opportunities to solidify this knowledge. Guillaume is
> perhaps the most experienced having used OSGi practically for years now.
>
> From my limited understanding the fragments feature was created to handle
> situations when you had split packages but could not merge those packages
> because you did not have access to the code or some sh*t like that.
>
> These extensions should not be in the same package space. Those adding
> schema or other extensions should not be adding them into the same packages
> used by the API. In that case my logic leads me to the question: why use
> fragments?
>
>
>>
>>
>>>
>>>  But for sure, if you ask an OSGi purist, the recommandation will be to
>>>> not
>>>> use fragments.
>>>>
>>>>
>>>>  Right and maybe we should not be purists ;).
>>>
>>
>> Purity is good for virgins. We aren't...
>>
>>
> Heh, yeah but sometimes non-virgins like to look like virgins :-D.
>
> --
> Best Regards,
> -- Alex
>
>


-- 
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Pierre-Arnaud Marcelot <pa...@marcelot.net>]

On 20 oct. 2011, at 13:38, Emmanuel Lécharny wrote:
>> I must suggest some other idea that will change the way we think. We can
>> embed OSGI inside shared. By embedding i really mean embedding. So our
>> shared api launchs its own OSGI framework and its own extension points. so
>> whether it is launched in or out of OSGI environment, it will have its own
>> execution context. And we provide that context just for extensions. It might
>> work like that too. But if you ask me, i say it is not necessary.
> 
> We did that months a go, intergrating Felix in the API. We decided it was not a good idea (don't remember extacly why). In any case, we removed felix from the API.Göktürk

For the record, here's the associated issue.
https://issues.apache.org/jira/browse/DIRSHARED-125

Regards,
Pierre-Arnaud

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Emmanuel Lécharny <el...@apache.org>]

On 10/20/11 1:19 PM, Göktürk Gezer wrote:
> Hi All,
>
> I see lots of discussions took place. So i would like to say some more.
>
> On Thu, Oct 20, 2011 at 1:58 PM, Emmanuel Lécharny<el...@apache.org>wrote:
>
>> On 10/20/11 12:33 PM, Alex Karasulu wrote:
>>
>>> On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny<el...@gmail.com>**
>>> wrote:
>>>
>>>   On 10/20/11 12:06 PM, Alex Karasulu wrote:
>>>>   On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>
>>>>>   wrote:
>>>>>
>>>>>   I think you first need to define your constraints and what you want to
>>>>>
>>>>>> focus on.
>>>>>>
>>>>>>   Excellent point so let's recap a little. In the API we have certain
>>>>>>
>>>>> functionality extension points:
>>>>>
>>>>> 1). Codec Extensions
>>>>>
>>>>> These extension points are for adding support LDAP extended operation
>>>>> and
>>>>> LDAP control support so the API can handle them explicitly as structured
>>>>> elements rather than opaque unstructured data. This is pretty well
>>>>> defined
>>>>> and understood by the Directory Team I think and we opted for using a
>>>>> non-OSGi based ClassLoader mechanism. If this has changed please correct
>>>>> me.
>>>>>
>>>>>    Those codec are not supposed to be written by external users.
>>> I disagree with the this statement above. If codec extensions are not to
>>> be
>>> written by non-API committers then why are we going to all this trouble
>>> class loading them and making them pluggable?
>>>
>> That's a good question. The problem is that in some case, and mostly on the
>> API, one codec for an extended operation *might* have different
>> implementations depending on the targeted software (and we probably can
>> thank M$ for such a mess).
>>
>>
>>> Our users might want to implement a control using our API for an LDAP
>>> server
>>> which exposes controls or extended operations which we have not supported.
>>> This provides a means for them to extend our API and achieve their goals.
>>>
>> Well, assuming that writing a codec is not frankly easy, I'd rather have a
>> request and write it myself...
>>
>>
>>>   Another part of the problem is making this work in both a non-OSGi client
>>>>> side API environment in addition to an OSGi based environment in both
>>>>> ApacheDS and Studio.
>>>>>
>>>>> Is this correct?
>>>>>
>>>>>   Aye aye, sir !
>>>>
>>>>   Since when was I knighted by the Queen :-D.
>> Well, I was more thinking about Full Metal Jacket, here :)
>>
>>
>>
>>>   Also we had some conversations in the past of not actually using OSGi to
>>>>> load schema into the API. Sorry I don't have a email reference to the
>>>>> past
>>>>> thread. Did our position change on this topic in the recent past?
>>>>>
>>>>>   Nope, this is exactly the problem we have : be able to class-load those
>>>> schema extensions the ol' Java way (ie, class.forName() )
>>>>
>>>>
>>>>   Sorry to beat a dead horse here but OK so we are not trying to use OSGi
>>> for
>>> loading schema in all environments: client API, apacheds, and studio?
>>>
>> If we can have a solution that cover the three cases, I buy it. If we have
>> to class load the Java way schema element because the API will be used
>> outside of an OSGi container, then be it.
>
> It didn't take much attention as i see, but we handled that topic yesterday
> :) I think preserving the shared api ıs a must. And if we want to provide a
> mechanism that can work on both OSGI and nonOSGI environment, fragments is
> not an option, it is a must too. But everybody is missing something here. By
> using fragments, we're not adding any new extension point other than it
> already provided. It is just a way to polish our looking in OSGI
> environment.for nonOSGI environment everything will be just same,hard.
>
> So the aim of the code i currently patched is to show we can seperate OSGI
> and nonOSGI environments.
>
> When launched inside an OSGI, that code creates the necessary listeners for
> new Comparator addition, in OSGI way using Service registrations. And use
> them internally.
>
> When it is launched outsıde the OSGI, it does not change anything and
> somebody can provide their extension as fragment or plain jar at all.
>
> So while providing fragment extensions out of the box, we also create an
> extra OSGI extension point right now(for Comparators) So we provide 2 main
> way. Its developer's choice to choose which one.

Ok, get it now. Thanks for the clarification.


> I must suggest some other idea that will change the way we think. We can
> embed OSGI inside shared. By embedding i really mean embedding. So our
> shared api launchs its own OSGI framework and its own extension points. so
> whether it is launched in or out of OSGI environment, it will have its own
> execution context. And we provide that context just for extensions. It might
> work like that too. But if you ask me, i say it is not necessary.

We did that months a go, intergrating Felix in the API. We decided it 
was not a good idea (don't remember extacly why). In any case, we 
removed felix from the API.Göktürk


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Göktürk Gezer <go...@gmail.com>]

On Thu, Oct 20, 2011 at 2:39 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net>wrote:

> Hi Göktürk,
>
> On 20 oct. 2011, at 13:19, Göktürk Gezer wrote:
>
> Hi All,
>
> I see lots of discussions took place. So i would like to say some more.
>
> On Thu, Oct 20, 2011 at 1:58 PM, Emmanuel Lécharny <el...@apache.org>wrote:
>
>> On 10/20/11 12:33 PM, Alex Karasulu wrote:
>>
>>> On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny<el...@gmail.com>*
>>> *wrote:
>>>
>>>  On 10/20/11 12:06 PM, Alex Karasulu wrote:
>>>>
>>>>  On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>
>>>>>  wrote:
>>>>>
>>>>>  I think you first need to define your constraints and what you want to
>>>>>
>>>>>> focus on.
>>>>>>
>>>>>>  Excellent point so let's recap a little. In the API we have certain
>>>>>>
>>>>> functionality extension points:
>>>>>
>>>>> 1). Codec Extensions
>>>>>
>>>>> These extension points are for adding support LDAP extended operation
>>>>> and
>>>>> LDAP control support so the API can handle them explicitly as
>>>>> structured
>>>>> elements rather than opaque unstructured data. This is pretty well
>>>>> defined
>>>>> and understood by the Directory Team I think and we opted for using a
>>>>> non-OSGi based ClassLoader mechanism. If this has changed please
>>>>> correct
>>>>> me.
>>>>>
>>>>>   Those codec are not supposed to be written by external users.
>>>>
>>>
>>> I disagree with the this statement above. If codec extensions are not to
>>> be
>>> written by non-API committers then why are we going to all this trouble
>>> class loading them and making them pluggable?
>>>
>> That's a good question. The problem is that in some case, and mostly on
>> the API, one codec for an extended operation *might* have different
>> implementations depending on the targeted software (and we probably can
>> thank M$ for such a mess).
>>
>>
>>> Our users might want to implement a control using our API for an LDAP
>>> server
>>> which exposes controls or extended operations which we have not
>>> supported.
>>> This provides a means for them to extend our API and achieve their goals.
>>>
>> Well, assuming that writing a codec is not frankly easy, I'd rather have a
>> request and write it myself...
>>
>>
>>>  Another part of the problem is making this work in both a non-OSGi
>>>>> client
>>>>> side API environment in addition to an OSGi based environment in both
>>>>> ApacheDS and Studio.
>>>>>
>>>>> Is this correct?
>>>>>
>>>>>  Aye aye, sir !
>>>>
>>>>
>>>>  Since when was I knighted by the Queen :-D.
>>>
>>
>> Well, I was more thinking about Full Metal Jacket, here :)
>>
>>
>>
>>>
>>>  Also we had some conversations in the past of not actually using OSGi to
>>>>> load schema into the API. Sorry I don't have a email reference to the
>>>>> past
>>>>> thread. Did our position change on this topic in the recent past?
>>>>>
>>>>>  Nope, this is exactly the problem we have : be able to class-load
>>>> those
>>>> schema extensions the ol' Java way (ie, class.forName() )
>>>>
>>>>
>>>>  Sorry to beat a dead horse here but OK so we are not trying to use OSGi
>>> for
>>> loading schema in all environments: client API, apacheds, and studio?
>>>
>> If we can have a solution that cover the three cases, I buy it. If we have
>> to class load the Java way schema element because the API will be used
>> outside of an OSGi container, then be it.
>
>
> It didn't take much attention as i see, but we handled that topic yesterday
> :) I think preserving the shared api ıs a must. And if we want to provide a
> mechanism that can work on both OSGI and nonOSGI environment, fragments is
> not an option, it is a must too. But everybody is missing something here. By
> using fragments, we're not adding any new extension point other than it
> already provided. It is just a way to polish our looking in OSGI
> environment.for nonOSGI environment everything will be just same,hard.
>
>
> Actually, no. It's a way to solve in an OSGI environment the class loading
> of external classes we're currently facing.
> At the moment, we have no way to provide a third party comparator for
> example. We won't be able to load the class since it's defined in another
> bundle.
>
I just meant, loading some plain jar file in classpath and referencing its
classname through some schema.ldif file. Not much like extendibility :)

>
> So the aim of the code i currently patched is to show we can seperate OSGI
> and nonOSGI environments.
>
> When launched inside an OSGI, that code creates the necessary listeners for
> new Comparator addition, in OSGI way using Service registrations. And use
> them internally.
>
> When it is launched outsıde the OSGI, it does not change anything and
> somebody can provide their extension as fragment or plain jar at all.
>
> So while providing fragment extensions out of the box, we also create an
> extra OSGI extension point right now(for Comparators) So we provide 2 main
> way. Its developer's choice to choose which one.
>
>
> Interesting. I have yet to see how it works. I have imported your patch,
> but didn't have the time to play with the code yet.
> I'm going to do this right now.
>
Code is working but remember, it only handles the comparator part. You must
get errors on syntaxcheckers first. it was Comparators throwing exceptions
first. By the way, to preserve the old tests and shared completely. I will
mode the logic out from DefaultSchemaManager class. By creating a new class.
ApacheDS will even be executable without OSGI for now.

So don't use that patch. I will slightly modify it.

>
>
>>>  If you want someone to be able to write a jar to extend the api which
>>>>> will
>>>>>
>>>>>  work in a non osgi environment and (with minimal changes) in an osgi
>>>>>> environment, go for fragments.
>>>>>>
>>>>>>  Forgive me, but the fragment approach seems like a hack. Fragments
>>>>>> were
>>>>>>
>>>>> created in OSGi to handle intractable problems with split packages
>>>>> where
>>>>> you
>>>>> had no way to merge the split. Now we're using it as a main stream
>>>>> feature
>>>>> to work around these hairy issues. Please note that I am not saying we
>>>>> should abandon it, just playing devil's advocate here. We obviously
>>>>> need
>>>>> more thought on the constraints which you've made very apparent here in
>>>>> your
>>>>> post. Thanks for it.
>>>>>
>>>>>  Seems currently to be the only path we can use. We may ask on Felix's
>>>> mailing list to see if there is some other way, but I trust Guillaume
>>>> here.
>>>>
>>>>
>>>>  This is not a trust issue with anyone.
>>>
>> Well, when I wrote 'I trust Guillaume', that does not mean I don't trsut
>> you. It's just that I have not the background to dispute his knowledge.
>>
>>
>>  It's a matter of exploration and lack
>>> of being up to date on my part. Hope no one misinterprets my questions as
>>> a
>>> lack of trust. I'm finding discrepancies in what I learned in the past
>>> with
>>> the proposed direction today and asking about it.
>>>
>>
>> I must admit that I'm more a lurker than an actor of this discussion. I
>> can just brng some light on what we currently have, and the needs that we
>> have.
>>
>>>
>>>
>>>> can you be a bit more explicit here, for people like me who ae
>>>> semi-OSGi-agnostic ?
>>>> (damn, I *knew* I should have jumped into the OSGi wagon one year ago,
>>>> at
>>>> least... That's the problem when beig lazzy :/ )
>>>>
>>>>
>>>>  > From my limited understanding the fragments feature was created to
>>> handle
>>> situations when you had split packages but could not merge those packages
>>> because you did not have access to the code or some sh*t like that.
>>>
>>> These extensions should not be in the same package space. Those adding
>>> schema or other extensions should not be adding them into the same
>>> packages
>>> used by the API. In that case my logic leads me to the question: why use
>>> fragments?
>>>
>>>
>>>  I'm not sure that simply declaring extensions in a different package is
>> enough. It seems more like a class-loader and visibility issue in this case,
>> AFAIU.
>
> fragments outside the OSGI will give you an global class visibility. In
> OSGI they will be merged with intended host (shared-ldap-model in our case)
> and its packages will be served under its classloader. So no problem here.
> If we want to use them use them, they are not that evil. What i must say is,
> we don't have to do anything for supporting it right now. The way we load
> schema elements (classnames comes from schema ldifs) already let them work
> out of the box right now.
>
>>
>>
>> --
>> Regards,
>> Cordialement,
>> Emmanuel Lécharny
>> www.iktek.com
>>
>>
> I must suggest some other idea that will change the way we think. We can
> embed OSGI inside shared. By embedding i really mean embedding. So our
> shared api launchs its own OSGI framework and its own extension points. so
> whether it is launched in or out of OSGI environment, it will have its own
> execution context. And we provide that context just for extensions. It might
> work like that too. But if you ask me, i say it is not necessary.
>
>
> Indeed, that's one thing we already tried back in February.
> I think we had to rollback this because we had no great way to correctly
> shutdown the Felix instance at the end.
> Also it was kind of heavy to have to launch a full OSGI container in order
> to use an LDAP API.
>
> Regards,
> Pierre-Arnaud
>
> Regards,
> Göktürk
>
>
>

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Pierre-Arnaud Marcelot <pa...@marcelot.net>]

Hi Göktürk,

On 20 oct. 2011, at 13:19, Göktürk Gezer wrote:
> Hi All,
> 
> I see lots of discussions took place. So i would like to say some more.
> 
> On Thu, Oct 20, 2011 at 1:58 PM, Emmanuel Lécharny <el...@apache.org> wrote:
> On 10/20/11 12:33 PM, Alex Karasulu wrote:
> On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny<el...@gmail.com>wrote:
> 
> On 10/20/11 12:06 PM, Alex Karasulu wrote:
> 
> On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>
>  wrote:
> 
>  I think you first need to define your constraints and what you want to
> focus on.
> 
>  Excellent point so let's recap a little. In the API we have certain
> functionality extension points:
> 
> 1). Codec Extensions
> 
> These extension points are for adding support LDAP extended operation and
> LDAP control support so the API can handle them explicitly as structured
> elements rather than opaque unstructured data. This is pretty well defined
> and understood by the Directory Team I think and we opted for using a
> non-OSGi based ClassLoader mechanism. If this has changed please correct
> me.
> 
>  Those codec are not supposed to be written by external users.
> 
> I disagree with the this statement above. If codec extensions are not to be
> written by non-API committers then why are we going to all this trouble
> class loading them and making them pluggable?
> That's a good question. The problem is that in some case, and mostly on the API, one codec for an extended operation *might* have different implementations depending on the targeted software (and we probably can thank M$ for such a mess).
> 
> 
> Our users might want to implement a control using our API for an LDAP server
> which exposes controls or extended operations which we have not supported.
> This provides a means for them to extend our API and achieve their goals.
> Well, assuming that writing a codec is not frankly easy, I'd rather have a request and write it myself...
> 
> 
> Another part of the problem is making this work in both a non-OSGi client
> side API environment in addition to an OSGi based environment in both
> ApacheDS and Studio.
> 
> Is this correct?
> 
> Aye aye, sir !
> 
> 
> Since when was I knighted by the Queen :-D.
> 
> Well, I was more thinking about Full Metal Jacket, here :)
> 
> 
> 
> 
> Also we had some conversations in the past of not actually using OSGi to
> load schema into the API. Sorry I don't have a email reference to the past
> thread. Did our position change on this topic in the recent past?
> 
> Nope, this is exactly the problem we have : be able to class-load those
> schema extensions the ol' Java way (ie, class.forName() )
> 
> 
> Sorry to beat a dead horse here but OK so we are not trying to use OSGi for
> loading schema in all environments: client API, apacheds, and studio?
> If we can have a solution that cover the three cases, I buy it. If we have to class load the Java way schema element because the API will be used outside of an OSGi container, then be it.
> 
> It didn't take much attention as i see, but we handled that topic yesterday :) I think preserving the shared api ıs a must. And if we want to provide a mechanism that can work on both OSGI and nonOSGI environment, fragments is not an option, it is a must too. But everybody is missing something here. By using fragments, we're not adding any new extension point other than it already provided. It is just a way to polish our looking in OSGI environment.for nonOSGI environment everything will be just same,hard.

Actually, no. It's a way to solve in an OSGI environment the class loading of external classes we're currently facing.
At the moment, we have no way to provide a third party comparator for example. We won't be able to load the class since it's defined in another bundle. 

> So the aim of the code i currently patched is to show we can seperate OSGI and nonOSGI environments. 
> 
> When launched inside an OSGI, that code creates the necessary listeners for new Comparator addition, in OSGI way using Service registrations. And use them internally. 
> 
> When it is launched outsıde the OSGI, it does not change anything and somebody can provide their extension as fragment or plain jar at all. 
> 
> So while providing fragment extensions out of the box, we also create an extra OSGI extension point right now(for Comparators) So we provide 2 main way. Its developer's choice to choose which one.

Interesting. I have yet to see how it works. I have imported your patch, but didn't have the time to play with the code yet.
I'm going to do this right now.

> 
> If you want someone to be able to write a jar to extend the api which will
> 
> work in a non osgi environment and (with minimal changes) in an osgi
> environment, go for fragments.
> 
>  Forgive me, but the fragment approach seems like a hack. Fragments were
> created in OSGi to handle intractable problems with split packages where
> you
> had no way to merge the split. Now we're using it as a main stream feature
> to work around these hairy issues. Please note that I am not saying we
> should abandon it, just playing devil's advocate here. We obviously need
> more thought on the constraints which you've made very apparent here in
> your
> post. Thanks for it.
> 
> Seems currently to be the only path we can use. We may ask on Felix's
> mailing list to see if there is some other way, but I trust Guillaume here.
> 
> 
> This is not a trust issue with anyone.
> Well, when I wrote 'I trust Guillaume', that does not mean I don't trsut you. It's just that I have not the background to dispute his knowledge.
> 
> 
> It's a matter of exploration and lack
> of being up to date on my part. Hope no one misinterprets my questions as a
> lack of trust. I'm finding discrepancies in what I learned in the past with
> the proposed direction today and asking about it.
> 
> I must admit that I'm more a lurker than an actor of this discussion. I can just brng some light on what we currently have, and the needs that we have.
> 
> 
> can you be a bit more explicit here, for people like me who ae
> semi-OSGi-agnostic ?
> (damn, I *knew* I should have jumped into the OSGi wagon one year ago, at
> least... That's the problem when beig lazzy :/ )
> 
> 
> > From my limited understanding the fragments feature was created to handle
> situations when you had split packages but could not merge those packages
> because you did not have access to the code or some sh*t like that.
> 
> These extensions should not be in the same package space. Those adding
> schema or other extensions should not be adding them into the same packages
> used by the API. In that case my logic leads me to the question: why use
> fragments?
> 
> 
> I'm not sure that simply declaring extensions in a different package is enough. It seems more like a class-loader and visibility issue in this case, AFAIU.
> fragments outside the OSGI will give you an global class visibility. In OSGI they will be merged with intended host (shared-ldap-model in our case) and its packages will be served under its classloader. So no problem here. If we want to use them use them, they are not that evil. What i must say is, we don't have to do anything for supporting it right now. The way we load schema elements (classnames comes from schema ldifs) already let them work out of the box right now.
> 
> 
> -- 
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
> 
> 
> I must suggest some other idea that will change the way we think. We can embed OSGI inside shared. By embedding i really mean embedding. So our shared api launchs its own OSGI framework and its own extension points. so whether it is launched in or out of OSGI environment, it will have its own execution context. And we provide that context just for extensions. It might work like that too. But if you ask me, i say it is not necessary.

Indeed, that's one thing we already tried back in February.
I think we had to rollback this because we had no great way to correctly shutdown the Felix instance at the end.
Also it was kind of heavy to have to launch a full OSGI container in order to use an LDAP API.

Regards,
Pierre-Arnaud

> Regards,
> Göktürk

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Göktürk Gezer <go...@gmail.com>]

Hi All,

I see lots of discussions took place. So i would like to say some more.

On Thu, Oct 20, 2011 at 1:58 PM, Emmanuel Lécharny <el...@apache.org>wrote:

> On 10/20/11 12:33 PM, Alex Karasulu wrote:
>
>> On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny<el...@gmail.com>**
>> wrote:
>>
>>  On 10/20/11 12:06 PM, Alex Karasulu wrote:
>>>
>>>  On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>
>>>>  wrote:
>>>>
>>>>  I think you first need to define your constraints and what you want to
>>>>
>>>>> focus on.
>>>>>
>>>>>  Excellent point so let's recap a little. In the API we have certain
>>>>>
>>>> functionality extension points:
>>>>
>>>> 1). Codec Extensions
>>>>
>>>> These extension points are for adding support LDAP extended operation
>>>> and
>>>> LDAP control support so the API can handle them explicitly as structured
>>>> elements rather than opaque unstructured data. This is pretty well
>>>> defined
>>>> and understood by the Directory Team I think and we opted for using a
>>>> non-OSGi based ClassLoader mechanism. If this has changed please correct
>>>> me.
>>>>
>>>>   Those codec are not supposed to be written by external users.
>>>
>>
>> I disagree with the this statement above. If codec extensions are not to
>> be
>> written by non-API committers then why are we going to all this trouble
>> class loading them and making them pluggable?
>>
> That's a good question. The problem is that in some case, and mostly on the
> API, one codec for an extended operation *might* have different
> implementations depending on the targeted software (and we probably can
> thank M$ for such a mess).
>
>
>> Our users might want to implement a control using our API for an LDAP
>> server
>> which exposes controls or extended operations which we have not supported.
>> This provides a means for them to extend our API and achieve their goals.
>>
> Well, assuming that writing a codec is not frankly easy, I'd rather have a
> request and write it myself...
>
>
>>  Another part of the problem is making this work in both a non-OSGi client
>>>> side API environment in addition to an OSGi based environment in both
>>>> ApacheDS and Studio.
>>>>
>>>> Is this correct?
>>>>
>>>>  Aye aye, sir !
>>>
>>>
>>>  Since when was I knighted by the Queen :-D.
>>
>
> Well, I was more thinking about Full Metal Jacket, here :)
>
>
>
>>
>>  Also we had some conversations in the past of not actually using OSGi to
>>>> load schema into the API. Sorry I don't have a email reference to the
>>>> past
>>>> thread. Did our position change on this topic in the recent past?
>>>>
>>>>  Nope, this is exactly the problem we have : be able to class-load those
>>> schema extensions the ol' Java way (ie, class.forName() )
>>>
>>>
>>>  Sorry to beat a dead horse here but OK so we are not trying to use OSGi
>> for
>> loading schema in all environments: client API, apacheds, and studio?
>>
> If we can have a solution that cover the three cases, I buy it. If we have
> to class load the Java way schema element because the API will be used
> outside of an OSGi container, then be it.


It didn't take much attention as i see, but we handled that topic yesterday
:) I think preserving the shared api ıs a must. And if we want to provide a
mechanism that can work on both OSGI and nonOSGI environment, fragments is
not an option, it is a must too. But everybody is missing something here. By
using fragments, we're not adding any new extension point other than it
already provided. It is just a way to polish our looking in OSGI
environment.for nonOSGI environment everything will be just same,hard.

So the aim of the code i currently patched is to show we can seperate OSGI
and nonOSGI environments.

When launched inside an OSGI, that code creates the necessary listeners for
new Comparator addition, in OSGI way using Service registrations. And use
them internally.

When it is launched outsıde the OSGI, it does not change anything and
somebody can provide their extension as fragment or plain jar at all.

So while providing fragment extensions out of the box, we also create an
extra OSGI extension point right now(for Comparators) So we provide 2 main
way. Its developer's choice to choose which one.

>
>
>>
>>  If you want someone to be able to write a jar to extend the api which
>>>> will
>>>>
>>>>  work in a non osgi environment and (with minimal changes) in an osgi
>>>>> environment, go for fragments.
>>>>>
>>>>>  Forgive me, but the fragment approach seems like a hack. Fragments
>>>>> were
>>>>>
>>>> created in OSGi to handle intractable problems with split packages where
>>>> you
>>>> had no way to merge the split. Now we're using it as a main stream
>>>> feature
>>>> to work around these hairy issues. Please note that I am not saying we
>>>> should abandon it, just playing devil's advocate here. We obviously need
>>>> more thought on the constraints which you've made very apparent here in
>>>> your
>>>> post. Thanks for it.
>>>>
>>>>  Seems currently to be the only path we can use. We may ask on Felix's
>>> mailing list to see if there is some other way, but I trust Guillaume
>>> here.
>>>
>>>
>>>  This is not a trust issue with anyone.
>>
> Well, when I wrote 'I trust Guillaume', that does not mean I don't trsut
> you. It's just that I have not the background to dispute his knowledge.
>
>
>  It's a matter of exploration and lack
>> of being up to date on my part. Hope no one misinterprets my questions as
>> a
>> lack of trust. I'm finding discrepancies in what I learned in the past
>> with
>> the proposed direction today and asking about it.
>>
>
> I must admit that I'm more a lurker than an actor of this discussion. I can
> just brng some light on what we currently have, and the needs that we have.
>
>>
>>
>>> can you be a bit more explicit here, for people like me who ae
>>> semi-OSGi-agnostic ?
>>> (damn, I *knew* I should have jumped into the OSGi wagon one year ago, at
>>> least... That's the problem when beig lazzy :/ )
>>>
>>>
>>>  > From my limited understanding the fragments feature was created to
>> handle
>> situations when you had split packages but could not merge those packages
>> because you did not have access to the code or some sh*t like that.
>>
>> These extensions should not be in the same package space. Those adding
>> schema or other extensions should not be adding them into the same
>> packages
>> used by the API. In that case my logic leads me to the question: why use
>> fragments?
>>
>>
>>  I'm not sure that simply declaring extensions in a different package is
> enough. It seems more like a class-loader and visibility issue in this case,
> AFAIU.

fragments outside the OSGI will give you an global class visibility. In OSGI
they will be merged with intended host (shared-ldap-model in our case) and
its packages will be served under its classloader. So no problem here. If we
want to use them use them, they are not that evil. What i must say is, we
don't have to do anything for supporting it right now. The way we load
schema elements (classnames comes from schema ldifs) already let them work
out of the box right now.

>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
>
I must suggest some other idea that will change the way we think. We can
embed OSGI inside shared. By embedding i really mean embedding. So our
shared api launchs its own OSGI framework and its own extension points. so
whether it is launched in or out of OSGI environment, it will have its own
execution context. And we provide that context just for extensions. It might
work like that too. But if you ask me, i say it is not necessary.


Regards,
Göktürk

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Emmanuel Lécharny <el...@apache.org>]

On 10/20/11 12:33 PM, Alex Karasulu wrote:
> On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny<el...@gmail.com>wrote:
>
>> On 10/20/11 12:06 PM, Alex Karasulu wrote:
>>
>>> On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>
>>>   wrote:
>>>
>>>   I think you first need to define your constraints and what you want to
>>>> focus on.
>>>>
>>>>   Excellent point so let's recap a little. In the API we have certain
>>> functionality extension points:
>>>
>>> 1). Codec Extensions
>>>
>>> These extension points are for adding support LDAP extended operation and
>>> LDAP control support so the API can handle them explicitly as structured
>>> elements rather than opaque unstructured data. This is pretty well defined
>>> and understood by the Directory Team I think and we opted for using a
>>> non-OSGi based ClassLoader mechanism. If this has changed please correct
>>> me.
>>>
>>   Those codec are not supposed to be written by external users.
>
> I disagree with the this statement above. If codec extensions are not to be
> written by non-API committers then why are we going to all this trouble
> class loading them and making them pluggable?
That's a good question. The problem is that in some case, and mostly on 
the API, one codec for an extended operation *might* have different 
implementations depending on the targeted software (and we probably can 
thank M$ for such a mess).
>
> Our users might want to implement a control using our API for an LDAP server
> which exposes controls or extended operations which we have not supported.
> This provides a means for them to extend our API and achieve their goals.
Well, assuming that writing a codec is not frankly easy, I'd rather have 
a request and write it myself...
>
>>> Another part of the problem is making this work in both a non-OSGi client
>>> side API environment in addition to an OSGi based environment in both
>>> ApacheDS and Studio.
>>>
>>> Is this correct?
>>>
>> Aye aye, sir !
>>
>>
> Since when was I knighted by the Queen :-D.

Well, I was more thinking about Full Metal Jacket, here :)

>
>
>>> Also we had some conversations in the past of not actually using OSGi to
>>> load schema into the API. Sorry I don't have a email reference to the past
>>> thread. Did our position change on this topic in the recent past?
>>>
>> Nope, this is exactly the problem we have : be able to class-load those
>> schema extensions the ol' Java way (ie, class.forName() )
>>
>>
> Sorry to beat a dead horse here but OK so we are not trying to use OSGi for
> loading schema in all environments: client API, apacheds, and studio?
If we can have a solution that cover the three cases, I buy it. If we 
have to class load the Java way schema element because the API will be 
used outside of an OSGi container, then be it.
>
>
>>> If you want someone to be able to write a jar to extend the api which will
>>>
>>>> work in a non osgi environment and (with minimal changes) in an osgi
>>>> environment, go for fragments.
>>>>
>>>>   Forgive me, but the fragment approach seems like a hack. Fragments were
>>> created in OSGi to handle intractable problems with split packages where
>>> you
>>> had no way to merge the split. Now we're using it as a main stream feature
>>> to work around these hairy issues. Please note that I am not saying we
>>> should abandon it, just playing devil's advocate here. We obviously need
>>> more thought on the constraints which you've made very apparent here in
>>> your
>>> post. Thanks for it.
>>>
>> Seems currently to be the only path we can use. We may ask on Felix's
>> mailing list to see if there is some other way, but I trust Guillaume here.
>>
>>
> This is not a trust issue with anyone.
Well, when I wrote 'I trust Guillaume', that does not mean I don't trsut 
you. It's just that I have not the background to dispute his knowledge.

> It's a matter of exploration and lack
> of being up to date on my part. Hope no one misinterprets my questions as a
> lack of trust. I'm finding discrepancies in what I learned in the past with
> the proposed direction today and asking about it.

I must admit that I'm more a lurker than an actor of this discussion. I 
can just brng some light on what we currently have, and the needs that 
we have.
>
>>
>> can you be a bit more explicit here, for people like me who ae
>> semi-OSGi-agnostic ?
>> (damn, I *knew* I should have jumped into the OSGi wagon one year ago, at
>> least... That's the problem when beig lazzy :/ )
>>
>>
> > From my limited understanding the fragments feature was created to handle
> situations when you had split packages but could not merge those packages
> because you did not have access to the code or some sh*t like that.
>
> These extensions should not be in the same package space. Those adding
> schema or other extensions should not be adding them into the same packages
> used by the API. In that case my logic leads me to the question: why use
> fragments?
>
>
I'm not sure that simply declaring extensions in a different package is 
enough. It seems more like a class-loader and visibility issue in this 
case, AFAIU.

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Alex Karasulu <ak...@apache.org>]

On Thu, Oct 20, 2011 at 1:18 PM, Emmanuel Lecharny <el...@gmail.com>wrote:

> On 10/20/11 12:06 PM, Alex Karasulu wrote:
>
>> On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>
>>  wrote:
>>
>>  I think you first need to define your constraints and what you want to
>>> focus on.
>>>
>>>  Excellent point so let's recap a little. In the API we have certain
>> functionality extension points:
>>
>> 1). Codec Extensions
>>
>> These extension points are for adding support LDAP extended operation and
>> LDAP control support so the API can handle them explicitly as structured
>> elements rather than opaque unstructured data. This is pretty well defined
>> and understood by the Directory Team I think and we opted for using a
>> non-OSGi based ClassLoader mechanism. If this has changed please correct
>> me.
>>
>

>  Those codec are not supposed to be written by external users.


I disagree with the this statement above. If codec extensions are not to be
written by non-API committers then why are we going to all this trouble
class loading them and making them pluggable?

Our users might want to implement a control using our API for an LDAP server
which exposes controls or extended operations which we have not supported.
This provides a means for them to extend our API and achieve their goals.


> This is just fine to keep them as they are. We can even make them plain
> OSGi services at some point. In any case, this is a non issue.
>
>
This I agree with because it allows extension as-is. It's a clean straight
forward solution for now.


>
>> 2). LDAP Schema
>>
>> Now from what I understand enabling new schema on clients via the API is
>> the
>> main issue introducing additional constraints requiring us to consider the
>> use of fragments.
>>
>> Is this correct?
>>
> Correct, sir !
>
>
:-D


>
>> Another part of the problem is making this work in both a non-OSGi client
>> side API environment in addition to an OSGi based environment in both
>> ApacheDS and Studio.
>>
>> Is this correct?
>>
> Aye aye, sir !
>
>
Since when was I knighted by the Queen :-D.


>
>> Also we had some conversations in the past of not actually using OSGi to
>> load schema into the API. Sorry I don't have a email reference to the past
>> thread. Did our position change on this topic in the recent past?
>>
> Nope, this is exactly the problem we have : be able to class-load those
> schema extensions the ol' Java way (ie, class.forName() )
>
>
Sorry to beat a dead horse here but OK so we are not trying to use OSGi for
loading schema in all environments: client API, apacheds, and studio?


>
>> If you want someone to be able to write a jar to extend the api which will
>>
>>> work in a non osgi environment and (with minimal changes) in an osgi
>>> environment, go for fragments.
>>>
>>>  Forgive me, but the fragment approach seems like a hack. Fragments were
>> created in OSGi to handle intractable problems with split packages where
>> you
>> had no way to merge the split. Now we're using it as a main stream feature
>> to work around these hairy issues. Please note that I am not saying we
>> should abandon it, just playing devil's advocate here. We obviously need
>> more thought on the constraints which you've made very apparent here in
>> your
>> post. Thanks for it.
>>
> Seems currently to be the only path we can use. We may ask on Felix's
> mailing list to see if there is some other way, but I trust Guillaume here.
>
>
This is not a trust issue with anyone. It's a matter of exploration and lack
of being up to date on my part. Hope no one misinterprets my questions as a
lack of trust. I'm finding discrepancies in what I learned in the past with
the proposed direction today and asking about it.


>
>> If you want to go a cleaner osgi way, go for services, but forget about
>>
>>> class names in the schema directly, and you need to define two different
>>> ways, one for osgi and one in a non osgi environment (as you won't have
>>> osgi
>>> services at all in a flat classloader).
>>>
>>> It's all about trade-offs.
>>>
>>>
>>>  Right.
>>
> Yeah...
>
>
>
>>
>>  When talking with Emmanuel this week, it seems to me that for the api,
>>> extending the api was not a very common operation and did not really
>>> require
>>> the osgi dynamism.  Fragments are perfect for those simple use cases.
>>>
>>>  Yes but the extension should not happen in the same package. Hence we're
>> using this feature for something it was not intended for. Again just
>> pointing this out, not saying it's not ideal. Perhaps this is fact is not
>> at
>> all that important in the end.
>>
>
> can you be a bit more explicit here, for people like me who ae
> semi-OSGi-agnostic ?
> (damn, I *knew* I should have jumped into the OSGi wagon one year ago, at
> least... That's the problem when beig lazzy :/ )
>
>
Well I jumped on the wagon several times and fell off repeatedly without
concrete application opportunities to solidify this knowledge. Guillaume is
perhaps the most experienced having used OSGi practically for years now.

>From my limited understanding the fragments feature was created to handle
situations when you had split packages but could not merge those packages
because you did not have access to the code or some sh*t like that.

These extensions should not be in the same package space. Those adding
schema or other extensions should not be adding them into the same packages
used by the API. In that case my logic leads me to the question: why use
fragments?


>
>
>>
>>  But for sure, if you ask an OSGi purist, the recommandation will be to
>>> not
>>> use fragments.
>>>
>>>
>>>  Right and maybe we should not be purists ;).
>>
>
> Purity is good for virgins. We aren't...
>
>
Heh, yeah but sometimes non-virgins like to look like virgins :-D.

-- 
Best Regards,
-- Alex

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Emmanuel Lecharny <el...@gmail.com>]

On 10/20/11 12:06 PM, Alex Karasulu wrote:
> On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet<gn...@gmail.com>  wrote:
>
>> I think you first need to define your constraints and what you want to
>> focus on.
>>
> Excellent point so let's recap a little. In the API we have certain
> functionality extension points:
>
> 1). Codec Extensions
>
> These extension points are for adding support LDAP extended operation and
> LDAP control support so the API can handle them explicitly as structured
> elements rather than opaque unstructured data. This is pretty well defined
> and understood by the Directory Team I think and we opted for using a
> non-OSGi based ClassLoader mechanism. If this has changed please correct me.
Those codec are not supposed to be written by external users. This is 
just fine to keep them as they are. We can even make them plain OSGi 
services at some point. In any case, this is a non issue.
>
> 2). LDAP Schema
>
> Now from what I understand enabling new schema on clients via the API is the
> main issue introducing additional constraints requiring us to consider the
> use of fragments.
>
> Is this correct?
Correct, sir !
>
> Another part of the problem is making this work in both a non-OSGi client
> side API environment in addition to an OSGi based environment in both
> ApacheDS and Studio.
>
> Is this correct?
Aye aye, sir !
>
> Also we had some conversations in the past of not actually using OSGi to
> load schema into the API. Sorry I don't have a email reference to the past
> thread. Did our position change on this topic in the recent past?
Nope, this is exactly the problem we have : be able to class-load those 
schema extensions the ol' Java way (ie, class.forName() )
>
> If you want someone to be able to write a jar to extend the api which will
>> work in a non osgi environment and (with minimal changes) in an osgi
>> environment, go for fragments.
>>
> Forgive me, but the fragment approach seems like a hack. Fragments were
> created in OSGi to handle intractable problems with split packages where you
> had no way to merge the split. Now we're using it as a main stream feature
> to work around these hairy issues. Please note that I am not saying we
> should abandon it, just playing devil's advocate here. We obviously need
> more thought on the constraints which you've made very apparent here in your
> post. Thanks for it.
Seems currently to be the only path we can use. We may ask on Felix's 
mailing list to see if there is some other way, but I trust Guillaume here.
>
> If you want to go a cleaner osgi way, go for services, but forget about
>> class names in the schema directly, and you need to define two different
>> ways, one for osgi and one in a non osgi environment (as you won't have osgi
>> services at all in a flat classloader).
>>
>> It's all about trade-offs.
>>
>>
> Right.
Yeah...

>
>
>> When talking with Emmanuel this week, it seems to me that for the api,
>> extending the api was not a very common operation and did not really require
>> the osgi dynamism.  Fragments are perfect for those simple use cases.
>>
> Yes but the extension should not happen in the same package. Hence we're
> using this feature for something it was not intended for. Again just
> pointing this out, not saying it's not ideal. Perhaps this is fact is not at
> all that important in the end.

can you be a bit more explicit here, for people like me who ae 
semi-OSGi-agnostic ?
(damn, I *knew* I should have jumped into the OSGi wagon one year ago, 
at least... That's the problem when beig lazzy :/ )

>
>
>> But for sure, if you ask an OSGi purist, the recommandation will be to not
>> use fragments.
>>
>>
> Right and maybe we should not be purists ;).

Purity is good for virgins. We aren't...


-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Alex Karasulu <ak...@apache.org>]

On Thu, Oct 20, 2011 at 12:43 PM, Guillaume Nodet <gn...@gmail.com> wrote:

> I think you first need to define your constraints and what you want to
> focus on.
>

Excellent point so let's recap a little. In the API we have certain
functionality extension points:

1). Codec Extensions

These extension points are for adding support LDAP extended operation and
LDAP control support so the API can handle them explicitly as structured
elements rather than opaque unstructured data. This is pretty well defined
and understood by the Directory Team I think and we opted for using a
non-OSGi based ClassLoader mechanism. If this has changed please correct me.

2). LDAP Schema

Now from what I understand enabling new schema on clients via the API is the
main issue introducing additional constraints requiring us to consider the
use of fragments.

Is this correct?

Another part of the problem is making this work in both a non-OSGi client
side API environment in addition to an OSGi based environment in both
ApacheDS and Studio.

Is this correct?

Also we had some conversations in the past of not actually using OSGi to
load schema into the API. Sorry I don't have a email reference to the past
thread. Did our position change on this topic in the recent past?

If you want someone to be able to write a jar to extend the api which will
> work in a non osgi environment and (with minimal changes) in an osgi
> environment, go for fragments.
>

Forgive me, but the fragment approach seems like a hack. Fragments were
created in OSGi to handle intractable problems with split packages where you
had no way to merge the split. Now we're using it as a main stream feature
to work around these hairy issues. Please note that I am not saying we
should abandon it, just playing devil's advocate here. We obviously need
more thought on the constraints which you've made very apparent here in your
post. Thanks for it.

If you want to go a cleaner osgi way, go for services, but forget about
> class names in the schema directly, and you need to define two different
> ways, one for osgi and one in a non osgi environment (as you won't have osgi
> services at all in a flat classloader).
>
> It's all about trade-offs.
>
>
Right.


> When talking with Emmanuel this week, it seems to me that for the api,
> extending the api was not a very common operation and did not really require
> the osgi dynamism.  Fragments are perfect for those simple use cases.
>

Yes but the extension should not happen in the same package. Hence we're
using this feature for something it was not intended for. Again just
pointing this out, not saying it's not ideal. Perhaps this is fact is not at
all that important in the end.


> But for sure, if you ask an OSGi purist, the recommandation will be to not
> use fragments.
>
>
Right and maybe we should not be purists ;).

Regards,
Alex


> On Thu, Oct 20, 2011 at 11:29, Alex Karasulu <ak...@apache.org> wrote:
>
>> Excuse the pre post but my comment is somewhat general. Overall the Felix
>> and general OSGi community considers fragments as a problematic construct in
>> the specification: something they recommend you avoid but if stuck you can
>> use. It's use is discouraged.
>>
>>
>> On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net>wrote:
>>
>>> Hi Dev,
>>>
>>> Following our efforts on the OSGI side, I took some time to think about
>>> (and experiment) how we could solve our extensibility issue on the LDAP API,
>>> where we want our users to be able to provide their own custom
>>> implementation of various schema objects like comparators, normalizers, etc.
>>>
>>> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out
>>> using OSGI fragments bundles could probably be our best solution.
>>>
>>> Here's a definition of what an OSGI fragment bundle is (more information
>>> available at [1]):
>>> "An OSGi fragment is a Java archive file with specific manifest headers
>>> that enable it to attach to a specified host bundle or specified host
>>> bundles in order to function.
>>> Fragments are treated as part of the host bundles. Relevant definitions
>>> of the fragment are merged with the host bundles definitions before the host
>>> is resolved, as long as the information does not conflict.
>>> Fragment dependencies are resolved if possible. If the fragment
>>> dependencies can not be resolved, the fragment does not attach to the host
>>> bundle.
>>> A fragment can not have its own class loader or bundle activator. It can
>>> not override the information present in the host bundles. Fragments extend
>>> bundles with resources, classes, and permitted headers enabling you to
>>> customize your bundles."
>>>
>>> The great thing about fragments is that they *share* the same class
>>> loader as their host bundle. Which was pretty much the kind of issue we were
>>> having with classloaders being differents from one bundle to the other.
>>>
>>> The other great thing I see with this approach, it that it would also
>>> work great outside of an OSGI container application (which is a strong
>>> requirement for the LDAP API). A fragment bundle is nothing else than a
>>> regular bundle with a specific OSGI directive (Fragment-Host) added to its
>>> MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when
>>> it's not included in an OSGI container. Thus, it would allow us to support
>>> third parties extensions.
>>>
>>> I have done a small experiment in my Eclipse workspace with two bundles
>>> (one being the host and the other being the fragment ) and I was able to
>>> classload, without any classloader issue, a class defined in the fragment
>>> bundle from another class inside the host bundle.
>>>
>>> I'd like to go a step further and experiment this on the API itself.
>>>
>>> Ideally, I'd like to use the 'shared-ldap-model' module
>>> ('org.apache.directory.shared.ldap.model' bundle) as a host for all our
>>> schema objects implementations and move these implementations into a
>>> specific fragment bundle containing them all.
>>> Users would have to do the same to include their extensions. A simple
>>> fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as
>>> host and it works.
>>>
>>> One other thing that should be done, is to let the 'shared-ldap-model'
>>> module do the instanciation of the schema elements. At the moment, two
>>> classes from the 'shared-ldap-schema-data' module are responsible for this,
>>> 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and
>>> 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We
>>> would need to move these classes to the 'shared-ldap-model' module, for them
>>> to have access to the right class loader.
>>>
>>> All in all, I think that's the only things we need to do to get the
>>> extensibility we wanted inside and outside an OSGI container.
>>>
>>> What do you think of this plan?
>>>
>>> Regards,
>>> Pierre-Arnaud
>>>
>>> [1] -
>>> http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html
>>>
>>
>>
>>
>> --
>> Best Regards,
>> -- Alex
>>
>>
>
>
> --
> ------------------------
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com
>
>


-- 
Best Regards,
-- Alex

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Emmanuel Lecharny <el...@gmail.com>]

On 10/20/11 11:43 AM, Guillaume Nodet wrote:
> I think you first need to define your constraints and what you want to focus
> on.
> If you want someone to be able to write a jar to extend the api which will
> work in a non osgi environment and (with minimal changes) in an osgi
> environment, go for fragments.
> If you want to go a cleaner osgi way, go for services, but forget about
> class names in the schema directly, and you need to define two different
> ways, one for osgi and one in a non osgi environment (as you won't have osgi
> services at all in a flat classloader).
>
> It's all about trade-offs.
>
> When talking with Emmanuel this week, it seems to me that for the api,
> extending the api was not a very common operation and did not really require
> the osgi dynamism.  Fragments are perfect for those simple use cases.   But
> for sure, if you ask an OSGi purist, the recommandation will be to not use
> fragments.

We have two different needs :
1) provide a OSGi compliant server, including an OSGi container for 
those who don't have one. That means we include Felix, and we will have 
a mode that let the user set his own container, excluding Felix in this 
case. We will have to provide two launchers to cover those two different 
cases.
In any case, all the extension points will be plain OSGi services.
2) For the API, this is a different story, as the API will most 
certainly be used outside an OSGi container, except for ADS and Studio, 
which will run into Felix (ADS) and Equinox (Studio). The extension 
points will be plain java classes, class-loaded using their FQCN or even 
loading the bytecode stored in an entry.

This is somehow what we discussed with Guillaume yesterday, and I do 
think that fragments are a good trde-off for the API.

At least, it seems to work...

-- 
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Guillaume Nodet <gn...@gmail.com>]

I think you first need to define your constraints and what you want to focus
on.
If you want someone to be able to write a jar to extend the api which will
work in a non osgi environment and (with minimal changes) in an osgi
environment, go for fragments.
If you want to go a cleaner osgi way, go for services, but forget about
class names in the schema directly, and you need to define two different
ways, one for osgi and one in a non osgi environment (as you won't have osgi
services at all in a flat classloader).

It's all about trade-offs.

When talking with Emmanuel this week, it seems to me that for the api,
extending the api was not a very common operation and did not really require
the osgi dynamism.  Fragments are perfect for those simple use cases.   But
for sure, if you ask an OSGi purist, the recommandation will be to not use
fragments.

On Thu, Oct 20, 2011 at 11:29, Alex Karasulu <ak...@apache.org> wrote:

> Excuse the pre post but my comment is somewhat general. Overall the Felix
> and general OSGi community considers fragments as a problematic construct in
> the specification: something they recommend you avoid but if stuck you can
> use. It's use is discouraged.
>
>
> On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net>wrote:
>
>> Hi Dev,
>>
>> Following our efforts on the OSGI side, I took some time to think about
>> (and experiment) how we could solve our extensibility issue on the LDAP API,
>> where we want our users to be able to provide their own custom
>> implementation of various schema objects like comparators, normalizers, etc.
>>
>> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out
>> using OSGI fragments bundles could probably be our best solution.
>>
>> Here's a definition of what an OSGI fragment bundle is (more information
>> available at [1]):
>> "An OSGi fragment is a Java archive file with specific manifest headers
>> that enable it to attach to a specified host bundle or specified host
>> bundles in order to function.
>> Fragments are treated as part of the host bundles. Relevant definitions of
>> the fragment are merged with the host bundles definitions before the host is
>> resolved, as long as the information does not conflict.
>> Fragment dependencies are resolved if possible. If the fragment
>> dependencies can not be resolved, the fragment does not attach to the host
>> bundle.
>> A fragment can not have its own class loader or bundle activator. It can
>> not override the information present in the host bundles. Fragments extend
>> bundles with resources, classes, and permitted headers enabling you to
>> customize your bundles."
>>
>> The great thing about fragments is that they *share* the same class loader
>> as their host bundle. Which was pretty much the kind of issue we were having
>> with classloaders being differents from one bundle to the other.
>>
>> The other great thing I see with this approach, it that it would also work
>> great outside of an OSGI container application (which is a strong
>> requirement for the LDAP API). A fragment bundle is nothing else than a
>> regular bundle with a specific OSGI directive (Fragment-Host) added to its
>> MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when
>> it's not included in an OSGI container. Thus, it would allow us to support
>> third parties extensions.
>>
>> I have done a small experiment in my Eclipse workspace with two bundles
>> (one being the host and the other being the fragment ) and I was able to
>> classload, without any classloader issue, a class defined in the fragment
>> bundle from another class inside the host bundle.
>>
>> I'd like to go a step further and experiment this on the API itself.
>>
>> Ideally, I'd like to use the 'shared-ldap-model' module
>> ('org.apache.directory.shared.ldap.model' bundle) as a host for all our
>> schema objects implementations and move these implementations into a
>> specific fragment bundle containing them all.
>> Users would have to do the same to include their extensions. A simple
>> fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as
>> host and it works.
>>
>> One other thing that should be done, is to let the 'shared-ldap-model'
>> module do the instanciation of the schema elements. At the moment, two
>> classes from the 'shared-ldap-schema-data' module are responsible for this,
>> 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and
>> 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We
>> would need to move these classes to the 'shared-ldap-model' module, for them
>> to have access to the right class loader.
>>
>> All in all, I think that's the only things we need to do to get the
>> extensibility we wanted inside and outside an OSGI container.
>>
>> What do you think of this plan?
>>
>> Regards,
>> Pierre-Arnaud
>>
>> [1] -
>> http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html
>>
>
>
>
> --
> Best Regards,
> -- Alex
>
>


-- 
------------------------
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com

Re: [API] Experimenting OSGI fragments to solve the extensibility issue

[Alex Karasulu <ak...@apache.org>]

Excuse the pre post but my comment is somewhat general. Overall the Felix
and general OSGi community considers fragments as a problematic construct in
the specification: something they recommend you avoid but if stuck you can
use. It's use is discouraged.


On Wed, Oct 19, 2011 at 6:36 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net>wrote:

> Hi Dev,
>
> Following our efforts on the OSGI side, I took some time to think about
> (and experiment) how we could solve our extensibility issue on the LDAP API,
> where we want our users to be able to provide their own custom
> implementation of various schema objects like comparators, normalizers, etc.
>
> After a quick discussion with Emmanuel and Guillaume Nodet, it turned out
> using OSGI fragments bundles could probably be our best solution.
>
> Here's a definition of what an OSGI fragment bundle is (more information
> available at [1]):
> "An OSGi fragment is a Java archive file with specific manifest headers
> that enable it to attach to a specified host bundle or specified host
> bundles in order to function.
> Fragments are treated as part of the host bundles. Relevant definitions of
> the fragment are merged with the host bundles definitions before the host is
> resolved, as long as the information does not conflict.
> Fragment dependencies are resolved if possible. If the fragment
> dependencies can not be resolved, the fragment does not attach to the host
> bundle.
> A fragment can not have its own class loader or bundle activator. It can
> not override the information present in the host bundles. Fragments extend
> bundles with resources, classes, and permitted headers enabling you to
> customize your bundles."
>
> The great thing about fragments is that they *share* the same class loader
> as their host bundle. Which was pretty much the kind of issue we were having
> with classloaders being differents from one bundle to the other.
>
> The other great thing I see with this approach, it that it would also work
> great outside of an OSGI container application (which is a strong
> requirement for the LDAP API). A fragment bundle is nothing else than a
> regular bundle with a specific OSGI directive (Fragment-Host) added to its
> MANIFEST.MF file, and a bundle behaves exactly like a plain jar file when
> it's not included in an OSGI container. Thus, it would allow us to support
> third parties extensions.
>
> I have done a small experiment in my Eclipse workspace with two bundles
> (one being the host and the other being the fragment ) and I was able to
> classload, without any classloader issue, a class defined in the fragment
> bundle from another class inside the host bundle.
>
> I'd like to go a step further and experiment this on the API itself.
>
> Ideally, I'd like to use the 'shared-ldap-model' module
> ('org.apache.directory.shared.ldap.model' bundle) as a host for all our
> schema objects implementations and move these implementations into a
> specific fragment bundle containing them all.
> Users would have to do the same to include their extensions. A simple
> fragment bundle with the 'org.apache.directory.shared.ldap.model' bundle as
> host and it works.
>
> One other thing that should be done, is to let the 'shared-ldap-model'
> module do the instanciation of the schema elements. At the moment, two
> classes from the 'shared-ldap-schema-data' module are responsible for this,
> 'org.apache.directory.shared.ldap.schemaloader.SchemaEntityFactory' and
> 'org.apache.directory.shared.ldap.schemaloader.AttributeClassLoader'. We
> would need to move these classes to the 'shared-ldap-model' module, for them
> to have access to the right class loader.
>
> All in all, I think that's the only things we need to do to get the
> extensibility we wanted inside and outside an OSGI container.
>
> What do you think of this plan?
>
> Regards,
> Pierre-Arnaud
>
> [1] -
> http://publib.boulder.ibm.com/infocenter/radhelp/v8/index.jsp?topic=/com.ibm.osgi.common.doc/topics/cbundlefragment.html
>



-- 
Best Regards,
-- Alex