You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Alex Karasulu <ak...@apache.org> on 2011/05/11 17:35:20 UTC
[LDAP API] Codec extension mechanism direction (WAS: Re: OSGi startup
and shutdown problems)
Hi dev peeps,
So after a long thread I just wanted to summarize the realizations and
conclusions so we can set a clear direction for managing the codec extension
mechanism. I created a separate clean thread for this here with Guillaume's
core recommendation following.
For the sake of speed I will define a direction and people can opine:
1). Expose a means in the LDAP API (really SPI) to have codec extensions
programmatically registered. I this already exists.
2). Remove the standalone codec factory implementation that starts up Felix.
3). Add a simple ClassLoader component to be used in standalone mode to load
the plugin classes (from the plugin bundles defaulting to regular jars
presumed to be on the classpath). Some configuration information drives how
this component discovers what plugin classes to attempt to class load.
4). Set it up so by default, the LDAP API uses the simple ClassLoader based
discover/load/register mechanism. In an OSGi environment this is disabled to
enable plugins to self register.
This should serious remove some ugly and dangerous code we've got at this
point in the LDAP API.
Thoughts?
Thanks,
Alex
On Wed, May 11, 2011 at 6:14 PM, Alex Karasulu <ak...@apache.org> wrote:
>
>
> On Wed, May 11, 2011 at 6:03 PM, Guillaume Nodet <gn...@gmail.com> wrote:
>
>> What we've done for Camel for example is to allow those libraries to
>> leverage OSGi if deployed in OSGi and default to a basic mode in other
>> cases.
>>
>> For Camel, components can be create by users (though I guess it's the
>> same here, not average users). Camel can discover components using a
>> pluggable discovery mechanism. When deployed in OSGi, Camel
>> automatically select an OSGi aware discovery mechanism that will be
>> able to look into existing bundles for such components (well, it's in
>> reality slightly more complicacted, but thats the idea). When in a
>> non OSGi environement, the default discovery mechanism just scans for
>> META-INF/services/xxx which define camel components.
>>
>> The idea here is that if the user wants to use OSGi, he'll be able to
>> use the OSGi modularity layer and dynamism to deploy things at runtime
>> and all the OSGi goodness. Else, we just use a very simple and basic
>> things that works very nicely in a flat classloader.
>>
>>
> That sounds very reasonable to me. This is the best path. Thanks Guillaume
> for helping us see this clearly.
>
Re: [LDAP API] Codec extension mechanism direction (WAS: Re: OSGi
startup and shutdown problems)
Posted by Guillaume Nodet <gn...@gmail.com>.
In Karaf, we have a simple provisioning mechanism called 'features'.
You can see an example at [1]
In its simpliest form, it's a list of urls pointing to OSGi bundles.
We usually use maven urls (which are real urls with a custom url
handler) so that we can download easily from configured maven
repositories.
We're working on automatic generation of such features from maven for
karaf 3.x, not sure when that'll be ready though (but we've lived with
hand-written ones for a while without problems).
Another way is to use OBR (OSGi Bundle Repository) and the maven
bundle plugin can easily generate obr xmls from an existing repo using
the bundle:index goal at [2]
Anyway, as long as Directory provide bundles, that should be enough.
If a user wants to install it in its own framework, he'll use a
provisioning mechanism of his choice. Directory can help by providing
pre-built files as described above, but at the end, what's really
needed is a list of bundles to install.
[1] http://svn.apache.org/viewvc/karaf/branches/karaf-2.2.x/assemblies/features/standard/src/main/resources/features.xml?view=markup
[2] http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html#ApacheFelixMavenBundlePlugin%28BND%29-bundle%3Aindex
On Wed, May 11, 2011 at 18:46, Alex Karasulu <ak...@apache.org> wrote:
> On Wed, May 11, 2011 at 6:56 PM, Emmanuel Lecharny <el...@gmail.com>
> wrote:
>>
>> On 5/11/11 5:35 PM, Alex Karasulu wrote:
>>>
>>> Hi dev peeps,
>>>
>>> So after a long thread I just wanted to summarize the realizations and
>>> conclusions so we can set a clear direction for managing the codec
>>> extension
>>> mechanism. I created a separate clean thread for this here with
>>> Guillaume's
>>> core recommendation following.
>>>
>>> For the sake of speed I will define a direction and people can opine:
>>>
>>> 1). Expose a means in the LDAP API (really SPI) to have codec extensions
>>> programmatically registered. I this already exists.
>>
>> +1
>>>
>>> 2). Remove the standalone codec factory implementation that starts up
>>> Felix.
>>
>> Ok, that would be a relief.
>>>
>>> 3). Add a simple ClassLoader component to be used in standalone mode to
>>> load
>>> the plugin classes (from the plugin bundles defaulting to regular jars
>>> presumed to be on the classpath). Some configuration information drives
>>> how
>>> this component discovers what plugin classes to attempt to class load.
>>
>> Ok, fine.
>>>
>>> 4). Set it up so by default, the LDAP API uses the simple ClassLoader
>>> based
>>> discover/load/register mechanism. In an OSGi environment this is disabled
>>> to
>>> enable plugins to self register.
>>
>> Ok.
>>>
>>> This should serious remove some ugly and dangerous code we've got at this
>>> point in the LDAP API.
>>
>> Absolutely.
>>
>> Question : are we sure that teh API will continue to work in Studio, using
>> Equinox ? And are we sure we will be able to have felix started in ApacheDS
>> ?
>>
>
> Actually I think it is using equinox at the present moment. If I remember
> correctly we're not using the standalone mechanism when in the eclipse
> environment. PAM I think found a way to get the bundle loaded under eclipse
> OSGi (a.k.a. equinox). Might want to get confirmation from PAM though.
>
>>
>> That raises another related issue for ADS : what if an application is
>> already using an OSGi framework and wants to embed ADS ?
>
> That's not a problem since basically they can load just the bundles. We can
> carry ApacheDS with installers inside Karaf with our standalone
> distribution. Those loading ADS into an existing environment need only issue
> a load from the bundle repository. I think there's some magic there making
> your local maven repository into an OSGi bundle repository. I wonder if
> Karaf knows how to pull bundles down too from remote repositories, I bet
> Guillaume can answer that :).
> Regards,
> Alex
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com
Connect at CamelOne May 24-26
The Open Source Integration Conference
http://camelone.com/
Re: Some basic question
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/12/11 12:02 PM, Rajesh Akkineni wrote:
> Hi,
>>> So, If I need to do some kind of storage optimization, I can
>>> implement my own subclasses of Value and implement my own
>>> externalization logic?
>>> And it should not cause much problem to existing code, right?
>> Why would you do that ? Again, we didn't spent any time thinking
>> about optimizing this part because the gain would be absolutely
>> unnoticed. There are way more time consummed in many other parts of
>> the server, and any kind of optimisation in this area would just be a
>> waste of time...
>>
>> Just do one thing : write a simple test looping on some entries,
>> searching for a boolean value or an integer value, and check with a
>> profiler where exactly the time is being consumed...
>
> I understand your point. My question is a theoretical one. Anyway now
> I understand why those are not implemented.
> Surly I will look at other places to do optimization.
If you are digging in this area, we have left some work to be done. Let
me explain.
A LDAP server is used to store data, some of them could be quite large
(pictures, large texts, etc). Right now, with the StringValue and
BinaryValue, those values are stored in memory. When we created those
classes we knew that at some point, we would like to have some streamed
version of those classes, by extending them.
A streamed value will contain a pointer to the file storing the real
value, but the access to the data will be the same. That means we must
be able to access the data byte by byte (or char by char), but reading
them from a file.
This is a part we didn't had time to work on, but most certainly
something worth some love...
>
>>
>>
>
>
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some basic question
Posted by Rajesh Akkineni <ra...@vimukti.com>.
Hi,
On Thu, 12 May 2011 15:28:13 +0530, Emmanuel Lecharny
<el...@gmail.com> wrote:
> On 5/12/11 11:51 AM, Rajesh Akkineni wrote:
>> Hi Emmanuel,
>>
>> Thank you for quick answers...
>>
>>>> 4) When do Normalizers get called? LDAP v3 RFC is not talking
>>>> anything about normalization.
>>> Normalizers are used because we need to compare values, and some AT
>>> are case insensitive, or spaces aren't meaningful, so we have to
>>> 'normalize' those values to ease the comparison task.
>>
>> Do these comparisons are done only on SearchOperation? If Yes, When we
>> need to save normalized values fist of all. Why can not we do it in the
>> Comparator itself?
> Because every time you do a search and need to compare a value, if it's
> not already normalized, you will have to do it again, and again, and
> again...
>>
Got it.
>>>> 5) Numeric, Boolean values are stored as strings in store? If Yes,
>>>> Why? Will it not be optimal to store them in more compact form?
>>> The Boolean value is supposed o be TRUE or FALSE, nothing else (upper
>>> case). Numeric values are not defined as bounded values.
>>>
>>> In general, LDAP is about manipulating Strings, so such an
>>> optimization would be totally useless.
>>
>> So, If I need to do some kind of storage optimization, I can implement
>> my own subclasses of Value and implement my own externalization logic?
>> And it should not cause much problem to existing code, right?
> Why would you do that ? Again, we didn't spent any time thinking about
> optimizing this part because the gain would be absolutely unnoticed.
> There are way more time consummed in many other parts of the server, and
> any kind of optimisation in this area would just be a waste of time...
>
> Just do one thing : write a simple test looping on some entries,
> searching for a boolean value or an integer value, and check with a
> profiler where exactly the time is being consumed...
I understand your point. My question is a theoretical one. Anyway now I
understand why those are not implemented.
Surly I will look at other places to do optimization.
>
>
--
Rajesh Akkineni
CEO, Vimukti Technologies,
Re: Some basic question
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/12/11 11:51 AM, Rajesh Akkineni wrote:
> Hi Emmanuel,
>
> Thank you for quick answers...
>
>>> 4) When do Normalizers get called? LDAP v3 RFC is not talking
>>> anything about normalization.
>> Normalizers are used because we need to compare values, and some AT
>> are case insensitive, or spaces aren't meaningful, so we have to
>> 'normalize' those values to ease the comparison task.
>
> Do these comparisons are done only on SearchOperation? If Yes, When we
> need to save normalized values fist of all. Why can not we do it in
> the Comparator itself?
Because every time you do a search and need to compare a value, if it's
not already normalized, you will have to do it again, and again, and
again...
>
>
>>> 5) Numeric, Boolean values are stored as strings in store? If Yes,
>>> Why? Will it not be optimal to store them in more compact form?
>> The Boolean value is supposed o be TRUE or FALSE, nothing else (upper
>> case). Numeric values are not defined as bounded values.
>>
>> In general, LDAP is about manipulating Strings, so such an
>> optimization would be totally useless.
>
> So, If I need to do some kind of storage optimization, I can implement
> my own subclasses of Value and implement my own externalization logic?
> And it should not cause much problem to existing code, right?
Why would you do that ? Again, we didn't spent any time thinking about
optimizing this part because the gain would be absolutely unnoticed.
There are way more time consummed in many other parts of the server, and
any kind of optimisation in this area would just be a waste of time...
Just do one thing : write a simple test looping on some entries,
searching for a boolean value or an integer value, and check with a
profiler where exactly the time is being consumed...
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Some basic question
Posted by Rajesh Akkineni <ra...@vimukti.com>.
Hi Emmanuel,
Thank you for quick answers...
>> 4) When do Normalizers get called? LDAP v3 RFC is not talking anything
>> about normalization.
> Normalizers are used because we need to compare values, and some AT are
> case insensitive, or spaces aren't meaningful, so we have to 'normalize'
> those values to ease the comparison task.
Do these comparisons are done only on SearchOperation? If Yes, When we
need to save normalized values fist of all. Why can not we do it in the
Comparator itself?
>> 5) Numeric, Boolean values are stored as strings in store? If Yes, Why?
>> Will it not be optimal to store them in more compact form?
> The Boolean value is supposed o be TRUE or FALSE, nothing else (upper
> case). Numeric values are not defined as bounded values.
>
> In general, LDAP is about manipulating Strings, so such an optimization
> would be totally useless.
So, If I need to do some kind of storage optimization, I can implement my
own subclasses of Value and implement my own externalization logic?
And it should not cause much problem to existing code, right?
>
> Hope it helps.
>
>
--
Rajesh Akkineni
CEO, Vimukti Technologies,
Phone: +91-9989696514
Skype: akkinenirajesh
Re: Some basic question
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/12/11 9:45 AM, Rajesh Akkineni wrote:
> Hi,
>
> I was going trough the source of ApacheDS and trying to understand its
> functionality.
> I have the following question..along with some assumptions which I
> want to confirm.
>
> 1) Each Entry is saved as key-value pair where key is Long and value
> is serialized form of Entry. Is it correct?
Yes.
>
> 2) All attribute values are stored along with serialized entry as
> StringValue, BinayValue. Right?
Right.
>
> 3) Why these two Values save wrappedValues where they can only save
> normalziedValue?
because we have to return the value has the user provided it.
For instance, if you created an entry with a givename = Rasjesh, you
want to get it as 'Rajesh', not 'rajesh'
>
> 4) When do Normalizers get called? LDAP v3 RFC is not talking anything
> about normalization.
Normalizers are used because we need to compare values, and some AT are
case insensitive, or spaces aren't meaningful, so we have to 'normalize'
those values to ease the comparison task.
>
> 5) Numeric, Boolean values are stored as strings in store? If Yes,
> Why? Will it not be optimal to store them in more compact form?
The Boolean value is supposed o be TRUE or FALSE, nothing else (upper
case). Numeric values are not defined as bounded values.
In general, LDAP is about manipulating Strings, so such an optimization
would be totally useless.
>
> 6) Which syntaxes are stored as String and which are stored as Byte[]?
Human-readable (HR) values are stored as String, others are stored as
byte[]. HR ATs are defined in the RFC or by adding a ;binary option in
the attribute name. We default to String.
Hope it helps.
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Some basic question
Posted by Rajesh Akkineni <ra...@vimukti.com>.
Hi,
I was going trough the source of ApacheDS and trying to understand its
functionality.
I have the following question..along with some assumptions which I want to
confirm.
1) Each Entry is saved as key-value pair where key is Long and value is
serialized form of Entry. Is it correct?
2) All attribute values are stored along with serialized entry as
StringValue, BinayValue. Right?
3) Why these two Values save wrappedValues where they can only save
normalziedValue?
4) When do Normalizers get called? LDAP v3 RFC is not talking anything
about normalization.
5) Numeric, Boolean values are stored as strings in store? If Yes, Why?
Will it not be optimal to store them in more compact form?
6) Which syntaxes are stored as String and which are stored as Byte[]?
Let me know if my question are stupid... :)
--
Rajesh Akkineni
Re: [LDAP API] Codec extension mechanism direction (WAS: Re: OSGi startup and shutdown problems)
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
On 11 mai 2011, at 18:46, Alex Karasulu wrote:
> On Wed, May 11, 2011 at 6:56 PM, Emmanuel Lecharny <el...@gmail.com> wrote:
> On 5/11/11 5:35 PM, Alex Karasulu wrote:
> Hi dev peeps,
>
> So after a long thread I just wanted to summarize the realizations and
> conclusions so we can set a clear direction for managing the codec extension
> mechanism. I created a separate clean thread for this here with Guillaume's
> core recommendation following.
>
> For the sake of speed I will define a direction and people can opine:
>
> 1). Expose a means in the LDAP API (really SPI) to have codec extensions
> programmatically registered. I this already exists.
>
> +1
>
> 2). Remove the standalone codec factory implementation that starts up Felix.
>
> Ok, that would be a relief.
>
> 3). Add a simple ClassLoader component to be used in standalone mode to load
> the plugin classes (from the plugin bundles defaulting to regular jars
> presumed to be on the classpath). Some configuration information drives how
> this component discovers what plugin classes to attempt to class load.
> Ok, fine.
>
> 4). Set it up so by default, the LDAP API uses the simple ClassLoader based
> discover/load/register mechanism. In an OSGi environment this is disabled to
> enable plugins to self register.
> Ok.
>
> This should serious remove some ugly and dangerous code we've got at this
> point in the LDAP API.
> Absolutely.
>
> Question : are we sure that teh API will continue to work in Studio, using Equinox ? And are we sure we will be able to have felix started in ApacheDS ?
>
>
> Actually I think it is using equinox at the present moment. If I remember correctly we're not using the standalone mechanism when in the eclipse environment. PAM I think found a way to get the bundle loaded under eclipse OSGi (a.k.a. equinox). Might want to get confirmation from PAM though.
Yeah, the current implementation works fine in Equinox, without the standalone mechanisms. Custom extensions are registered via bundle activators.
I'll keep an eye and help to maintain the complete OSGI compliance as we go forward the refactoring.
> That raises another related issue for ADS : what if an application is already using an OSGi framework and wants to embed ADS ?
I guess, the easiest thing would be to follow what we did in Studio:
- Deploy the Shared/LDAP API bundles in the OSGI framework
- Declare dependencies on Shared/LDAP API artifacts as provided in pom.xml files
- Access classes as usual
Regards,
Pierre-Arnaud
Re: [LDAP API] Codec extension mechanism direction (WAS: Re: OSGi
startup and shutdown problems)
Posted by Alex Karasulu <ak...@apache.org>.
On Wed, May 11, 2011 at 6:56 PM, Emmanuel Lecharny <el...@gmail.com>wrote:
> On 5/11/11 5:35 PM, Alex Karasulu wrote:
>
>> Hi dev peeps,
>>
>> So after a long thread I just wanted to summarize the realizations and
>> conclusions so we can set a clear direction for managing the codec
>> extension
>> mechanism. I created a separate clean thread for this here with
>> Guillaume's
>> core recommendation following.
>>
>> For the sake of speed I will define a direction and people can opine:
>>
>> 1). Expose a means in the LDAP API (really SPI) to have codec extensions
>> programmatically registered. I this already exists.
>>
>
> +1
>
> 2). Remove the standalone codec factory implementation that starts up
>> Felix.
>>
>
> Ok, that would be a relief.
>
> 3). Add a simple ClassLoader component to be used in standalone mode to
>> load
>> the plugin classes (from the plugin bundles defaulting to regular jars
>> presumed to be on the classpath). Some configuration information drives
>> how
>> this component discovers what plugin classes to attempt to class load.
>>
> Ok, fine.
>
> 4). Set it up so by default, the LDAP API uses the simple ClassLoader
>> based
>> discover/load/register mechanism. In an OSGi environment this is disabled
>> to
>> enable plugins to self register.
>>
> Ok.
>
> This should serious remove some ugly and dangerous code we've got at this
>> point in the LDAP API.
>>
> Absolutely.
>
> Question : are we sure that teh API will continue to work in Studio, using
> Equinox ? And are we sure we will be able to have felix started in ApacheDS
> ?
>
>
Actually I think it is using equinox at the present moment. If I remember
correctly we're not using the standalone mechanism when in the eclipse
environment. PAM I think found a way to get the bundle loaded under eclipse
OSGi (a.k.a. equinox). Might want to get confirmation from PAM though.
> That raises another related issue for ADS : what if an application is
> already using an OSGi framework and wants to embed ADS ?
>
That's not a problem since basically they can load just the bundles. We can
carry ApacheDS with installers inside Karaf with our standalone
distribution. Those loading ADS into an existing environment need only issue
a load from the bundle repository. I think there's some magic there making
your local maven repository into an OSGi bundle repository. I wonder if
Karaf knows how to pull bundles down too from remote repositories, I bet
Guillaume can answer that :).
Regards,
Alex
Re: [LDAP API] Codec extension mechanism direction (WAS: Re: OSGi
startup and shutdown problems)
Posted by Emmanuel Lecharny <el...@gmail.com>.
On 5/11/11 5:35 PM, Alex Karasulu wrote:
> Hi dev peeps,
>
> So after a long thread I just wanted to summarize the realizations and
> conclusions so we can set a clear direction for managing the codec extension
> mechanism. I created a separate clean thread for this here with Guillaume's
> core recommendation following.
>
> For the sake of speed I will define a direction and people can opine:
>
> 1). Expose a means in the LDAP API (really SPI) to have codec extensions
> programmatically registered. I this already exists.
+1
> 2). Remove the standalone codec factory implementation that starts up Felix.
Ok, that would be a relief.
> 3). Add a simple ClassLoader component to be used in standalone mode to load
> the plugin classes (from the plugin bundles defaulting to regular jars
> presumed to be on the classpath). Some configuration information drives how
> this component discovers what plugin classes to attempt to class load.
Ok, fine.
> 4). Set it up so by default, the LDAP API uses the simple ClassLoader based
> discover/load/register mechanism. In an OSGi environment this is disabled to
> enable plugins to self register.
Ok.
> This should serious remove some ugly and dangerous code we've got at this
> point in the LDAP API.
Absolutely.
Question : are we sure that teh API will continue to work in Studio,
using Equinox ? And are we sure we will be able to have felix started in
ApacheDS ?
That raises another related issue for ADS : what if an application is
already using an OSGi framework and wants to embed ADS ?
Thanks !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com