You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2016/03/05 09:26:29 UTC
Work in progress...
Hi guys,
a bit of a heads up on what I'm on atm. being pretty busy aside, and I
guess its the same for all of you, things are moving forward slowly.
- support for RDN like 'cn=A + cn=B'. This is legit, but it's currently
rejected. We have to change the way we store AVA inside the RDN to use a
sorted list of AVA based on the AttributeType. Should take a day.
- handling of escaped chars in values and DN. This one is a complicated
beast, it requires a complete review on the way we process DN parsing
and schema support. I don't expect that to be simple nor fast...
- I have added two AttributeTypes in the ApacheDS schema : nbChildren
and nbSubordinates. They will contain the number of children and
subordinates an entry has (we have this information in ApacheDS
associated with each RDN), they are operational attributes that will be
added on the fly. I still have to add them in the AbstractBtreePartition
(in the fetch method, which calls the buildEntryDn metho, which has
access to this information) On day of work.
- The commons-io dependency has been removed from teh LDAP API and
ApacheDS, we still have a dependency on it in Studio, but I'm not sure
it worths it to get it removed too.
- I have checked if commons-pool could not be upgraded from 1.6 to
2.4.2. There are some API changes, and it would take a bit of time to
get through it.
- I'm working on the Opendlap ACL parser in Studio. This is a work in
progress which will take quite some time. Atm, I'm rewritting the
grammar, modifying the model to be able to accept all the possible ACL
syntax. The next step will be to review and complete the GUI for ACLs
- I tried, with Shawn, to cut a release of fortress core, realm and
enmasse. The release is ok for core and realm, but we rollbacked it as
Shawn found some problems with dependencies. This was just a matter to
kick in tyres anyway, and we now have a clean pom.xml for realm (the
work is not completed for enmasse). We will restart next week.
I have had no time recently to work on Mavibot, which really deserves
some love. The problem is that it's quite impossible to focus on such
thing one hour there, one hour here. It requires a full week of work
that I don't currently have :/
So things are moving forward, slowly, but still...
Thanks !
Re: Work in progress...
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 03/05/2016 10:14 AM, Emmanuel Lécharny wrote:
>>> - support for RDN like 'cn=A + cn=B'. This is legit, but it's currently
>>> rejected. We have to change the way we store AVA inside the RDN to use a
>>> sorted list of AVA based on the AttributeType. Should take a day.
>> Seems you already committed two tests which fail, if you don't mind I'll
>> @Ignore them for the time being.
> Sure, and sorry for that.
Oh, I was wrong, the two failing tests are old ones, they checked that
same AT is not possible and expected an exception. I changed the
assertion: http://svn.apache.org/viewvc?rev=1733693&view=rev
Re: Work in progress...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 05/03/16 09:47, Stefan Seelmann a écrit :
> Thanks Emmanuel for the update!
>
> On 03/05/2016 09:26 AM, Emmanuel Lécharny wrote:
>> a bit of a heads up on what I'm on atm. being pretty busy aside, and I
>> guess its the same for all of you, things are moving forward slowly.
> I just started a new job last week which takes my time...
Day job is killing us ;-) I'm waiting for retirment ! 10 more years...
>
>> - support for RDN like 'cn=A + cn=B'. This is legit, but it's currently
>> rejected. We have to change the way we store AVA inside the RDN to use a
>> sorted list of AVA based on the AttributeType. Should take a day.
> Seems you already committed two tests which fail, if you don't mind I'll
> @Ignore them for the time being.
Sure, and sorry for that.
>
>> - The commons-io dependency has been removed from teh LDAP API and
>> ApacheDS, we still have a dependency on it in Studio, but I'm not sure
>> it worths it to get it removed too.
> I'll have a look at it this weekend. But Studio is already a monster
> with 472 jars, I think it does't matter if there is one more or less ;)
Totally agree.
>
> Other than that I'll focus on Studio, updating to Mars.2, and fix some
> remaining bugs.
Thanks !
Re: Work in progress...
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
Thanks Emmanuel for the update!
On 03/05/2016 09:26 AM, Emmanuel Lécharny wrote:
> a bit of a heads up on what I'm on atm. being pretty busy aside, and I
> guess its the same for all of you, things are moving forward slowly.
I just started a new job last week which takes my time...
> - support for RDN like 'cn=A + cn=B'. This is legit, but it's currently
> rejected. We have to change the way we store AVA inside the RDN to use a
> sorted list of AVA based on the AttributeType. Should take a day.
Seems you already committed two tests which fail, if you don't mind I'll
@Ignore them for the time being.
> - The commons-io dependency has been removed from teh LDAP API and
> ApacheDS, we still have a dependency on it in Studio, but I'm not sure
> it worths it to get it removed too.
I'll have a look at it this weekend. But Studio is already a monster
with 472 jars, I think it does't matter if there is one more or less ;)
Other than that I'll focus on Studio, updating to Mars.2, and fix some
remaining bugs.
Kind Regards,
Stefan
Re: nbSubordnates, nbChildren, was: Work in progress...
Posted by Ludovic Poitou <lu...@gmail.com>.
For someone which follows the discussion with distance, I find it quite confusing.
Netscape Directory Server, Sun Directory Server, Red Hat Directory Server, OpenDS, OpenDJ, OUD and most likely UnboundID directory are all providing the standard hasSubordinates, and the numSubordinates attribute.
So I find it confusing that you are now talking about defining new attributes nbSubordinates, nbChildren… that will possibly make things more complex for the interoperability of client applications.
Just my 2 cents of “let’s not try to reinvent the wheel each time”.
Ludo
--
Ludovic Poitou
http://ludopoitou.com
From: Emmanuel Lécharny <el...@gmail.com>
Reply: Apache Directory Developers List <de...@directory.apache.org>
Date: 8 March 2016 at 00:26:11
To: dev@directory.apache.org <de...@directory.apache.org>
Subject: Re: nbSubordnates, nbChildren, was: Work in progress...
Le 07/03/16 21:29, Howard Chu a écrit :
> Emmanuel Lécharny wrote:
>> Le 07/03/16 20:45, Stefan Seelmann a écrit :
>>> On 03/06/2016 02:49 AM, Emmanuel Lécharny wrote:
>>>> I think it will be very useful for studio, when running against
>>>> ApacheDS, as once can now expose the number of *real* childrens and
>>>> subrdinates without having to fetch them from teh server (more
>>>> important, we won't be limited by the SizeLimit that can be set -
>>>> typically 1000).
>>>>
>>>> I wish we can have the same feature added to OpenLDAP and the other
>>>> servers...
>>> Well some servers support something similar: Novell has a
>>> "subordinateCount" AT, SunDS/OpenDS/OpenDJ have a "numSubordinates" and
>>> "hasSubordinates" AT. numSubordinates is defined in an expired draft
>>> [1]
>>> and has OID 1.3.6.1.4.1.453.16.2.103 assigned. hasSubordinates is even
>>> defined in X.501 and has official OID 2.5.18.9 assigned.
>> I took the idea from SunDS, leveraging the fact that we have the
>> information available.
>>>
>>> In Studio we use those attributes but only to indicate if an entry
>>> contains children or not and ot make the item in the tree expandable
>>> or not.
>>
>> Good
>>>
>>> Maybe we can add at lease hasSubordinates additionally, then other LDAP
>>> clients that already support it can leverage it.
>>
>> I can add the hasSubordinates AT easily.
>>
>>>
>>> I think numSubordinates and subordinateCount only mean the number of
>>> direct children, whereas our nbSubordinates means all.
>>
>> Because we have the nbChildren that gives the direct number of children.
>>
>>>
>>> I think in Studio we can add support for those, we can extend the
>>> number
>>> shown for an entry like that:
>>>
>>> ou=users (1000/23456/456789)
>>>
>>> which means 1000 entries fetched, out of 23456 direct children out of
>>> 456789 total subordinates.
>>
>> Sounds like a good idea, although it's a bit heavy. How about having
>> just the number of children as a default, and a hover to give the number
>> of fetched and subordinates ?
>>
>> Most of the time, people won't fetch the entries.
>
> OpenLDAP has supported hasSubordinates for quite a long time. The
> current back-mdb has the necessary info to provide numSubordinates but
> we didn't see a need to implement it.
Same here, but in Studio, the need is obvious : when you expose a set of
entries, you have no clue about those having children, and no clue about
how many of them (actually, as Stefan says, it depends on the diretcory
server Studio is talking to). So the only way is actually to read the
entry's children to have this count.
The hasSubordinates is clearly good to have, as it allows Studio to at
last inform the user that there are some entries below.
ApacheDS had none of hasSubordinates/nbChildren/nbSubordinates exposed,
leaving us in a position where studio was offering a better user
experience with any other LDAP server :/
Now, assuming you don't have the number of subordinates (or children)
available, you won't get it by fetching teh entry's children because you
may reach the server's limit. Using a PagedSearch to browse all the
children would not only be a tehcnical non-sense, but will also only
work if teh remote server supports such a control. Bottom line, haing at
least the number of children returned within each entry offers us some
insight.
There is also one frequent need from LDAP users : "give us the number of
children, so that we can manage the presentation properly".
> back-bdb/hdb don't have the info for numSubordinates and since they
> are deprecated, we would not be adding anything to them any more.
>
> (Actually, I don't know which you're referring to. They could all
> provide the number of immediate children; the total number of
> subordinates in the subtree is only directly available in back-mdb at
> the moment.)
I do think that the number of direct children is really the important
information. The number of subordinates (ie children and all the
descendant) is quite cosmetic. It's more about being able to knwo how
much entries your database has when fetching the context entry. We have
this information in ApacheDS, it was easy to provide it at the same time
we provided the nbChildren.
Last, not least : the cost of providing this information is not null. We
have to do a fetch on the Rdn index for every entry, which adds an extra
cost of 10% the time it takes to grab the same entry without this
attribute. We could have cached this information, or grab it while
constructing the DN, but that would have meant a way more complex
refactoring and a complicated entry/DN cache management... Not worths
the effort, IMHO.
Re: nbSubordnates, nbChildren, was: Work in progress...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 07/03/16 21:29, Howard Chu a écrit :
> Emmanuel Lécharny wrote:
>> Le 07/03/16 20:45, Stefan Seelmann a écrit :
>>> On 03/06/2016 02:49 AM, Emmanuel Lécharny wrote:
>>>> I think it will be very useful for studio, when running against
>>>> ApacheDS, as once can now expose the number of *real* childrens and
>>>> subrdinates without having to fetch them from teh server (more
>>>> important, we won't be limited by the SizeLimit that can be set -
>>>> typically 1000).
>>>>
>>>> I wish we can have the same feature added to OpenLDAP and the other
>>>> servers...
>>> Well some servers support something similar: Novell has a
>>> "subordinateCount" AT, SunDS/OpenDS/OpenDJ have a "numSubordinates" and
>>> "hasSubordinates" AT. numSubordinates is defined in an expired draft
>>> [1]
>>> and has OID 1.3.6.1.4.1.453.16.2.103 assigned. hasSubordinates is even
>>> defined in X.501 and has official OID 2.5.18.9 assigned.
>> I took the idea from SunDS, leveraging the fact that we have the
>> information available.
>>>
>>> In Studio we use those attributes but only to indicate if an entry
>>> contains children or not and ot make the item in the tree expandable
>>> or not.
>>
>> Good
>>>
>>> Maybe we can add at lease hasSubordinates additionally, then other LDAP
>>> clients that already support it can leverage it.
>>
>> I can add the hasSubordinates AT easily.
>>
>>>
>>> I think numSubordinates and subordinateCount only mean the number of
>>> direct children, whereas our nbSubordinates means all.
>>
>> Because we have the nbChildren that gives the direct number of children.
>>
>>>
>>> I think in Studio we can add support for those, we can extend the
>>> number
>>> shown for an entry like that:
>>>
>>> ou=users (1000/23456/456789)
>>>
>>> which means 1000 entries fetched, out of 23456 direct children out of
>>> 456789 total subordinates.
>>
>> Sounds like a good idea, although it's a bit heavy. How about having
>> just the number of children as a default, and a hover to give the number
>> of fetched and subordinates ?
>>
>> Most of the time, people won't fetch the entries.
>
> OpenLDAP has supported hasSubordinates for quite a long time. The
> current back-mdb has the necessary info to provide numSubordinates but
> we didn't see a need to implement it.
Same here, but in Studio, the need is obvious : when you expose a set of
entries, you have no clue about those having children, and no clue about
how many of them (actually, as Stefan says, it depends on the diretcory
server Studio is talking to). So the only way is actually to read the
entry's children to have this count.
The hasSubordinates is clearly good to have, as it allows Studio to at
last inform the user that there are some entries below.
ApacheDS had none of hasSubordinates/nbChildren/nbSubordinates exposed,
leaving us in a position where studio was offering a better user
experience with any other LDAP server :/
Now, assuming you don't have the number of subordinates (or children)
available, you won't get it by fetching teh entry's children because you
may reach the server's limit. Using a PagedSearch to browse all the
children would not only be a tehcnical non-sense, but will also only
work if teh remote server supports such a control. Bottom line, haing at
least the number of children returned within each entry offers us some
insight.
There is also one frequent need from LDAP users : "give us the number of
children, so that we can manage the presentation properly".
> back-bdb/hdb don't have the info for numSubordinates and since they
> are deprecated, we would not be adding anything to them any more.
>
> (Actually, I don't know which you're referring to. They could all
> provide the number of immediate children; the total number of
> subordinates in the subtree is only directly available in back-mdb at
> the moment.)
I do think that the number of direct children is really the important
information. The number of subordinates (ie children and all the
descendant) is quite cosmetic. It's more about being able to knwo how
much entries your database has when fetching the context entry. We have
this information in ApacheDS, it was easy to provide it at the same time
we provided the nbChildren.
Last, not least : the cost of providing this information is not null. We
have to do a fetch on the Rdn index for every entry, which adds an extra
cost of 10% the time it takes to grab the same entry without this
attribute. We could have cached this information, or grab it while
constructing the DN, but that would have meant a way more complex
refactoring and a complicated entry/DN cache management... Not worths
the effort, IMHO.
Re: nbSubordnates, nbChildren, was: Work in progress...
Posted by Howard Chu <hy...@symas.com>.
Emmanuel Lécharny wrote:
> Le 07/03/16 20:45, Stefan Seelmann a écrit :
>> On 03/06/2016 02:49 AM, Emmanuel Lécharny wrote:
>>> I think it will be very useful for studio, when running against
>>> ApacheDS, as once can now expose the number of *real* childrens and
>>> subrdinates without having to fetch them from teh server (more
>>> important, we won't be limited by the SizeLimit that can be set -
>>> typically 1000).
>>>
>>> I wish we can have the same feature added to OpenLDAP and the other
>>> servers...
>> Well some servers support something similar: Novell has a
>> "subordinateCount" AT, SunDS/OpenDS/OpenDJ have a "numSubordinates" and
>> "hasSubordinates" AT. numSubordinates is defined in an expired draft [1]
>> and has OID 1.3.6.1.4.1.453.16.2.103 assigned. hasSubordinates is even
>> defined in X.501 and has official OID 2.5.18.9 assigned.
> I took the idea from SunDS, leveraging the fact that we have the
> information available.
>>
>> In Studio we use those attributes but only to indicate if an entry
>> contains children or not and ot make the item in the tree expandable or not.
>
> Good
>>
>> Maybe we can add at lease hasSubordinates additionally, then other LDAP
>> clients that already support it can leverage it.
>
> I can add the hasSubordinates AT easily.
>
>>
>> I think numSubordinates and subordinateCount only mean the number of
>> direct children, whereas our nbSubordinates means all.
>
> Because we have the nbChildren that gives the direct number of children.
>
>>
>> I think in Studio we can add support for those, we can extend the number
>> shown for an entry like that:
>>
>> ou=users (1000/23456/456789)
>>
>> which means 1000 entries fetched, out of 23456 direct children out of
>> 456789 total subordinates.
>
> Sounds like a good idea, although it's a bit heavy. How about having
> just the number of children as a default, and a hover to give the number
> of fetched and subordinates ?
>
> Most of the time, people won't fetch the entries.
OpenLDAP has supported hasSubordinates for quite a long time. The current
back-mdb has the necessary info to provide numSubordinates but we didn't see a
need to implement it. back-bdb/hdb don't have the info for numSubordinates and
since they are deprecated, we would not be adding anything to them any more.
(Actually, I don't know which you're referring to. They could all provide the
number of immediate children; the total number of subordinates in the subtree
is only directly available in back-mdb at the moment.)
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Re: nbSubordnates, nbChildren, was: Work in progress...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 07/03/16 20:45, Stefan Seelmann a écrit :
> On 03/06/2016 02:49 AM, Emmanuel Lécharny wrote:
>> I think it will be very useful for studio, when running against
>> ApacheDS, as once can now expose the number of *real* childrens and
>> subrdinates without having to fetch them from teh server (more
>> important, we won't be limited by the SizeLimit that can be set -
>> typically 1000).
>>
>> I wish we can have the same feature added to OpenLDAP and the other
>> servers...
> Well some servers support something similar: Novell has a
> "subordinateCount" AT, SunDS/OpenDS/OpenDJ have a "numSubordinates" and
> "hasSubordinates" AT. numSubordinates is defined in an expired draft [1]
> and has OID 1.3.6.1.4.1.453.16.2.103 assigned. hasSubordinates is even
> defined in X.501 and has official OID 2.5.18.9 assigned.
I took the idea from SunDS, leveraging the fact that we have the
information available.
>
> In Studio we use those attributes but only to indicate if an entry
> contains children or not and ot make the item in the tree expandable or not.
Good
>
> Maybe we can add at lease hasSubordinates additionally, then other LDAP
> clients that already support it can leverage it.
I can add the hasSubordinates AT easily.
>
> I think numSubordinates and subordinateCount only mean the number of
> direct children, whereas our nbSubordinates means all.
Because we have the nbChildren that gives the direct number of children.
>
> I think in Studio we can add support for those, we can extend the number
> shown for an entry like that:
>
> ou=users (1000/23456/456789)
>
> which means 1000 entries fetched, out of 23456 direct children out of
> 456789 total subordinates.
Sounds like a good idea, although it's a bit heavy. How about having
just the number of children as a default, and a hover to give the number
of fetched and subordinates ?
Most of the time, people won't fetch the entries.
nbSubordnates, nbChildren, was: Work in progress...
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 03/06/2016 02:49 AM, Emmanuel Lécharny wrote:
> I think it will be very useful for studio, when running against
> ApacheDS, as once can now expose the number of *real* childrens and
> subrdinates without having to fetch them from teh server (more
> important, we won't be limited by the SizeLimit that can be set -
> typically 1000).
>
> I wish we can have the same feature added to OpenLDAP and the other
> servers...
Well some servers support something similar: Novell has a
"subordinateCount" AT, SunDS/OpenDS/OpenDJ have a "numSubordinates" and
"hasSubordinates" AT. numSubordinates is defined in an expired draft [1]
and has OID 1.3.6.1.4.1.453.16.2.103 assigned. hasSubordinates is even
defined in X.501 and has official OID 2.5.18.9 assigned.
In Studio we use those attributes but only to indicate if an entry
contains children or not and ot make the item in the tree expandable or not.
Maybe we can add at lease hasSubordinates additionally, then other LDAP
clients that already support it can leverage it.
I think numSubordinates and subordinateCount only mean the number of
direct children, whereas our nbSubordinates means all.
I think in Studio we can add support for those, we can extend the number
shown for an entry like that:
ou=users (1000/23456/456789)
which means 1000 entries fetched, out of 23456 direct children out of
456789 total subordinates.
Kind Regards,
Stefan
[1] https://tools.ietf.org/html/draft-boreham-numsubordinates-01
Re: Work in progress...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 05/03/16 09:26, Emmanuel Lécharny a écrit :
> Hi guys,
>
> a bit of a heads up on what I'm on atm. being pretty busy aside, and I
> guess its the same for all of you, things are moving forward slowly.
>
> - I have added two AttributeTypes in the ApacheDS schema : nbChildren
> and nbSubordinates. They will contain the number of children and
> subordinates an entry has (we have this information in ApacheDS
> associated with each RDN), they are operational attributes that will be
> added on the fly. I still have to add them in the AbstractBtreePartition
> (in the fetch method, which calls the buildEntryDn metho, which has
> access to this information) On day of work.
So now, I can get back those values for a Lookup operation. Here is a
sample test that works well :
@Test
public void testLookupSubordinates() throws LdapException
{
Entry entry = connection.lookup( "cn=test,ou=system", "*", "+" );
assertNotNull( entry );
// We should have 11 attributes
assertEquals( 11, entry.size() );
assertTrue( entry.containsAttribute( "nbChildren",
"nbSubordinates" ) );
assertEquals( 0L, Long.parseLong( entry.get( "nbChildren"
).getString() ) );
assertEquals( 0L, Long.parseLong( entry.get( "nbSubordinates"
).getString() ) );
// Now lookup for the "ou=system"
entry = connection.lookup( "ou=system", "*", "+" );
assertNotNull( entry );
// We should have 11 attributes
assertEquals( 11, entry.size() );
assertTrue( entry.containsAttribute( "nbChildren",
"nbSubordinates" ) );
// we will have 6 children :
// - ou=configuration
// - ou=consumer
// - ou = groups
// - ou=users
// - ou=prefNodeNames
// - uid=admin
// and 10 subordinates, as we have 3 children under
ou=configuration and one under ou=groups
assertEquals( 6L, Long.parseLong( entry.get( "nbChildren"
).getString() ) );
assertEquals( 10L, Long.parseLong( entry.get( "nbSubordinates"
).getString() ) );
}
I still have to add the Search filter, which should not be complicated
(but it's already almost 3am ;-)
This was a funny feature to add that leverages something we had in the
backend for years. It works for every partition extending teh
AbtsractBtreePartition (so for the moment, it doesn't return anything
for the schema partition).
I think it will be very useful for studio, when running against
ApacheDS, as once can now expose the number of *real* childrens and
subrdinates without having to fetch them from teh server (more
important, we won't be limited by the SizeLimit that can be set -
typically 1000).
I wish we can have the same feature added to OpenLDAP and the other
servers...
More to come !