You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2015/09/15 12:03:24 UTC
[LDAP API] Schema aware connection
Hi,
Radovan has added some way to support schemas that have some slight
differences with the RFC, using the quirkmode flag. If you are using
tehe NetworkLdapConnection, it works this way :
SchemaLoader schemaLoader = new DefaultSchemaLoader(
ldapConnection, true );
ldapConnection.loadSchema( schemaLoader );
Thats fine, except if you have declared your ldapConnection thsi way :
LdapConnection ldapConnection = new
LdapNetworkConnection( server.getServerName(), server.getPort() );
because the LdapConnection interface does not expose the loadSchema(
schemaLoader ) method.
I suggest we expose it.
I also think we should extend the ldapConnection.loadSchema() method, in
order to spare the users from declaring a SchemaLoader on their own.
That would make the code looks like :
ldapConnection.loadSchemaRelaxed();
which would replace
SchemaLoader schemaLoader = new DefaultSchemaLoader(
ldapConnection, true );
ldapConnection.loadSchema( schemaLoader );
Thoughts ?
Re: Fwd: Re: [LDAP API] Schema aware connection
Posted by Radovan Semancik <ra...@evolveum.com>.
On 09/16/2015 01:12 PM, Emmanuel Lécharny wrote:
> 3) This all not very user-friendly. The modifications that Emmanuel
> suggested seem to be a good improvements.
>
> 4) We have quirks mode and relaxed mode. I've realized that quicks
> mode is for parsing and relaxed mode is for schema processing and
> validation. But, do we need to expose both to the API user? Maybe
> relaxed mode should imply quirks mode.
> I was thinking about something slightly different : a constant that can
> take 4 values :
> - STRICT_MODE
> - QUIRK_MODE
> - RELAXED_MODE
> - QUIRK_RELAXED_MODE
>
> we can pass that as an argument. OTOH, yes, relaxed might also imply
> quick_mode (and to be frank, I prefer 'relaxed' to 'quirk_mode', it's
> more explicit).
Yes. I do not like "quirks" either. It can be used inside the API
implementation, but it should not be exposed to API user. I like using
the enumeration instead of boolean. But I would suggest slightly
different names:
- STRICT_MODE
- RELAXED_MODE
- LAX_MODE
> Sounds good to me. Should we still have a loadSchema() method that
> implies a Strict mode ?
I think this is OK. It probably won't work with anything else than
ApacheDS ... but I like it anyway.
> Fully agreed. I *wish* I had time to get this correct the very first
> time. I wasn't expecting OpenLDAP or some other LDAP Server to be so
> permissive... :/ My bad.
Do not blame yourself. I'm meddling with LDAP for more than 15 years
myself. And I have never suspected this magnitude of schema
inconsistencies either. I guess that the real world always has a bunch
of surprises in store, no matter how experienced you are.
--
Radovan Semancik
Software Architect
evolveum.com
Re: Fwd: Re: [LDAP API] Schema aware connection
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 16/09/15 10:46, Radovan Semancik a écrit :
> Hi,
>
> I agree, of course.
>
> Just a bit more information. Currently there is a way how to do this
> ... kind of ...
>
> boolean schemaQuirksMode = ....
> DefaultSchemaLoader schemaLoader = new
> DefaultSchemaLoader(connection, schemaQuirksMode);
> DefaultSchemaManager schemaManager = new
> DefaultSchemaManager(schemaLoader);
> try {
> if (schemaQuirksMode) {
> schemaManager.setRelaxed();
> schemaManager.loadAllEnabledRelaxed();
> } else {
> schemaManager.loadAllEnabled();
> }
> } catch (Exception e) {
> throw new ConnectorIOException(e.getMessage(), e);
> }
> if ( !schemaManager.getErrors().isEmpty() ) {
> if (schemaQuirksMode) {
> LOG.ok("There are {0} schema errors, but we
> are in quirks mode so we are ignoring them",
> defSchemaManager.getErrors().size());
> for (Throwable error:
> schemaManager.getErrors()) {
> LOG.ok("Schema error (ignored): {0}: {1}",
> error.getClass().getName(), error.getMessage());
> }
> } else {
> throw new ConnectorIOException("Errors loading
> schema "+schemaManager.getErrors());
> }
> }
>
>
> This code works even with bad LDAP servers. But .... there are some
> drawbacks:
>
> 1) The schemaManager cannot be attached to the connection. I.e. you
> must not call connection.setSchemaManager(schemaManager). Otherwise
> the LDAP operations will fail on many places as the loaded schema is
> inconsistent. I guess that this is mostly OK. If the schema is not
> consistent then it cannot be expected that it will work normally for
> all the cases. But I believe it should work normally unless a the
> specific inconsistency is encountered. I.e. if there are schema
> inconsistencies in an attribute/syntax that is never used then th API
> should work normally even with inconsistent schema.
The pb you can't do that is that when we try to load a schema from a
remote server, we create a temproraty SM which is not relaxed. This is a
mistake that need to be fixed, thus the proposal to extend teh
LdapConnection class with a loadRelaxed method.
>
> 2) Some errors do not appear in the error list
> (schemaManager.getErrors()). E.g. the OID syntax error that I was
> fixing recently. The error handling in schema processing is a bit
> inconsistent. But I think this can be fixed in an evolutionary fashion.
Probably.
>
>
> 3) This all not very user-friendly. The modifications that Emmanuel
> suggested seem to be a good improvements.
>
> 4) We have quirks mode and relaxed mode. I've realized that quicks
> mode is for parsing and relaxed mode is for schema processing and
> validation. But, do we need to expose both to the API user? Maybe
> relaxed mode should imply quirks mode.
I was thinking about something slightly different : a constant that can
take 4 values :
- STRICT_MODE
- QUIRK_MODE
- RELAXED_MODE
- QUIRK_RELAXED_MODE
we can pass that as an argument. OTOH, yes, relaxed might also imply
quick_mode (and to be frank, I prefer 'relaxed' to 'quirk_mode', it's
more explicit).
So, yes, I think your proposal makes sense.
TBH, I first thought that the quirk_mode was implying the relaxed mode
until I realized it waqs different (too long without playing with that
code ...)
>
> To be more specific, I think we need this (ideally):
>
> connection.loadSchema(mode)
>
> ... and internal implementation will set up quirks/relaxed mode as
> appropriate. The current code is not very clean and I think that we
> will need some "leeway" until we figure out what exactly needs to be
> done here. Therefore we should provide a simple facade to API users
> that will not change even if we change in internal interfaces
> (SchemaManager, SchemaLoader).
Sounds good to me. Should we still have a loadSchema() method that
implies a Strict mode ?
>
> To go one step further I guess we also need to clean up the
> SchemaManager and SchemaLoader a bit. E.g. DefaultSchemaManager has
> setRelaxed(), loadAllEnabled() and loadAllEnabledRelaxed(). One of
> these methods seems to be redundant.
Yep. They are hacks.
> I would expect either to use setRelaxed(true) and then
> loadAllEnabled(), but as far as I can remember that does not work.
> Speaking politically correctly this is a little bit of a mess :-) I
> deserves to be cleaned up.
Fully agreed. I *wish* I had time to get this correct the very first
time. I wasn't expecting OpenLDAP or some other LDAP Server to be so
permissive... :/ My bad.
Expect the worse, get ready for it, and rejoy when it's better than expect !
Re: Fwd: Re: [LDAP API] Schema aware connection
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 16/09/15 10:46, Radovan Semancik a écrit :
> Hi,
>
> I agree, of course.
>
> Just a bit more information. Currently there is a way how to do this
> ... kind of ...
>
> boolean schemaQuirksMode = ....
> DefaultSchemaLoader schemaLoader = new
> DefaultSchemaLoader(connection, schemaQuirksMode);
> DefaultSchemaManager schemaManager = new
> DefaultSchemaManager(schemaLoader);
> try {
> if (schemaQuirksMode) {
> schemaManager.setRelaxed();
> schemaManager.loadAllEnabledRelaxed();
> } else {
> schemaManager.loadAllEnabled();
> }
> } catch (Exception e) {
> throw new ConnectorIOException(e.getMessage(), e);
> }
> if ( !schemaManager.getErrors().isEmpty() ) {
> if (schemaQuirksMode) {
> LOG.ok("There are {0} schema errors, but we
> are in quirks mode so we are ignoring them",
> defSchemaManager.getErrors().size());
> for (Throwable error:
> schemaManager.getErrors()) {
> LOG.ok("Schema error (ignored): {0}: {1}",
> error.getClass().getName(), error.getMessage());
> }
> } else {
> throw new ConnectorIOException("Errors loading
> schema "+schemaManager.getErrors());
> }
> }
I know come with a much simpler code to do teh same thing :
LdapConnection ldapConnection = new LdapNetworkConnection(
serverName, serverPort );
ldapConnection.loadSchemaRelaxed();
And that's it ! The SchemaManager can be taken from the ldapConnection,
it contains whatever the loader was able to load...
Re: Fwd: Re: [LDAP API] Schema aware connection
Posted by Radovan Semancik <ra...@evolveum.com>.
Hi,
I agree, of course.
Just a bit more information. Currently there is a way how to do this ...
kind of ...
boolean schemaQuirksMode = ....
DefaultSchemaLoader schemaLoader = new DefaultSchemaLoader(connection, schemaQuirksMode);
DefaultSchemaManager schemaManager = new DefaultSchemaManager(schemaLoader);
try {
if (schemaQuirksMode) {
schemaManager.setRelaxed();
schemaManager.loadAllEnabledRelaxed();
} else {
schemaManager.loadAllEnabled();
}
} catch (Exception e) {
throw new ConnectorIOException(e.getMessage(), e);
}
if ( !schemaManager.getErrors().isEmpty() ) {
if (schemaQuirksMode) {
LOG.ok("There are {0} schema errors, but we are in quirks mode so we are ignoring them", defSchemaManager.getErrors().size());
for (Throwable error: schemaManager.getErrors()) {
LOG.ok("Schema error (ignored): {0}: {1}", error.getClass().getName(), error.getMessage());
}
} else {
throw new ConnectorIOException("Errors loading schema "+schemaManager.getErrors());
}
}
This code works even with bad LDAP servers. But .... there are some
drawbacks:
1) The schemaManager cannot be attached to the connection. I.e. you must
not call connection.setSchemaManager(schemaManager). Otherwise the LDAP
operations will fail on many places as the loaded schema is
inconsistent. I guess that this is mostly OK. If the schema is not
consistent then it cannot be expected that it will work normally for all
the cases. But I believe it should work normally unless a the specific
inconsistency is encountered. I.e. if there are schema inconsistencies
in an attribute/syntax that is never used then th API should work
normally even with inconsistent schema.
2) Some errors do not appear in the error list
(schemaManager.getErrors()). E.g. the OID syntax error that I was fixing
recently. The error handling in schema processing is a bit inconsistent.
But I think this can be fixed in an evolutionary fashion.
3) This all not very user-friendly. The modifications that Emmanuel
suggested seem to be a good improvements.
4) We have quirks mode and relaxed mode. I've realized that quicks mode
is for parsing and relaxed mode is for schema processing and validation.
But, do we need to expose both to the API user? Maybe relaxed mode
should imply quirks mode.
To be more specific, I think we need this (ideally):
connection.loadSchema(mode)
... and internal implementation will set up quirks/relaxed mode as
appropriate. The current code is not very clean and I think that we will
need some "leeway" until we figure out what exactly needs to be done
here. Therefore we should provide a simple facade to API users that will
not change even if we change in internal interfaces (SchemaManager,
SchemaLoader).
To go one step further I guess we also need to clean up the
SchemaManager and SchemaLoader a bit. E.g. DefaultSchemaManager has
setRelaxed(), loadAllEnabled() and loadAllEnabledRelaxed(). One of these
methods seems to be redundant. I would expect either to use
setRelaxed(true) and then loadAllEnabled(), but as far as I can remember
that does not work. Speaking politically correctly this is a little bit
of a mess :-) I deserves to be cleaned up.
--
Radovan Semancik
Software Architect
evolveum.com
On 09/15/2015 02:46 PM, Emmanuel Lécharny wrote:
> Le 15/09/15 14:39, Emmanuel Lécharny a écrit :
>> Forwarding this mail who deserves to be sent to API@directory.a.o...
>>
>>
>>
>> Le 15/09/15 12:03, Emmanuel Lécharny a écrit :
>>> Hi,
>>>
>>> Radovan has added some way to support schemas that have some slight
>>> differences with the RFC, using the quirkmode flag. If you are using
>>> tehe NetworkLdapConnection, it works this way :
>>>
>>> SchemaLoader schemaLoader = new DefaultSchemaLoader(
>>> ldapConnection, true );
>>> ldapConnection.loadSchema( schemaLoader );
>>>
>>> Thats fine, except if you have declared your ldapConnection thsi way :
>>>
>>> LdapConnection ldapConnection = new
>>> LdapNetworkConnection( server.getServerName(), server.getPort() );
>>>
>>> because the LdapConnection interface does not expose the loadSchema(
>>> schemaLoader ) method.
>>>
>>> I suggest we expose it.
>>>
>>> I also think we should extend the ldapConnection.loadSchema() method, in
>>> order to spare the users from declaring a SchemaLoader on their own.
>>> That would make the code looks like :
>>>
>>>
>>> ldapConnection.loadSchemaRelaxed();
>>>
>>> which would replace
>>>
>>> SchemaLoader schemaLoader = new DefaultSchemaLoader(
>>> ldapConnection, true );
>>> ldapConnection.loadSchema( schemaLoader );
>> It's a bit more complex that just using the quirkmode (which just relax
>> the schema syntax). Sometime, we want to accept some inconsistencies in
>> the schema itself. The relaxed mode is there to accept such schema, but
>> it's not exposed.
>>
>> Typically, the loadSchema() methods starts with :
>>
>>
>> public void loadSchema( SchemaLoader loader ) throws LdapException
>> {
>> try
>> {
>> SchemaManager tmp = new DefaultSchemaManager( loader );
>>
>> tmp.loadAllEnabled();
>>
>> if ( !tmp.getErrors().isEmpty() )
>> {
>> String msg = "there are errors while loading the schema";
>> LOG.error( msg + " {}", tmp.getErrors() );
>> throw new LdapException( msg );
>> }
>> ...
>>
>> which creates a SchemaManager in strict mode. Obviously, the
>> tmp.loadAllEnabled() will fail. We should propagate the 'relaxed' flag
>> down to the SchemaManager.
>>
>>
>>
>>
> I will give a clear example :
>
> when you connect to an OpenLDAP server, and fetch the schema, you will
> get a failure because some MatchingRule are using a Syntax which is not
> loaded. Typically, a standard OpenLDAP server with core, cosine and
> inetorgperson schemas will have 248 ATs, 34 OCs, and 72 Syntaxes (from
> cn=schema,ou=config). The same server will expose only 32 Syntaxes (but
> 248 ATs and 63 OCs. Go fish...)
>
> The 40 missing Syntaxes are syntaxes which have no associated
> validators. That does not mean those syntax can't be used, though. The
> pb is that we don't see them in the SchemaLoader, and we default by
> rejecting the load.
>
> At this point, one solution would be to signal those errors, but don't
> stop and use default elements to replace the missing ones (ie, defining
> a Syntax with no specific check).
>
> I see no other ways to deal with that, becauee as Radovan says, it won't
> be fixed any time soon...
>
> thoughts ?
>
Re: Fwd: Re: [LDAP API] Schema aware connection
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 15/09/15 14:39, Emmanuel Lécharny a écrit :
> Forwarding this mail who deserves to be sent to API@directory.a.o...
>
>
>
> Le 15/09/15 12:03, Emmanuel Lécharny a écrit :
>> Hi,
>>
>> Radovan has added some way to support schemas that have some slight
>> differences with the RFC, using the quirkmode flag. If you are using
>> tehe NetworkLdapConnection, it works this way :
>>
>> SchemaLoader schemaLoader = new DefaultSchemaLoader(
>> ldapConnection, true );
>> ldapConnection.loadSchema( schemaLoader );
>>
>> Thats fine, except if you have declared your ldapConnection thsi way :
>>
>> LdapConnection ldapConnection = new
>> LdapNetworkConnection( server.getServerName(), server.getPort() );
>>
>> because the LdapConnection interface does not expose the loadSchema(
>> schemaLoader ) method.
>>
>> I suggest we expose it.
>>
>> I also think we should extend the ldapConnection.loadSchema() method, in
>> order to spare the users from declaring a SchemaLoader on their own.
>> That would make the code looks like :
>>
>>
>> ldapConnection.loadSchemaRelaxed();
>>
>> which would replace
>>
>> SchemaLoader schemaLoader = new DefaultSchemaLoader(
>> ldapConnection, true );
>> ldapConnection.loadSchema( schemaLoader );
> It's a bit more complex that just using the quirkmode (which just relax
> the schema syntax). Sometime, we want to accept some inconsistencies in
> the schema itself. The relaxed mode is there to accept such schema, but
> it's not exposed.
>
> Typically, the loadSchema() methods starts with :
>
>
> public void loadSchema( SchemaLoader loader ) throws LdapException
> {
> try
> {
> SchemaManager tmp = new DefaultSchemaManager( loader );
>
> tmp.loadAllEnabled();
>
> if ( !tmp.getErrors().isEmpty() )
> {
> String msg = "there are errors while loading the schema";
> LOG.error( msg + " {}", tmp.getErrors() );
> throw new LdapException( msg );
> }
> ...
>
> which creates a SchemaManager in strict mode. Obviously, the
> tmp.loadAllEnabled() will fail. We should propagate the 'relaxed' flag
> down to the SchemaManager.
>
>
>
>
I will give a clear example :
when you connect to an OpenLDAP server, and fetch the schema, you will
get a failure because some MatchingRule are using a Syntax which is not
loaded. Typically, a standard OpenLDAP server with core, cosine and
inetorgperson schemas will have 248 ATs, 34 OCs, and 72 Syntaxes (from
cn=schema,ou=config). The same server will expose only 32 Syntaxes (but
248 ATs and 63 OCs. Go fish...)
The 40 missing Syntaxes are syntaxes which have no associated
validators. That does not mean those syntax can't be used, though. The
pb is that we don't see them in the SchemaLoader, and we default by
rejecting the load.
At this point, one solution would be to signal those errors, but don't
stop and use default elements to replace the missing ones (ie, defining
a Syntax with no specific check).
I see no other ways to deal with that, becauee as Radovan says, it won't
be fixed any time soon...
thoughts ?
Fwd: Re: [LDAP API] Schema aware connection
Posted by Emmanuel Lécharny <el...@gmail.com>.
Forwarding this mail who deserves to be sent to API@directory.a.o...
Le 15/09/15 12:03, Emmanuel Lécharny a écrit :
> Hi,
>
> Radovan has added some way to support schemas that have some slight
> differences with the RFC, using the quirkmode flag. If you are using
> tehe NetworkLdapConnection, it works this way :
>
> SchemaLoader schemaLoader = new DefaultSchemaLoader(
> ldapConnection, true );
> ldapConnection.loadSchema( schemaLoader );
>
> Thats fine, except if you have declared your ldapConnection thsi way :
>
> LdapConnection ldapConnection = new
> LdapNetworkConnection( server.getServerName(), server.getPort() );
>
> because the LdapConnection interface does not expose the loadSchema(
> schemaLoader ) method.
>
> I suggest we expose it.
>
> I also think we should extend the ldapConnection.loadSchema() method, in
> order to spare the users from declaring a SchemaLoader on their own.
> That would make the code looks like :
>
>
> ldapConnection.loadSchemaRelaxed();
>
> which would replace
>
> SchemaLoader schemaLoader = new DefaultSchemaLoader(
> ldapConnection, true );
> ldapConnection.loadSchema( schemaLoader );
It's a bit more complex that just using the quirkmode (which just relax
the schema syntax). Sometime, we want to accept some inconsistencies in
the schema itself. The relaxed mode is there to accept such schema, but
it's not exposed.
Typically, the loadSchema() methods starts with :
public void loadSchema( SchemaLoader loader ) throws LdapException
{
try
{
SchemaManager tmp = new DefaultSchemaManager( loader );
tmp.loadAllEnabled();
if ( !tmp.getErrors().isEmpty() )
{
String msg = "there are errors while loading the schema";
LOG.error( msg + " {}", tmp.getErrors() );
throw new LdapException( msg );
}
...
which creates a SchemaManager in strict mode. Obviously, the
tmp.loadAllEnabled() will fail. We should propagate the 'relaxed' flag
down to the SchemaManager.
Re: [LDAP API] Schema aware connection
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 15/09/15 12:03, Emmanuel Lécharny a écrit :
> Hi,
>
> Radovan has added some way to support schemas that have some slight
> differences with the RFC, using the quirkmode flag. If you are using
> tehe NetworkLdapConnection, it works this way :
>
> SchemaLoader schemaLoader = new DefaultSchemaLoader(
> ldapConnection, true );
> ldapConnection.loadSchema( schemaLoader );
>
> Thats fine, except if you have declared your ldapConnection thsi way :
>
> LdapConnection ldapConnection = new
> LdapNetworkConnection( server.getServerName(), server.getPort() );
>
> because the LdapConnection interface does not expose the loadSchema(
> schemaLoader ) method.
>
> I suggest we expose it.
>
> I also think we should extend the ldapConnection.loadSchema() method, in
> order to spare the users from declaring a SchemaLoader on their own.
> That would make the code looks like :
>
>
> ldapConnection.loadSchemaRelaxed();
>
> which would replace
>
> SchemaLoader schemaLoader = new DefaultSchemaLoader(
> ldapConnection, true );
> ldapConnection.loadSchema( schemaLoader );
It's a bit more complex that just using the quirkmode (which just relax
the schema syntax). Sometime, we want to accept some inconsistencies in
the schema itself. The relaxed mode is there to accept such schema, but
it's not exposed.
Typically, the loadSchema() methods starts with :
public void loadSchema( SchemaLoader loader ) throws LdapException
{
try
{
SchemaManager tmp = new DefaultSchemaManager( loader );
tmp.loadAllEnabled();
if ( !tmp.getErrors().isEmpty() )
{
String msg = "there are errors while loading the schema";
LOG.error( msg + " {}", tmp.getErrors() );
throw new LdapException( msg );
}
...
which creates a SchemaManager in strict mode. Obviously, the
tmp.loadAllEnabled() will fail. We should propagate the 'relaxed' flag
down to the SchemaManager.