You are viewing a plain text version of this content. The canonical link for it is here.
Posted to api@directory.apache.org by Emmanuel Lécharny <el...@symas.com> on 2015/10/08 09:52:48 UTC
DN parsing troubles...
Hi guys,
I'm reviewing the DN Parser, as I think we are overdoing. So far, I have
something that works, except a couple of tests that fail. One otf them is :
public void testRdnWithEscapedComa() throws LdapException
{
...
Rdn rdn = new Rdn( "a=\"b\\,c\"" );
assertEquals( "a=\"b\\,c\"", rdn.getName() );
assertEquals( "a=b\\,c", rdn.getNormName() ); // WRONG !!!
Here, I think the test is wrong. The initial value is surrounded by
quotes, which means that we don't need to escape special chars like ','
into it. Actually, the following value is accepted :
Rdn rdn1 = new Rdn( "a=b\\,c" );
and the RDN value is 'b,c' (the ',' is escaped to avoid a confusion with
the next RDN in a DN)
but when we use quotes, this is equivalent to :
Rdn rdn2 = new Rdn( "a=\"b,c\"" );
Here rdn1 <==> rdn2 (or it should be equivalent.
On the other hand, this rdn :
Rdn rdn3 = new Rdn( "a=\"b\\,c\"" );
should be equivalent to :
Rdn rdn4 = new Rdn( "a=b\\\,c" );
where we must keep an escaped escape and an escaped coma.
Thoughts ?
Re: DN parsing troubles...
Posted by Emmanuel Lécharny <el...@gmail.com>.
As a side effect on the changes I'm applying, I detect tests that should
have failed :
@Test
public void testSimpleRdnNoValueSerialization() throws
LdapException, IOException, ClassNotFoundException
{
Rdn rdn = new Rdn( schemaManager, " CN =" );
...
should simply fail immediately, as a cn should never accept an empty
value. Niow, I get this exception (plain normal) :
org.apache.directory.api.ldap.model.exception.LdapInvalidDnException:
ERR_04447_CANNOT_NORMALIZE_VALUE Cannot normalize the wrapped value
ERR_04473_NOT_VALID_VALUE Not a valid value '' for the AttributeType
'attributetype ( 2.5.4.3 NAME ( 'cn' 'commonName' )
DESC 'RFC2256: common name(s) for which the entity is known by'
SUP name
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
USAGE userApplications )'
at
org.apache.directory.api.ldap.model.name.Dn.atavOidToName(Dn.java:1117)
at
org.apache.directory.api.ldap.model.name.Dn.rdnOidToName(Dn.java:1154)
at org.apache.directory.api.ldap.model.name.Rdn.apply(Rdn.java:481)
at org.apache.directory.api.ldap.model.name.Rdn.<init>(Rdn.java:243)
at
org.apache.directory.api.ldap.model.name.SchemaAwareRdnTest.testSimpleRdnNoValueSerialization(SchemaAwareRdnTest.java:1066)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by:
org.apache.directory.api.ldap.model.exception.LdapInvalidAttributeValueException:
ERR_04447_CANNOT_NORMALIZE_VALUE Cannot normalize the wrapped value
ERR_04473_NOT_VALID_VALUE Not a valid value '' for the AttributeType
'attributetype ( 2.5.4.3 NAME ( 'cn' 'commonName' )
DESC 'RFC2256: common name(s) for which the entity is known by'
SUP name
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
USAGE userApplications )'
at
org.apache.directory.api.ldap.model.entry.AbstractValue.apply(AbstractValue.java:219)
at
org.apache.directory.api.ldap.model.entry.StringValue.<init>(StringValue.java:122)
at
org.apache.directory.api.ldap.model.name.Dn.atavOidToName(Dn.java:1099)
... 23 more
Caused by:
org.apache.directory.api.ldap.model.exception.LdapInvalidAttributeValueException:
ERR_04473_NOT_VALID_VALUE Not a valid value '' for the AttributeType
'attributetype ( 2.5.4.3 NAME ( 'cn' 'commonName' )
DESC 'RFC2256: common name(s) for which the entity is known by'
SUP name
EQUALITY caseIgnoreMatch
SUBSTR caseIgnoreSubstringsMatch
SYNTAX 1.3.6.1.4.1.1466.115.121.1.15
USAGE userApplications )'
at
org.apache.directory.api.ldap.model.entry.AbstractValue.apply(AbstractValue.java:212)
... 25 more
Re: DN parsing troubles...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 08/10/15 20:27, Emmanuel Lécharny a écrit :
> One mor ething I forgot to mention...
>
> Escaping a space can be done in many ways :
>
> - '\ '
> - '\20'
> - '#20' (unlikely, but this is possible)
>
> All those three kind of escaped space must be handled, so that the space
> is kept after the prepareString processing.
>
There is something I'm not sure of : what is considered as an escaped
space. Currently, in the RFC, only '\ ' is, none of the other chars that
are translated to a space (and there are many !) will be considered as
escaped spaces.
If I interpret the RFC strictly, \20 ans #20 are not supposed to be
considered as escaped spaces. OTOH, would some of those representation
being seen in a String, this should probably be considered as a will to
see those chars being escaped and present after the normalization...
There are more chars that are going to be translated to a space :
0x09 to 0x0D will be tranlsated to 0x20, so is 0x85, 0xA0, 0x2000 to
0x200A, 0x2028 and 0x2029, 0x202F, 0x205F and 0x3000. That means any of
those chars will be removed if they are at the beginning or at the end
of the String, or if we have 2 contiguous chars that translate to a
space. And every hexpair like \09, \0A, \0B, \0C, \0D, \85, \A0 are
going to be seen as spaces, too.
At this point, I guess what is important is to be consistant. As soon as
we respect one form of escaped space (ie '\ ' only), and ignore all the
others, then operations like comparison of values, assuming they have
been both normalized the same way, should lead to the same result. The
only risk is that we may have some different values being compared and
resorting to an equality, when it should not. OTOH, we are talking about
use case that are very unlikely to happen... I'm not going to spend days
fixing corner cases that will probably never be faced in real life, when
we have more urgent bugs to fix ;-)
Re: DN parsing troubles...
Posted by Emmanuel Lécharny <el...@gmail.com>.
One mor ething I forgot to mention...
Escaping a space can be done in many ways :
- '\ '
- '\20'
- '#20' (unlikely, but this is possible)
All those three kind of escaped space must be handled, so that the space
is kept after the prepareString processing.
Re: DN parsing troubles...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Ok, moving forward...
Let's first give you some insight about the full problem.
A Dn is a list of Rdn, a Rdn is a list of Ava, and an Ava is an
attribute, the '=' sign and a value. When we parse a DN, we have to
create a Dn instance which contains all those elements. A DN
representation is ruled by the RFC 4514, which describe what to do when
some special characters are present (escaping them).
Bottom line, after having parsed the DN; we have a representation of it
in the Dn instance, and we have two ways to get its value : as it was
provided by the user (User Provided value, or UPValue) and as a
normilized value (trimmed, escaped chars converted to their counterpart,
etc).
This is all good, but we also have one more step : apply the schema,
which will transform each attribute and value to get a normalized form
which is dictated by the AttributeType (typically, we may lowercase teh
chars, remove duplicate spaces, etc...). And this is where we are in
trouble !
For instance, consider this RDN : "cn = Test\ ". This is a RDN in which
the valid value is 'test ', once normalized - weel, this is what we
should get -. Here, the last space is not removed because it is escaped
in the RDN, so the resulting normalized value must be kept. But this is
not what happen :/
Let's see how the parser works :
- first, we parse the RDn. We obtain this data structure :
Rdn :
upName : 'cn= TEST\ '
normName : 'cn=TEST\ ' (the useless spaces has been removed)
Ava :
upName : 'cn= TEST\ '
upType : 'cn'
normType : 'cn'
Value :
wrappedValue : 'TEST\ '
normalizedValue : 'TEST ' (the escaped space has been
replace by a plain space)
Everything is correct so far.
- now, we apply the SchemaManager on this Rdn instance, which is
propagated to the Ava and the Value. This is done by calling the
Dn.rdnOidToName(Rdn SchemaManager) method, which itself call the
atavOidToName( Ava, SchemaManager) method. This method will transform
teh Attribute into the associated OID found in the AttributeType
registry ( 2.5.4.11 for 'cn'), and then will try to normalize the Value
with :
atavValue = new StringValue( attributeType, value.getString() );
value.getString() returns the wrappedValue (ie, 'TEST\ '), and the
StringValue class will apply the AttributeType to this value :
public StringValue( AttributeType attributeType, String value )
throws LdapInvalidAttributeValueException
{
this( value, value );
apply( attributeType );
}
bottom line, it calls :
normalizedValue = ( T )
normalizer.normalize( ( String ) wrappedValue );
The Normalizer we use here is the CachingDeepTrimToLowerNormalizer
class, which is just (atm) a wrapper around the
DeepTrimToLowerNormalizer class (ie, we don't keep a cache atm). Here is
what this normalizer does :
/**
* Normalizer which trims down whitespace replacing multiple whitespace
* characters on the edges and within the string with a single space
character
* thereby preserving tokenization order - while doing all this in
the same pass
* it lower cases all characters.
...
this normalizer is calling this method :
PrepareString.normalize( value.getString(),
PrepareString.StringType.CASE_IGNORE );
which will remove spurious spaces around the value by calling the
insignifiantSpacesStringAscii( str, IGNORE_CASE ) method, which will
remove the escaped space :/
At this point, we are screwed because we have trandformed the value
unduly, because the insignifiantSpacesStringAscii method does not check
that the space is escaped...
Bottom line, we have something that works when teh RDN is not schema
aware, and I have fixed the Value, the Dn complex parser and the tests.
I now have to fix teh prepare string method. That should be done tomorrow...
Note that the change in Value will change the way we serialize the DN,
hopefully for teh better, as we do'nt store 2 values per AVA, but only
one, saving some space.
Re: DN parsing troubles...
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 08/10/15 09:52, Emmanuel Lécharny a écrit :
> Hi guys,
>
> I'm reviewing the DN Parser, as I think we are overdoing. So far, I have
> something that works, except a couple of tests that fail. One otf them is :
>
> public void testRdnWithEscapedComa() throws LdapException
> {
> ...
> Rdn rdn = new Rdn( "a=\"b\\,c\"" );
> assertEquals( "a=\"b\\,c\"", rdn.getName() );
> assertEquals( "a=b\\,c", rdn.getNormName() ); // WRONG !!!
>
> Here, I think the test is wrong. The initial value is surrounded by
> quotes, which means that we don't need to escape special chars like ','
> into it. Actually, the following value is accepted :
>
> Rdn rdn1 = new Rdn( "a=b\\,c" );
>
> and the RDN value is 'b,c' (the ',' is escaped to avoid a confusion with
> the next RDN in a DN)
>
> but when we use quotes, this is equivalent to :
>
> Rdn rdn2 = new Rdn( "a=\"b,c\"" );
>
> Here rdn1 <==> rdn2 (or it should be equivalent.
>
> On the other hand, this rdn :
>
> Rdn rdn3 = new Rdn( "a=\"b\\,c\"" );
>
> should be equivalent to :
>
> Rdn rdn4 = new Rdn( "a=b\\\,c" );
>
> where we must keep an escaped escape and an escaped coma.
>
> Thoughts ?
btw, the correct test should be :
/**
* test an Rdn with escaped comma
*/
@Test
public void testRdnWithEscapedComa() throws LdapException
{
...
assertTrue( Rdn.isValid( "a=\"b\\,c\"" ) );
Rdn rdn = new Rdn( "a=\"b\\,c\"" );
assertEquals( "a=\"b\\,c\"", rdn.getName() );
assertEquals( "a=b\\\\\\,c", rdn.getNormName() ); // <<----
This has been changed
}