[Studio][API] Binary attributes corruption issue

[Pierre-Arnaud Marcelot <pa...@marcelot.net>]

Hi guys,

Just a status report on the major issue we have in the API and Apache Directory Studio regarding binary attributes corruption.

For more background about this, read the following JIRA issues:
- [1] DIRAPI-78 (Binary attributes are not correctly recognized and are retrieved with data corruption)
- [2] DIRSTUDIO-772 (Entry UUID Editor can't display eDirectory GUID)
- [3] DIRSTUDIO-764 (Image Editor does not work correctly)

In a nutshell, due to the lack of schema manager, the LDAP API, which is used inside Studio, has no knowledge about binary attributes and handles all values as if they were string value.
Do so, for binary attributes, the received data is not correct and ends up being corrupted.

In order to fix this issue, Emmanuel and I worked on adding 2 ways for users to have binary attributes handled correctly.

1°/ The lightweight solution: the BinaryAttributeDetector

A DefaultConfigurableBinaryAttributeDetector class has been added to the configuration of an LDAP connection.
By default it includes a list of known binary attributes and methods to add and removed values to/from this list.
A user can also provide his own detector implementation via the BinaryAttributeDetector interface.

When the LDAP connection retrieves a new attribute value from a server, it will check via the BinaryAttributeDetector whether or not the attribute is binary and get the value according to this flag.

This first simple solution has been completely implemented and works fine for both the API and Studio.

2°/ The complete solution: loading the remote schema

The second solution is to make the connection schema-aware by allowing the load of the schema remotely.
It requires a lot more work as we're trying to load the whole schema of the server via LDAP through the SubSchemaSubEntry.

This solution is much more complex to implement than it seems and there are a lot of different corner cases:
- whether or not the SubSchemaSubEntry is exposed
- the kind of server (ApacheDS, OpenLDAP, OpenDJ, Active Directory, etc)
- is the schema complete and correct?

The more we implement this solution, the more we're hitting important issues with the internal design of the schema and how it is loaded (through the schema manager).

Implementing this second solution requires heavy refactoring of several crucial parts of the code (schema elements, schema loading objects, etc.) and might take longer than expected.

In the meantime, users of Studio and the LDAP API keep getting corrupted values for binary attributes. :(

I'd like to propose that we provide a new milestone for both projects (Studio 2.0 M3 and LDAP API 1.0 M11) that includes only the first solution (we also already have 15 resolved issues pending for Studio and 5 for API).

The second solution could then take place in the next milestone and we could take the appropriate amount of time to implement it correctly.

WDYT?

Regards,
Pierre-Arnaud

[1] - https://issues.apache.org/jira/browse/DIRAPI-78
[2] - https://issues.apache.org/jira/browse/DIRSTUDIO-772
[3] - https://issues.apache.org/jira/browse/DIRSTUDIO-764

Re: [Studio][API] Binary attributes corruption issue

[Kiran Ayyagari <ka...@apache.org>]

On Fri, Feb 17, 2012 at 8:20 PM, Pierre-Arnaud Marcelot <pa...@marcelot.net> wrote:
> Hi guys,
>
> Just a status report on the major issue we have in the API and Apache
> Directory Studio regarding binary attributes corruption.
>
> For more background about this, read the following JIRA issues:
> - [1] DIRAPI-78 (Binary attributes are not correctly recognized and are
> retrieved with data corruption)
> - [2] DIRSTUDIO-772 (Entry UUID Editor can't display eDirectory GUID)
> - [3] DIRSTUDIO-764 (Image Editor does not work correctly)
>
> In a nutshell, due to the lack of schema manager, the LDAP API, which is
> used inside Studio, has no knowledge about binary attributes and handles all
> values as if they were string value.
> Do so, for binary attributes, the received data is not correct and ends up
> being corrupted.
>
> In order to fix this issue, Emmanuel and I worked on adding 2 ways for users
> to have binary attributes handled correctly.
>
> 1°/ The lightweight solution: the BinaryAttributeDetector
>
> A DefaultConfigurableBinaryAttributeDetector class has been added to the
> configuration of an LDAP connection.
> By default it includes a list of known binary attributes and methods to add
> and removed values to/from this list.
> A user can also provide his own detector implementation via
> the BinaryAttributeDetector interface.
>
> When the LDAP connection retrieves a new attribute value from a server, it
> will check via the BinaryAttributeDetector whether or not the attribute is
> binary and get the value according to this flag.
>
> This first simple solution has been completely implemented and works fine
> for both the API and Studio.
>
> 2°/ The complete solution: loading the remote schema
>
> The second solution is to make the connection schema-aware by allowing the
> load of the schema remotely.
> It requires a lot more work as we're trying to load the whole schema of the
> server via LDAP through the SubSchemaSubEntry.
>
> This solution is much more complex to implement than it seems and there are
> a lot of different corner cases:
> - whether or not the SubSchemaSubEntry is exposed
> - the kind of server (ApacheDS, OpenLDAP, OpenDJ, Active Directory, etc)
> - is the schema complete and correct?
>
> The more we implement this solution, the more we're hitting important issues
> with the internal design of the schema and how it is loaded (through the
> schema manager).
>
> Implementing this second solution requires heavy refactoring of several
> crucial parts of the code (schema elements, schema loading objects, etc.)
> and might take longer than expected.
>
> In the meantime, users of Studio and the LDAP API keep getting corrupted
> values for binary attributes. :(
>
> I'd like to propose that we provide a new milestone for both projects
> (Studio 2.0 M3 and LDAP API 1.0 M11) that includes only the first solution
> (we also already have 15 resolved issues pending for Studio and 5 for API).
>
+1 , IMHO first is the only perfect solution, cause even RFC clearly
states that a client
should NOT assume anything about a server's published schema
> The second solution could then take place in the next milestone and we could
> take the appropriate amount of time to implement it correctly.
>
> WDYT?
>
> Regards,
> Pierre-Arnaud
>
> [1] - https://issues.apache.org/jira/browse/DIRAPI-78
> [2] - https://issues.apache.org/jira/browse/DIRSTUDIO-772
> [3] - https://issues.apache.org/jira/browse/DIRSTUDIO-764



-- 
Kiran Ayyagari