You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Cheong Chung Onn <ch...@greenfossil.com> on 2009/01/22 01:57:31 UTC
Apache Studio ldif export
Hi,
I had exported a branch of entries to an ldif file using Apache Studio
version 1.30 and when i tried to import the this ldif file i got
numerous errors. I realized the exported ldif file is not sorted in a
parent-child entry entry sequence, hence when i re-import this ldif file
- i received errors messages that it can't find a entry.
Is this the expected behavior of the Apache Studio export function or
did I missed out something during the export?
Re: Apache Studio ldif export
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
Hi Stefan,
I confirm I was using the "subtree" scope.
Emmanuel suspects the embedded Apache DS to not deliver the search results
in the correct hierarchical order.
As far as I remember in the LDIF export job, we're iterating on each result
given by the server through a NamingEnumeration (then converted as an
LdifEnumeration, or something like that...).
Now, the question is, Is it Apache DS' duty to deliver the search results in
the correct hierarchical order, or is it more up to the client (Studio) to
reorder the results afterwards.
I don't know what the RFCs specify for that.
Regards,
P-A
On Thu, Jan 22, 2009 at 10:51 AM, Stefan Seelmann <se...@apache.org>wrote:
> Hi,
>
> I think the problem is when doing an export with scope "subtree". In that
> case Studio just sends one search request with subtree scope and writes the
> entries in the same order as received to the LDIF.
>
> I think reordering them in Studio doesn't make sense, we can't hold them
> all in memory.
>
> An option would be to do multiple one-level scope searches.
>
> Kind Regards,
> Stefan
>
>
> Quoting Emmanuel Lecharny <el...@gmail.com>:
>
> Cheong Chung Onn wrote:
>>
>>> We are using ADS 1.5.4, Studio 1.3.0. Java.
>>>
>>> We created an embedded server in Studio then import the ldif file. Did
>>> some modification and export it again. When we import this newly exported
>>> ldif, we receive a bunch of error messages about missing object.
>>>
>>> Please let me know if you require more info.
>>>
>> Ok, this was what we suspected... A clear bug somewhere.
>>
>> Can you fill a JIRA with the informations you gave by mail ?
>> Eventually, providing the data you injected would help to reproduce the
>> problem, and determinate if it's a Studio or ADS problem. (of course,
>> if there are nothing confidential in it).
>>
>> Many thanks !
>>
>> --
>> --
>> cordialement, regards,
>> Emmanuel Lécharny
>> www.iktek.com
>> directory.apache.org
>>
>
>
>
>
Re: Apache Studio ldif export
Posted by Emmanuel Lecharny <el...@gmail.com>.
Cheong Chung Onn wrote:
> Both ApacheDS and Studio are awesome products, they have help us a lot
> in speeding up our development. However we found some hidden features
> or kinks in them which I would be posting to the forum for clarifications
Hidden features ? You mean, bugs ? :) We are very interested in any
informations or bug reports, feel free to fill JIRAs !
>> You could also fix it by providing indices to your partition. It
>> would be
>> better than having to re-order the exported LDIF afterwards.
>> But, your proposal for a re-order command is a good idea. You should
>> propose
>> this as a Jira.
>> Maybe we can find some time to implement it.
>>
> I thought the quickest and easiest way to work around this issue is to
> put a note in the wiki to forewarn the user of this peculiarity. It
> then become the onus of the user to add the indicies to the partition.
> The reason i didn't add the indices is because i only wanted add a
> couple of attributes to a few entries and i thought the easiest way
> was to create an embedded Server and import the schema and data ldif
> then modify and export.
Yes, I think it's a good way to avoid problems like the one you had.
However, I also think that ADS _should_ index ObjectClass automatically.
We already index many other internal things, and I see no reason not to
index ObjectClass.
Thanks Chungonn !
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: Apache Studio ldif export
Posted by Cheong Chung Onn <ch...@greenfossil.com>.
Hi Pierre-Arnaud,
My reply is in-line.
> It's really nice to hear that. Looks like the couple Apache Directory Server
> and Apache Directory Studio is doing a great job in your development
> environement. Do not hesitate to give us feedback on how we can improve
> both.
>
Both ApacheDS and Studio are awesome products, they have help us a lot
in speeding up our development. However we found some hidden features or
kinks in them which I would be posting to the forum for clarifications
> You could also fix it by providing indices to your partition. It would be
> better than having to re-order the exported LDIF afterwards.
> But, your proposal for a re-order command is a good idea. You should propose
> this as a Jira.
> Maybe we can find some time to implement it.
>
I thought the quickest and easiest way to work around this issue is to
put a note in the wiki to forewarn the user of this peculiarity. It then
become the onus of the user to add the indicies to the partition. The
reason i didn't add the indices is because i only wanted add a couple of
attributes to a few entries and i thought the easiest way was to create
an embedded Server and import the schema and data ldif then modify and
export.
Regards
chungonn
Re: Apache Studio ldif export
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
Hi Cheong Chung Onn,
On Fri, Jan 23, 2009 at 3:05 AM, Cheong Chung Onn
<ch...@greenfossil.com>wrote:
> Hi Pierre-Arnaud,
>
> Yes, I created a new partition without any index attributes. I too am
> lazy.. :)
Then, I'm not the only one. ;)
I think it would be better to provide a minimum set of indices when creating
a new partition.
Same thing for the other fields.
Having default values for all fields would help a lot I think.
Here's my 2 cents worth about - should Client or Server side do the
> ordering.
>
> 1. If the Studio or Client side would to do the ordering, that would mean
> all clients must implement the same piece of logic over and over again. That
> may cause future support issues.
>
> 2. If the Server side do the ordering then what happen if Studio is used
> against other Ldap Servers that does not order an exported entries?
>
> For me now, it does not matter whether Client or Server do the ordering,
> because the only tool i now use for Ldap development is Apache DS and
> Studio and sometimes OpenLdap to confirm my development works on it too.
It's really nice to hear that. Looks like the couple Apache Directory Server
and Apache Directory Studio is doing a great job in your development
environement. Do not hesitate to give us feedback on how we can improve
both.
Actually, we fixed the problem by reordering the ldif entries ourselves
> manually. I thought it would be cool if Studio Ldif Editor has a feature to
> re-order the ldif hierarchy.
You could also fix it by providing indices to your partition. It would be
better than having to re-order the exported LDIF afterwards.
But, your proposal for a re-order command is a good idea. You should propose
this as a Jira.
Maybe we can find some time to implement it.
Best regards,
Pierre-Arnaud
Re: Apache Studio ldif export
Posted by Cheong Chung Onn <ch...@greenfossil.com>.
Hi Pierre-Arnaud,
Yes, I created a new partition without any index attributes. I too am
lazy.. :)
Here's my 2 cents worth about - should Client or Server side do the
ordering.
1. If the Studio or Client side would to do the ordering, that would
mean all clients must implement the same piece of logic over and over
again. That may cause future support issues.
2. If the Server side do the ordering then what happen if Studio is used
against other Ldap Servers that does not order an exported entries?
For me now, it does not matter whether Client or Server do the
ordering, because the only tool i now use for Ldap development is
Apache DS and Studio and sometimes OpenLdap to confirm my development
works on it too.
Actually, we fixed the problem by reordering the ldif entries ourselves
manually. I thought it would be cool if Studio Ldif Editor has a feature
to re-order the ldif hierarchy.
Pierre-Arnaud Marcelot wrote:
> Ok, I did more tests this morning.
>
> With Emmanuel we figured out this was a problem of indexed attributes on our
> testing partition.
>
> My initial partition, where I'm getting the error does not have any index (I
> was tool lazy to add some at the creation of the partition). This lack of
> index, particularly on the 'objectClass' attribute makes Apache DS return
> the search results in an unordered way.
>
> For the purpose of the test I created the exact same partition but added
> basic indices on it, I imported the very same data and the LDIF export was
> correctly ordered.
>
> Cheong Chung Onn, do you have the same thing? Did you create a new partition
> with an empty list of indexed attributes?
>
> Now, the question is: On the Studio side, Do we take for granted that search
> results given from the LDAP Server are always correctly hierarchically
> ordered (even if no RFC clearly specify it, just because it's the common way
> of returning the results), or do we add additional code to verify and,
> eventually, reorder the entries when writing the LDIF export file ?
>
> Another question: Do we need to raise a JIRA for the problem we've face with
> Apache DS when a partition does not have an index on the 'objectClass'
> attribute ?
>
> Regards,
> Pierre-Arnaud
>
> PS: A copy of this mail is attached to the JIRA
> (DIRSTUDIO-448<https://issues.apache.org/jira/browse/DIRSTUDIO-448>),
> to keep track of the discussion going on here.
>
>
> On Thu, Jan 22, 2009 at 11:06 AM, Cheong Chung Onn <chungonn@greenfossil.com
>
>> wrote:
>>
>
>
>> Hi,
>>
>> I have just created a JIRA
>> https://issues.apache.org/jira/browse/DIRSTUDIO-448
>>
>> The export i did was always with scope "subtree". I just did another 3
>> round of tests using a new embedded DS Server. I made modifications to the
>> entries and exported ldif, each time the exported ldif file is in the
>> correct sequence now.
>>
>>
>>
>> Stefan Seelmann wrote:
>>
>>
>>> Hi,
>>>
>>> I think the problem is when doing an export with scope "subtree". In that
>>> case Studio just sends one search request with subtree scope and writes the
>>> entries in the same order as received to the LDIF.
>>>
>>> I think reordering them in Studio doesn't make sense, we can't hold them
>>> all in memory.
>>>
>>> An option would be to do multiple one-level scope searches.
>>>
>>> Kind Regards,
>>> Stefan
>>>
>>> Quoting Emmanuel Lecharny <el...@gmail.com>:
>>>
>>> Cheong Chung Onn wrote:
>>>
>>>>> We are using ADS 1.5.4, Studio 1.3.0. Java.
>>>>>
>>>>> We created an embedded server in Studio then import the ldif file. Did
>>>>> some modification and export it again. When we import this newly exported
>>>>> ldif, we receive a bunch of error messages about missing object.
>>>>>
>>>>> Please let me know if you require more info.
>>>>>
>>>>>
>>>> Ok, this was what we suspected... A clear bug somewhere.
>>>>
>>>> Can you fill a JIRA with the informations you gave by mail ?
>>>> Eventually, providing the data you injected would help to reproduce the
>>>> problem, and determinate if it's a Studio or ADS problem. (of course,
>>>> if there are nothing confidential in it).
>>>>
>>>> Many thanks !
>>>>
>>>> --
>>>> --
>>>> cordialement, regards,
>>>> Emmanuel Lécharny
>>>> www.iktek.com
>>>> directory.apache.org
>>>>
>>>>
>>>
>>>
>>>
>
>
Re: Apache Studio ldif export
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
Ok, I did more tests this morning.
With Emmanuel we figured out this was a problem of indexed attributes on our
testing partition.
My initial partition, where I'm getting the error does not have any index (I
was tool lazy to add some at the creation of the partition). This lack of
index, particularly on the 'objectClass' attribute makes Apache DS return
the search results in an unordered way.
For the purpose of the test I created the exact same partition but added
basic indices on it, I imported the very same data and the LDIF export was
correctly ordered.
Cheong Chung Onn, do you have the same thing? Did you create a new partition
with an empty list of indexed attributes?
Now, the question is: On the Studio side, Do we take for granted that search
results given from the LDAP Server are always correctly hierarchically
ordered (even if no RFC clearly specify it, just because it's the common way
of returning the results), or do we add additional code to verify and,
eventually, reorder the entries when writing the LDIF export file ?
Another question: Do we need to raise a JIRA for the problem we've face with
Apache DS when a partition does not have an index on the 'objectClass'
attribute ?
Regards,
Pierre-Arnaud
PS: A copy of this mail is attached to the JIRA
(DIRSTUDIO-448<https://issues.apache.org/jira/browse/DIRSTUDIO-448>),
to keep track of the discussion going on here.
On Thu, Jan 22, 2009 at 11:06 AM, Cheong Chung Onn <chungonn@greenfossil.com
> wrote:
> Hi,
>
> I have just created a JIRA
> https://issues.apache.org/jira/browse/DIRSTUDIO-448
>
> The export i did was always with scope "subtree". I just did another 3
> round of tests using a new embedded DS Server. I made modifications to the
> entries and exported ldif, each time the exported ldif file is in the
> correct sequence now.
>
>
>
> Stefan Seelmann wrote:
>
>> Hi,
>>
>> I think the problem is when doing an export with scope "subtree". In that
>> case Studio just sends one search request with subtree scope and writes the
>> entries in the same order as received to the LDIF.
>>
>> I think reordering them in Studio doesn't make sense, we can't hold them
>> all in memory.
>>
>> An option would be to do multiple one-level scope searches.
>>
>> Kind Regards,
>> Stefan
>>
>> Quoting Emmanuel Lecharny <el...@gmail.com>:
>>
>> Cheong Chung Onn wrote:
>>>
>>>> We are using ADS 1.5.4, Studio 1.3.0. Java.
>>>>
>>>> We created an embedded server in Studio then import the ldif file. Did
>>>> some modification and export it again. When we import this newly exported
>>>> ldif, we receive a bunch of error messages about missing object.
>>>>
>>>> Please let me know if you require more info.
>>>>
>>> Ok, this was what we suspected... A clear bug somewhere.
>>>
>>> Can you fill a JIRA with the informations you gave by mail ?
>>> Eventually, providing the data you injected would help to reproduce the
>>> problem, and determinate if it's a Studio or ADS problem. (of course,
>>> if there are nothing confidential in it).
>>>
>>> Many thanks !
>>>
>>> --
>>> --
>>> cordialement, regards,
>>> Emmanuel Lécharny
>>> www.iktek.com
>>> directory.apache.org
>>>
>>
>>
>>
>>
Re: Apache Studio ldif export
Posted by Cheong Chung Onn <ch...@greenfossil.com>.
Hi,
I have just created a JIRA
https://issues.apache.org/jira/browse/DIRSTUDIO-448
The export i did was always with scope "subtree". I just did another 3
round of tests using a new embedded DS Server. I made modifications to
the entries and exported ldif, each time the exported ldif file is in
the correct sequence now.
Stefan Seelmann wrote:
> Hi,
>
> I think the problem is when doing an export with scope "subtree". In
> that case Studio just sends one search request with subtree scope and
> writes the entries in the same order as received to the LDIF.
>
> I think reordering them in Studio doesn't make sense, we can't hold
> them all in memory.
>
> An option would be to do multiple one-level scope searches.
>
> Kind Regards,
> Stefan
>
> Quoting Emmanuel Lecharny <el...@gmail.com>:
>
>> Cheong Chung Onn wrote:
>>> We are using ADS 1.5.4, Studio 1.3.0. Java.
>>>
>>> We created an embedded server in Studio then import the ldif file.
>>> Did some modification and export it again. When we import this
>>> newly exported ldif, we receive a bunch of error messages about
>>> missing object.
>>>
>>> Please let me know if you require more info.
>> Ok, this was what we suspected... A clear bug somewhere.
>>
>> Can you fill a JIRA with the informations you gave by mail ?
>> Eventually, providing the data you injected would help to reproduce the
>> problem, and determinate if it's a Studio or ADS problem. (of course,
>> if there are nothing confidential in it).
>>
>> Many thanks !
>>
>> --
>> --
>> cordialement, regards,
>> Emmanuel Lécharny
>> www.iktek.com
>> directory.apache.org
>
>
>
Re: Apache Studio ldif export
Posted by Stefan Seelmann <se...@apache.org>.
Hi,
I think the problem is when doing an export with scope "subtree". In
that case Studio just sends one search request with subtree scope and
writes the entries in the same order as received to the LDIF.
I think reordering them in Studio doesn't make sense, we can't hold
them all in memory.
An option would be to do multiple one-level scope searches.
Kind Regards,
Stefan
Quoting Emmanuel Lecharny <el...@gmail.com>:
> Cheong Chung Onn wrote:
>> We are using ADS 1.5.4, Studio 1.3.0. Java.
>>
>> We created an embedded server in Studio then import the ldif file.
>> Did some modification and export it again. When we import this
>> newly exported ldif, we receive a bunch of error messages about
>> missing object.
>>
>> Please let me know if you require more info.
> Ok, this was what we suspected... A clear bug somewhere.
>
> Can you fill a JIRA with the informations you gave by mail ?
> Eventually, providing the data you injected would help to reproduce the
> problem, and determinate if it's a Studio or ADS problem. (of course,
> if there are nothing confidential in it).
>
> Many thanks !
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
Re: Apache Studio ldif export
Posted by Emmanuel Lecharny <el...@gmail.com>.
Cheong Chung Onn wrote:
> We are using ADS 1.5.4, Studio 1.3.0. Java.
>
> We created an embedded server in Studio then import the ldif file. Did
> some modification and export it again. When we import this newly
> exported ldif, we receive a bunch of error messages about missing object.
>
> Please let me know if you require more info.
Ok, this was what we suspected... A clear bug somewhere.
Can you fill a JIRA with the informations you gave by mail ? Eventually,
providing the data you injected would help to reproduce the problem, and
determinate if it's a Studio or ADS problem. (of course, if there are
nothing confidential in it).
Many thanks !
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org
Re: Apache Studio ldif export
Posted by Cheong Chung Onn <ch...@greenfossil.com>.
We are using ADS 1.5.4, Studio 1.3.0. Java.
We created an embedded server in Studio then import the ldif file. Did
some modification and export it again. When we import this newly
exported ldif, we receive a bunch of error messages about missing object.
Please let me know if you require more info.
Emmanuel Lecharny wrote:
> Cheong Chung Onn wrote:
>> Hi,
>>
>> I had exported a branch of entries to an ldif file using Apache
>> Studio version 1.30 and when i tried to import the this ldif file i
>> got numerous errors. I realized the exported ldif file is not sorted
>> in a parent-child entry entry sequence, hence when i re-import this
>> ldif file - i received errors messages that it can't find a entry.
>
> Interesting ... Which server are you exporting entries from ?
>
> FYI, we have faced the very same problem this afternoon, with the
> inner server, and we are just wondering if it's a Studio problem or a
> ADS problem...
>
>
Re: Apache Studio ldif export
Posted by Pierre-Arnaud Marcelot <pa...@marcelot.net>.
Exactly.
We've had the same issue yesterday. What a coincidence.
I'm going to try to debug that this afternoon and see whose fault it is,
Client or Server.
Regards,
Pierre-Arnaud
On Thu, Jan 22, 2009 at 2:04 AM, Emmanuel Lecharny <el...@gmail.com>wrote:
> Cheong Chung Onn wrote:
>
>> Hi,
>>
>> I had exported a branch of entries to an ldif file using Apache Studio
>> version 1.30 and when i tried to import the this ldif file i got numerous
>> errors. I realized the exported ldif file is not sorted in a parent-child
>> entry entry sequence, hence when i re-import this ldif file - i received
>> errors messages that it can't find a entry.
>>
>
> Interesting ... Which server are you exporting entries from ?
>
> FYI, we have faced the very same problem this afternoon, with the inner
> server, and we are just wondering if it's a Studio problem or a ADS
> problem...
>
>
> --
> --
> cordialement, regards,
> Emmanuel Lécharny
> www.iktek.com
> directory.apache.org
>
>
>
Re: Apache Studio ldif export
Posted by Emmanuel Lecharny <el...@gmail.com>.
Cheong Chung Onn wrote:
> Hi,
>
> I had exported a branch of entries to an ldif file using Apache Studio
> version 1.30 and when i tried to import the this ldif file i got
> numerous errors. I realized the exported ldif file is not sorted in a
> parent-child entry entry sequence, hence when i re-import this ldif
> file - i received errors messages that it can't find a entry.
Interesting ... Which server are you exporting entries from ?
FYI, we have faced the very same problem this afternoon, with the inner
server, and we are just wondering if it's a Studio problem or a ADS
problem...
--
--
cordialement, regards,
Emmanuel Lécharny
www.iktek.com
directory.apache.org