You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Ezsra McDonald <ez...@gmail.com> on 2015/06/11 05:31:47 UTC
ApacheDS Import via CLI
So, I need to import 330k records to my directory in a short amount of
time. Is there a CLI that will do the job or is the Studio the only option?
I am running 2.0.0 M19.
Thanks,
--EZ
Re: ApacheDS Import via CLI
Posted by Kiran Ayyagari <ka...@apache.org>.
On Thu, Jun 25, 2015 at 4:49 AM, Ezsra McDonald <ez...@gmail.com>
wrote:
> So, after some testing we are planning to import to one of the four master
> with replication disabled. We will then stop the instance, tar up and
> distribute the partition folder to the other three nodes.
>
> just make sure you don't copy the ou=config and ou=system partitions to
the other nodes
> Thanks for the help.
>
> On Mon, Jun 15, 2015 at 9:49 PM, Kiran Ayyagari <ka...@apache.org>
> wrote:
>
> > On Tue, Jun 16, 2015 at 5:49 AM, Ezsra McDonald <
> ezsra.mcdonald@gmail.com>
> > wrote:
> >
> > > Carlo,
> > > Yes, Excellent suggestion Carlo! That did the trick.
> > >
> > > Now, I have a total of four servers in the Multi-Master setup, nodes A
> > and
> > > B in data center 1 and nodes C and D in data center 2.
> > >
> > > If I ldapadd to node A and only have replication enabled with node B I
> > get
> > > 4500 entries per minute. When I add nodes C and D to the replication I
> > get
> > > 1800 entries per minute.
> > >
> > > What is involved if I wanted to copy the data files after the import
> from
> > > node A to nodes C and D? Do I just tar up the related partition folder
> > from
> > > node A and untar it on nodes C and D? Or, is there more involved? What
> is
> > > the procedure?
> > >
> > stop the server before copying and exclude system and config partitions
> > while copying
> >
> > >
> > >
> > >
> > > On Mon, Jun 15, 2015 at 9:35 AM, <Ca...@ibs-ag.com> wrote:
> > >
> > > > Not sure how you have this set, but for our initial import we set the
> > > > following property:
> > > >
> > > > ads-partitionsynconwrite: FALSE
> > > >
> > > > This property is found where you define your partition.
> > > > dn:
> > > >
> > >
> >
> ads-partitionId=mypartition,ou=partitions,ads-directoryServiceId=default,ou=config
> > > >
> > > > Setting this false allowed us to import ~80K entries in about 15-20
> > > mins.
> > > >
> > > > Also, if you're using password policies, we disabled those too for
> the
> > > > initial import.
> > > >
> > > > Good Luck.
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Ezsra McDonald [mailto:ezsra.mcdonald@gmail.com]
> > > > Sent: Saturday, June 13, 2015 5:16 PM
> > > > To: users
> > > > Subject: Re: ApacheDS Import via CLI
> > > >
> > > > Having the server down for the initial import is not a problem.
> > > >
> > > > I am able to load 650 - 700 entries a minute. With almost 340k
> entries
> > to
> > > > import this will take too long. My estimate is approximate 8.5 hours.
> > > >
> > > > I tried breaking the import into four files and executing four
> ldapadds
> > > > simultaneously but that does not improve the load rate. I really did
> > not
> > > > expect the import would be faster but I am desperate to get the
> > migration
> > > > done in less than three hours.
> > > >
> > > > This is just informational. I really don't expect there are any
> > solutions
> > > > to meet my desired 3 hour window.
> > > >
> > > > Thanks for the assistance. I will inform my team of my findings.
> > > >
> > > >
> > > > On Fri, Jun 12, 2015 at 10:29 PM, Emmanuel Lécharny <
> > elecharny@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Le 12/06/15 23:18, Ezsra McDonald a écrit :
> > > > > > The OpenLDAP ldapadd utility works. I wish there was a direct
> load
> > > > > > kind
> > > > > of
> > > > > > utility like slapadd. Ldapadd is slow but studio is slower.
> > > > >
> > > > > Everything that inject data into a live server will be slow.
> > > > >
> > > > > In the near future, we will have a bulk import tool that will be
> way
> > > > > faster, but it will require the server to be down.
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: ApacheDS Import via CLI
Posted by Ezsra McDonald <ez...@gmail.com>.
So, after some testing we are planning to import to one of the four master
with replication disabled. We will then stop the instance, tar up and
distribute the partition folder to the other three nodes.
Thanks for the help.
On Mon, Jun 15, 2015 at 9:49 PM, Kiran Ayyagari <ka...@apache.org>
wrote:
> On Tue, Jun 16, 2015 at 5:49 AM, Ezsra McDonald <ez...@gmail.com>
> wrote:
>
> > Carlo,
> > Yes, Excellent suggestion Carlo! That did the trick.
> >
> > Now, I have a total of four servers in the Multi-Master setup, nodes A
> and
> > B in data center 1 and nodes C and D in data center 2.
> >
> > If I ldapadd to node A and only have replication enabled with node B I
> get
> > 4500 entries per minute. When I add nodes C and D to the replication I
> get
> > 1800 entries per minute.
> >
> > What is involved if I wanted to copy the data files after the import from
> > node A to nodes C and D? Do I just tar up the related partition folder
> from
> > node A and untar it on nodes C and D? Or, is there more involved? What is
> > the procedure?
> >
> stop the server before copying and exclude system and config partitions
> while copying
>
> >
> >
> >
> > On Mon, Jun 15, 2015 at 9:35 AM, <Ca...@ibs-ag.com> wrote:
> >
> > > Not sure how you have this set, but for our initial import we set the
> > > following property:
> > >
> > > ads-partitionsynconwrite: FALSE
> > >
> > > This property is found where you define your partition.
> > > dn:
> > >
> >
> ads-partitionId=mypartition,ou=partitions,ads-directoryServiceId=default,ou=config
> > >
> > > Setting this false allowed us to import ~80K entries in about 15-20
> > mins.
> > >
> > > Also, if you're using password policies, we disabled those too for the
> > > initial import.
> > >
> > > Good Luck.
> > >
> > >
> > > -----Original Message-----
> > > From: Ezsra McDonald [mailto:ezsra.mcdonald@gmail.com]
> > > Sent: Saturday, June 13, 2015 5:16 PM
> > > To: users
> > > Subject: Re: ApacheDS Import via CLI
> > >
> > > Having the server down for the initial import is not a problem.
> > >
> > > I am able to load 650 - 700 entries a minute. With almost 340k entries
> to
> > > import this will take too long. My estimate is approximate 8.5 hours.
> > >
> > > I tried breaking the import into four files and executing four ldapadds
> > > simultaneously but that does not improve the load rate. I really did
> not
> > > expect the import would be faster but I am desperate to get the
> migration
> > > done in less than three hours.
> > >
> > > This is just informational. I really don't expect there are any
> solutions
> > > to meet my desired 3 hour window.
> > >
> > > Thanks for the assistance. I will inform my team of my findings.
> > >
> > >
> > > On Fri, Jun 12, 2015 at 10:29 PM, Emmanuel Lécharny <
> elecharny@gmail.com
> > >
> > > wrote:
> > >
> > > > Le 12/06/15 23:18, Ezsra McDonald a écrit :
> > > > > The OpenLDAP ldapadd utility works. I wish there was a direct load
> > > > > kind
> > > > of
> > > > > utility like slapadd. Ldapadd is slow but studio is slower.
> > > >
> > > > Everything that inject data into a live server will be slow.
> > > >
> > > > In the near future, we will have a bulk import tool that will be way
> > > > faster, but it will require the server to be down.
> > > >
> > > >
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: ApacheDS Import via CLI
Posted by Kiran Ayyagari <ka...@apache.org>.
On Tue, Jun 16, 2015 at 5:49 AM, Ezsra McDonald <ez...@gmail.com>
wrote:
> Carlo,
> Yes, Excellent suggestion Carlo! That did the trick.
>
> Now, I have a total of four servers in the Multi-Master setup, nodes A and
> B in data center 1 and nodes C and D in data center 2.
>
> If I ldapadd to node A and only have replication enabled with node B I get
> 4500 entries per minute. When I add nodes C and D to the replication I get
> 1800 entries per minute.
>
> What is involved if I wanted to copy the data files after the import from
> node A to nodes C and D? Do I just tar up the related partition folder from
> node A and untar it on nodes C and D? Or, is there more involved? What is
> the procedure?
>
stop the server before copying and exclude system and config partitions
while copying
>
>
>
> On Mon, Jun 15, 2015 at 9:35 AM, <Ca...@ibs-ag.com> wrote:
>
> > Not sure how you have this set, but for our initial import we set the
> > following property:
> >
> > ads-partitionsynconwrite: FALSE
> >
> > This property is found where you define your partition.
> > dn:
> >
> ads-partitionId=mypartition,ou=partitions,ads-directoryServiceId=default,ou=config
> >
> > Setting this false allowed us to import ~80K entries in about 15-20
> mins.
> >
> > Also, if you're using password policies, we disabled those too for the
> > initial import.
> >
> > Good Luck.
> >
> >
> > -----Original Message-----
> > From: Ezsra McDonald [mailto:ezsra.mcdonald@gmail.com]
> > Sent: Saturday, June 13, 2015 5:16 PM
> > To: users
> > Subject: Re: ApacheDS Import via CLI
> >
> > Having the server down for the initial import is not a problem.
> >
> > I am able to load 650 - 700 entries a minute. With almost 340k entries to
> > import this will take too long. My estimate is approximate 8.5 hours.
> >
> > I tried breaking the import into four files and executing four ldapadds
> > simultaneously but that does not improve the load rate. I really did not
> > expect the import would be faster but I am desperate to get the migration
> > done in less than three hours.
> >
> > This is just informational. I really don't expect there are any solutions
> > to meet my desired 3 hour window.
> >
> > Thanks for the assistance. I will inform my team of my findings.
> >
> >
> > On Fri, Jun 12, 2015 at 10:29 PM, Emmanuel Lécharny <elecharny@gmail.com
> >
> > wrote:
> >
> > > Le 12/06/15 23:18, Ezsra McDonald a écrit :
> > > > The OpenLDAP ldapadd utility works. I wish there was a direct load
> > > > kind
> > > of
> > > > utility like slapadd. Ldapadd is slow but studio is slower.
> > >
> > > Everything that inject data into a live server will be slow.
> > >
> > > In the near future, we will have a bulk import tool that will be way
> > > faster, but it will require the server to be down.
> > >
> > >
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: ApacheDS Import via CLI
Posted by Ezsra McDonald <ez...@gmail.com>.
Carlo,
Yes, Excellent suggestion Carlo! That did the trick.
Now, I have a total of four servers in the Multi-Master setup, nodes A and
B in data center 1 and nodes C and D in data center 2.
If I ldapadd to node A and only have replication enabled with node B I get
4500 entries per minute. When I add nodes C and D to the replication I get
1800 entries per minute.
What is involved if I wanted to copy the data files after the import from
node A to nodes C and D? Do I just tar up the related partition folder from
node A and untar it on nodes C and D? Or, is there more involved? What is
the procedure?
On Mon, Jun 15, 2015 at 9:35 AM, <Ca...@ibs-ag.com> wrote:
> Not sure how you have this set, but for our initial import we set the
> following property:
>
> ads-partitionsynconwrite: FALSE
>
> This property is found where you define your partition.
> dn:
> ads-partitionId=mypartition,ou=partitions,ads-directoryServiceId=default,ou=config
>
> Setting this false allowed us to import ~80K entries in about 15-20 mins.
>
> Also, if you're using password policies, we disabled those too for the
> initial import.
>
> Good Luck.
>
>
> -----Original Message-----
> From: Ezsra McDonald [mailto:ezsra.mcdonald@gmail.com]
> Sent: Saturday, June 13, 2015 5:16 PM
> To: users
> Subject: Re: ApacheDS Import via CLI
>
> Having the server down for the initial import is not a problem.
>
> I am able to load 650 - 700 entries a minute. With almost 340k entries to
> import this will take too long. My estimate is approximate 8.5 hours.
>
> I tried breaking the import into four files and executing four ldapadds
> simultaneously but that does not improve the load rate. I really did not
> expect the import would be faster but I am desperate to get the migration
> done in less than three hours.
>
> This is just informational. I really don't expect there are any solutions
> to meet my desired 3 hour window.
>
> Thanks for the assistance. I will inform my team of my findings.
>
>
> On Fri, Jun 12, 2015 at 10:29 PM, Emmanuel Lécharny <el...@gmail.com>
> wrote:
>
> > Le 12/06/15 23:18, Ezsra McDonald a écrit :
> > > The OpenLDAP ldapadd utility works. I wish there was a direct load
> > > kind
> > of
> > > utility like slapadd. Ldapadd is slow but studio is slower.
> >
> > Everything that inject data into a live server will be slow.
> >
> > In the near future, we will have a bulk import tool that will be way
> > faster, but it will require the server to be down.
> >
> >
>
RE: ApacheDS Import via CLI
Posted by Ca...@ibs-ag.com.
Not sure how you have this set, but for our initial import we set the following property:
ads-partitionsynconwrite: FALSE
This property is found where you define your partition.
dn: ads-partitionId=mypartition,ou=partitions,ads-directoryServiceId=default,ou=config
Setting this false allowed us to import ~80K entries in about 15-20 mins.
Also, if you're using password policies, we disabled those too for the initial import.
Good Luck.
-----Original Message-----
From: Ezsra McDonald [mailto:ezsra.mcdonald@gmail.com]
Sent: Saturday, June 13, 2015 5:16 PM
To: users
Subject: Re: ApacheDS Import via CLI
Having the server down for the initial import is not a problem.
I am able to load 650 - 700 entries a minute. With almost 340k entries to import this will take too long. My estimate is approximate 8.5 hours.
I tried breaking the import into four files and executing four ldapadds simultaneously but that does not improve the load rate. I really did not expect the import would be faster but I am desperate to get the migration done in less than three hours.
This is just informational. I really don't expect there are any solutions to meet my desired 3 hour window.
Thanks for the assistance. I will inform my team of my findings.
On Fri, Jun 12, 2015 at 10:29 PM, Emmanuel Lécharny <el...@gmail.com>
wrote:
> Le 12/06/15 23:18, Ezsra McDonald a écrit :
> > The OpenLDAP ldapadd utility works. I wish there was a direct load
> > kind
> of
> > utility like slapadd. Ldapadd is slow but studio is slower.
>
> Everything that inject data into a live server will be slow.
>
> In the near future, we will have a bulk import tool that will be way
> faster, but it will require the server to be down.
>
>
Re: ApacheDS Import via CLI
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 13/06/15 23:16, Ezsra McDonald a écrit :
> Having the server down for the initial import is not a problem.
>
> I am able to load 650 - 700 entries a minute. With almost 340k entries to
> import this will take too long. My estimate is approximate 8.5 hours.
>
> I tried breaking the import into four files and executing four ldapadds
> simultaneously but that does not improve the load rate. I really did not
> expect the import would be faster but I am desperate to get the migration
> done in less than three hours.
>
> This is just informational. I really don't expect there are any solutions
> to meet my desired 3 hour window.
>
> Thanks for the assistance. I will inform my team of my findings.
Are you using a SSD ? On my laptop, with a SSD, the load rate is around
50 entries per second, which is around 5x faster than what you get.
Re: ApacheDS Import via CLI
Posted by Ezsra McDonald <ez...@gmail.com>.
Having the server down for the initial import is not a problem.
I am able to load 650 - 700 entries a minute. With almost 340k entries to
import this will take too long. My estimate is approximate 8.5 hours.
I tried breaking the import into four files and executing four ldapadds
simultaneously but that does not improve the load rate. I really did not
expect the import would be faster but I am desperate to get the migration
done in less than three hours.
This is just informational. I really don't expect there are any solutions
to meet my desired 3 hour window.
Thanks for the assistance. I will inform my team of my findings.
On Fri, Jun 12, 2015 at 10:29 PM, Emmanuel Lécharny <el...@gmail.com>
wrote:
> Le 12/06/15 23:18, Ezsra McDonald a écrit :
> > The OpenLDAP ldapadd utility works. I wish there was a direct load kind
> of
> > utility like slapadd. Ldapadd is slow but studio is slower.
>
> Everything that inject data into a live server will be slow.
>
> In the near future, we will have a bulk import tool that will be way
> faster, but it will require the server to be down.
>
>
Re: ApacheDS Import via CLI
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 12/06/15 23:18, Ezsra McDonald a écrit :
> The OpenLDAP ldapadd utility works. I wish there was a direct load kind of
> utility like slapadd. Ldapadd is slow but studio is slower.
Everything that inject data into a live server will be slow.
In the near future, we will have a bulk import tool that will be way
faster, but it will require the server to be down.
Re: ApacheDS Import via CLI
Posted by Ezsra McDonald <ez...@gmail.com>.
The OpenLDAP ldapadd utility works. I wish there was a direct load kind of
utility like slapadd. Ldapadd is slow but studio is slower.
Anyway, thanks for the help.
--Ez
On Thu, Jun 11, 2015 at 1:24 AM, Ezsra McDonald <ez...@gmail.com>
wrote:
> Thanks, I will give it a try.
>
> On Thu, Jun 11, 2015 at 1:23 AM, Kiran Ayyagari <ka...@apache.org>
> wrote:
>
>> On Thu, Jun 11, 2015 at 2:17 PM, Ezsra McDonald <ezsra.mcdonald@gmail.com
>> >
>> wrote:
>>
>> > Is that from the Linux openldap-clients RPM or somewhere else?
>> >
>> yes, from that package
>>
>> >
>> > On Wed, Jun 10, 2015 at 10:57 PM, Kiran Ayyagari <ka...@apache.org>
>> > wrote:
>> >
>> > > On Thu, Jun 11, 2015 at 11:31 AM, Ezsra McDonald <
>> > ezsra.mcdonald@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > So, I need to import 330k records to my directory in a short amount
>> of
>> > > > time. Is there a CLI that will do the job or is the Studio the only
>> > > option?
>> > > >
>> > > > one easy way is to use ldapadd command, e.g
>> > >
>> > > ldapadd -H ldap://localhost:10389 -x -D "uid=admin,ou=system" -W -f
>> > > mydata.ldif
>> > >
>> > > make sure that your LDIF file is sorted in parent entry first order.
>> > >
>> > > I am running 2.0.0 M19.
>> > > >
>> > > > Thanks,
>> > > >
>> > > > --EZ
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kiran Ayyagari
>> > > http://keydap.com
>> > >
>> >
>>
>>
>>
>> --
>> Kiran Ayyagari
>> http://keydap.com
>>
>
>
Re: ApacheDS Import via CLI
Posted by Ezsra McDonald <ez...@gmail.com>.
Thanks, I will give it a try.
On Thu, Jun 11, 2015 at 1:23 AM, Kiran Ayyagari <ka...@apache.org>
wrote:
> On Thu, Jun 11, 2015 at 2:17 PM, Ezsra McDonald <ez...@gmail.com>
> wrote:
>
> > Is that from the Linux openldap-clients RPM or somewhere else?
> >
> yes, from that package
>
> >
> > On Wed, Jun 10, 2015 at 10:57 PM, Kiran Ayyagari <ka...@apache.org>
> > wrote:
> >
> > > On Thu, Jun 11, 2015 at 11:31 AM, Ezsra McDonald <
> > ezsra.mcdonald@gmail.com
> > > >
> > > wrote:
> > >
> > > > So, I need to import 330k records to my directory in a short amount
> of
> > > > time. Is there a CLI that will do the job or is the Studio the only
> > > option?
> > > >
> > > > one easy way is to use ldapadd command, e.g
> > >
> > > ldapadd -H ldap://localhost:10389 -x -D "uid=admin,ou=system" -W -f
> > > mydata.ldif
> > >
> > > make sure that your LDIF file is sorted in parent entry first order.
> > >
> > > I am running 2.0.0 M19.
> > > >
> > > > Thanks,
> > > >
> > > > --EZ
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Ayyagari
> > > http://keydap.com
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: ApacheDS Import via CLI
Posted by Kiran Ayyagari <ka...@apache.org>.
On Thu, Jun 11, 2015 at 2:17 PM, Ezsra McDonald <ez...@gmail.com>
wrote:
> Is that from the Linux openldap-clients RPM or somewhere else?
>
yes, from that package
>
> On Wed, Jun 10, 2015 at 10:57 PM, Kiran Ayyagari <ka...@apache.org>
> wrote:
>
> > On Thu, Jun 11, 2015 at 11:31 AM, Ezsra McDonald <
> ezsra.mcdonald@gmail.com
> > >
> > wrote:
> >
> > > So, I need to import 330k records to my directory in a short amount of
> > > time. Is there a CLI that will do the job or is the Studio the only
> > option?
> > >
> > > one easy way is to use ldapadd command, e.g
> >
> > ldapadd -H ldap://localhost:10389 -x -D "uid=admin,ou=system" -W -f
> > mydata.ldif
> >
> > make sure that your LDIF file is sorted in parent entry first order.
> >
> > I am running 2.0.0 M19.
> > >
> > > Thanks,
> > >
> > > --EZ
> > >
> >
> >
> >
> > --
> > Kiran Ayyagari
> > http://keydap.com
> >
>
--
Kiran Ayyagari
http://keydap.com
Re: ApacheDS Import via CLI
Posted by Ezsra McDonald <ez...@gmail.com>.
Is that from the Linux openldap-clients RPM or somewhere else?
On Wed, Jun 10, 2015 at 10:57 PM, Kiran Ayyagari <ka...@apache.org>
wrote:
> On Thu, Jun 11, 2015 at 11:31 AM, Ezsra McDonald <ezsra.mcdonald@gmail.com
> >
> wrote:
>
> > So, I need to import 330k records to my directory in a short amount of
> > time. Is there a CLI that will do the job or is the Studio the only
> option?
> >
> > one easy way is to use ldapadd command, e.g
>
> ldapadd -H ldap://localhost:10389 -x -D "uid=admin,ou=system" -W -f
> mydata.ldif
>
> make sure that your LDIF file is sorted in parent entry first order.
>
> I am running 2.0.0 M19.
> >
> > Thanks,
> >
> > --EZ
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>
Re: ApacheDS Import via CLI
Posted by Kiran Ayyagari <ka...@apache.org>.
On Thu, Jun 11, 2015 at 11:31 AM, Ezsra McDonald <ez...@gmail.com>
wrote:
> So, I need to import 330k records to my directory in a short amount of
> time. Is there a CLI that will do the job or is the Studio the only option?
>
> one easy way is to use ldapadd command, e.g
ldapadd -H ldap://localhost:10389 -x -D "uid=admin,ou=system" -W -f
mydata.ldif
make sure that your LDIF file is sorted in parent entry first order.
I am running 2.0.0 M19.
>
> Thanks,
>
> --EZ
>
--
Kiran Ayyagari
http://keydap.com