You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@directory.apache.org by Kiran Ayyagari <ka...@apache.org> on 2014/01/29 17:12:37 UTC
Re: parseLdifFile Run out of memory
On Wed, Jan 29, 2014 at 9:23 PM, eric tse <hf...@gmail.com> wrote:
> Hello
>
> I was using parseLdifFile to parse my LDIF file.
> When the LDIF size is small, then it is okay.
> But if the LDIF file size is 500M, with 1 million entries.. then it runs
> out of memory
> even I assign 6G memory to java heap
>
> Can you please tell me if I just don't give them enough memory (how much)
> or
> this library does not support large LDIF size (can you please point me out
> what library I should use) ?
>
> LdifReader reader = new LdifReader();
> List<LdifEntry> entries = reader.parseLdifFile("/home/myname/export.ldif"
> );
>
> you are parsing the entire file and keeping the entries in memory, do not
use this method,
instead iterate on the reader using reader.next(), this will fix your issue.
> It is the hash map throwing the out of memory exception.
>
>
> *Attached the exception*
>
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
> at java.util.HashMap.<init>(Unknown Source)
>
> at java.util.LinkedHashMap.<init>(Unknown Source)
>
> at java.util.HashSet.<init>(Unknown Source)
>
> at java.util.LinkedHashSet.<init>(Unknown Source)
>
> at
> org.apache.directory.api.ldap.model.entry.DefaultAttribute.<init>(Def
>
> aultAttribute.java:57)
>
> at
> org.apache.directory.api.ldap.model.entry.DefaultEntry.add(DefaultEnt
>
> ry.java:921)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifEntry.addAttribute(LdifE
>
> ntry.java:444)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader.parseAttributeVal
>
> ue(LdifReader.java:942)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader.parseEntry(LdifRe
>
> ader.java:1403)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader.nextInternal(Ldif
>
> Reader.java:1755)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader.access$100(LdifRe
>
> ader.java:168)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader$1.next(LdifReader
>
> .java:1859)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader$1.next(LdifReader
>
> .java:1850)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdif(LdifRea
>
> der.java:1911)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdifFile(Ldi
>
> fReader.java:1652)
>
> at
> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdifFile(Ldi
>
> fReader.java:1616)
>
> at
> public_register_not_enrolled.main(public_register_not_enrolled.java:5
>
> 0)
>
>
--
Kiran Ayyagari
http://keydap.com
Re: parseLdifFile Run out of memory
Posted by eric tse <hf...@gmail.com>.
Thanks it works like a charm. I am going to run it with larger data set now
:)
On Wed, Jan 29, 2014 at 4:00 PM, eric tse <hf...@gmail.com> wrote:
> Thanks will try now :)
>
>
> On Wed, Jan 29, 2014 at 3:53 PM, Stefan Seelmann <ma...@stefan-seelmann.de>wrote:
>
>> On 01/29/2014 06:38 PM, eric tse wrote:
>>
>> > Can you point me out any simple sample java code that demonstrate the
>> most
>> > basic concept of using "iterate on the reader using reader.next()"?
>>
>> LdifReader implements java.lang.Iteratable, so you can do the following
>> (omitting resource handling):
>>
>>
>> File file = new File("/path/to/your/file.ldif" );
>> LdifReader ldifReader = new LdifReader( file );
>> Iterator<LdifEntry> iterator = ldifReader.iterator();
>> while(iterator.hasNext()) {
>> LdifEntry ldifEntry = iterator.next();
>> System.out.println(ldifEntry);
>> }
>> ldifReader.close();
>>
>>
>> Or much nicer using Java 7 try-with-resource statement and foreach loop:
>>
>>
>> try ( LdifReader ldifReader = new LdifReader( file ) )
>> {
>> for ( LdifEntry ldifEntry : ldifReader )
>> {
>> System.out.println( ldifEntry );
>> }
>> }
>>
>>
>> HTH,
>> Stefan
>>
>>
>
Re: parseLdifFile Run out of memory
Posted by eric tse <hf...@gmail.com>.
Thanks will try now :)
On Wed, Jan 29, 2014 at 3:53 PM, Stefan Seelmann <ma...@stefan-seelmann.de>wrote:
> On 01/29/2014 06:38 PM, eric tse wrote:
>
> > Can you point me out any simple sample java code that demonstrate the
> most
> > basic concept of using "iterate on the reader using reader.next()"?
>
> LdifReader implements java.lang.Iteratable, so you can do the following
> (omitting resource handling):
>
>
> File file = new File("/path/to/your/file.ldif" );
> LdifReader ldifReader = new LdifReader( file );
> Iterator<LdifEntry> iterator = ldifReader.iterator();
> while(iterator.hasNext()) {
> LdifEntry ldifEntry = iterator.next();
> System.out.println(ldifEntry);
> }
> ldifReader.close();
>
>
> Or much nicer using Java 7 try-with-resource statement and foreach loop:
>
>
> try ( LdifReader ldifReader = new LdifReader( file ) )
> {
> for ( LdifEntry ldifEntry : ldifReader )
> {
> System.out.println( ldifEntry );
> }
> }
>
>
> HTH,
> Stefan
>
>
Re: parseLdifFile Run out of memory
Posted by Stefan Seelmann <ma...@stefan-seelmann.de>.
On 01/29/2014 06:38 PM, eric tse wrote:
> Can you point me out any simple sample java code that demonstrate the most
> basic concept of using "iterate on the reader using reader.next()"?
LdifReader implements java.lang.Iteratable, so you can do the following
(omitting resource handling):
File file = new File("/path/to/your/file.ldif" );
LdifReader ldifReader = new LdifReader( file );
Iterator<LdifEntry> iterator = ldifReader.iterator();
while(iterator.hasNext()) {
LdifEntry ldifEntry = iterator.next();
System.out.println(ldifEntry);
}
ldifReader.close();
Or much nicer using Java 7 try-with-resource statement and foreach loop:
try ( LdifReader ldifReader = new LdifReader( file ) )
{
for ( LdifEntry ldifEntry : ldifReader )
{
System.out.println( ldifEntry );
}
}
HTH,
Stefan
Re: parseLdifFile Run out of memory
Posted by eric tse <hf...@gmail.com>.
Thanks Kiran,
Can you point me out any simple sample java code that demonstrate the most
basic concept of using "iterate on the reader using reader.next()"?
Thanks and regards,
Eric
On Wed, Jan 29, 2014 at 9:12 AM, Kiran Ayyagari <ka...@apache.org>wrote:
>
>
>
> On Wed, Jan 29, 2014 at 9:23 PM, eric tse <hf...@gmail.com> wrote:
>
>> Hello
>>
>> I was using parseLdifFile to parse my LDIF file.
>> When the LDIF size is small, then it is okay.
>> But if the LDIF file size is 500M, with 1 million entries.. then it runs
>> out of memory
>> even I assign 6G memory to java heap
>>
>> Can you please tell me if I just don't give them enough memory (how much)
>> or
>> this library does not support large LDIF size (can you please point me
>> out what library I should use) ?
>>
>> LdifReader reader = new LdifReader();
>> List<LdifEntry> entries = reader.parseLdifFile(
>> "/home/myname/export.ldif");
>>
>> you are parsing the entire file and keeping the entries in memory, do not
> use this method,
> instead iterate on the reader using reader.next(), this will fix your
> issue.
>
>> It is the hash map throwing the out of memory exception.
>>
>>
>> *Attached the exception*
>>
>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>>
>> at java.util.HashMap.<init>(Unknown Source)
>>
>> at java.util.LinkedHashMap.<init>(Unknown Source)
>>
>> at java.util.HashSet.<init>(Unknown Source)
>>
>> at java.util.LinkedHashSet.<init>(Unknown Source)
>>
>> at
>> org.apache.directory.api.ldap.model.entry.DefaultAttribute.<init>(Def
>>
>> aultAttribute.java:57)
>>
>> at
>> org.apache.directory.api.ldap.model.entry.DefaultEntry.add(DefaultEnt
>>
>> ry.java:921)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifEntry.addAttribute(LdifE
>>
>> ntry.java:444)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseAttributeVal
>>
>> ue(LdifReader.java:942)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseEntry(LdifRe
>>
>> ader.java:1403)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.nextInternal(Ldif
>>
>> Reader.java:1755)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.access$100(LdifRe
>>
>> ader.java:168)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader$1.next(LdifReader
>>
>> .java:1859)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader$1.next(LdifReader
>>
>> .java:1850)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdif(LdifRea
>>
>> der.java:1911)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdifFile(Ldi
>>
>> fReader.java:1652)
>>
>> at
>> org.apache.directory.api.ldap.model.ldif.LdifReader.parseLdifFile(Ldi
>>
>> fReader.java:1616)
>>
>> at
>> public_register_not_enrolled.main(public_register_not_enrolled.java:5
>>
>> 0)
>>
>>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>