You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by zzzzz shalev <zz...@yahoo.com> on 2006/03/08 14:53:22 UTC
1.4.3 and 64bit support? out of memory??
hi all,
i've been trying to load a 6GB index on linux (16GB RAM) but am having no success.
i wrote a program that allocates memory and it was able to allocate as much RAM as i requested (stopped at 12GB)
however
i am recieving the following stack trace:
JVMDUMP013I Processed Dump Event "uncaught", detail "java/lang/OutOfMemoryError".
Exception in thread "main" java.lang.OutOfMemoryError
at org.apache.lucene.index.TermInfosReader.readIndex(TermInfosReader.java:82)
at org.apache.lucene.index.TermInfosReader.<init>(TermInfosReader.java:45)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:112)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:89)
at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:118)
at org.apache.lucene.store.Lock$With.run(Lock.java:109)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:111)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:106)
at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:43)
when trying to load the indexes
any ideas????
thanks in advance,
---------------------------------
Yahoo! Mail
Bring photos to life! New PhotoMail makes sharing a breeze.
Re: 1.4.3 and 64bit support? out of memory??
Posted by zzzzz shalev <zz...@yahoo.com>.
yes,
100%
Dan Armbrust <da...@gmail.com> wrote:
zzzzz shalev wrote:
> hi all,
>
> i've been trying to load a 6GB index on linux (16GB RAM) but am having no success.
>
> i wrote a program that allocates memory and it was able to allocate as much RAM as i requested (stopped at 12GB)
>
Was your program that got up to 12GB of memory written in Java, and
using the same jvm with the same -Xmx settings as your lucene program?
Dan
--
****************************
Daniel Armbrust
Biomedical Informatics
Mayo Clinic Rochester
daniel.armbrust(at)mayo.edu
http://informatics.mayo.edu/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------
Yahoo! Mail
Bring photos to life! New PhotoMail makes sharing a breeze.
Re: 1.4.3 and 64bit support? out of memory??
Posted by Dan Armbrust <da...@gmail.com>.
zzzzz shalev wrote:
> hi all,
>
> i've been trying to load a 6GB index on linux (16GB RAM) but am having no success.
>
> i wrote a program that allocates memory and it was able to allocate as much RAM as i requested (stopped at 12GB)
>
Was your program that got up to 12GB of memory written in Java, and
using the same jvm with the same -Xmx settings as your lucene program?
Dan
--
****************************
Daniel Armbrust
Biomedical Informatics
Mayo Clinic Rochester
daniel.armbrust(at)mayo.edu
http://informatics.mayo.edu/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 1.4.3 and 64bit support? out of memory??
Posted by Daniel Noll <da...@nuix.com.au>.
zzzzz shalev wrote:
> hey chris,
>
> i will check and let you know just to make sure,
>
> basically i see the OS allocating memory (up to about 4GB) while
> loading the indexes to memory and then crashing on the
> TermInfosReader class. what i noticed was that the crash occured
> when lucene tried to create a Term array with the following code
>
> new Term[indexSize]
>
> i assume, since this is an array java was trying to allocate
> consecutive blocks in memory and this is hard to find , even in a 16
> GB RAM machine, especially since (if im not mistaken) indexSize here
> is the termEnum size (which in my case is rather large)
That's not exactly how memory works. When a program looks to allocate a
chunk of memory, the chunk is allocated from the virtual memory space.
In the case of Windows XP on a 32-bit machine, the maximum contiguous
virtual memory is somewhere just below 2GB in a best-case scenario
(usually it's more like 1.5GB) regardless of the amount of physical RAM.
In the case of a 64-bit machine, though, the virtual memory space is
much, much larger than your 16GB of RAM, so there should be no problem
allocating ridiculous amounts of memory (or for that matter, memory
mapping ridiculously large files to a byte buffer.)
It wasn't mentioned explicitly so it's probably worth checking... you
are using the 64-bit JVM, right? If you were still using the 32-bit
JVM, that would certainly exhibit this sort of behaviour.
Daniel
--
Daniel Noll
Nuix Pty Ltd
Suite 79, 89 Jones St, Ultimo NSW 2007, Australia Ph: +61 2 9280 0699
Web: http://www.nuix.com.au/ Fax: +61 2 9212 6902
This message is intended only for the named recipient. If you are not
the intended recipient you are notified that disclosing, copying,
distributing or taking any action in reliance on the contents of this
message or attachment is strictly prohibited.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: 1.4.3 and 64bit support? out of memory??
Posted by zzzzz shalev <zz...@yahoo.com>.
hey chris,
i will check and let you know just to make sure,
basically i see the OS allocating memory (up to about 4GB) while loading the indexes to memory and then crashing on the TermInfosReader class. what i noticed was that the crash occured when lucene tried to create a Term array with the following code
new Term[indexSize]
i assume, since this is an array java was trying to allocate consecutive blocks in memory and this is hard to find , even in a 16 GB RAM machine, especially since (if im not mistaken) indexSize here is the termEnum size (which in my case is rather large)
i will get back to you about the one liner, if you have any other thoughts id be extremely happy to hear them as this problem is a Major road block
thanks a million
Chris Hostetter <ho...@fucit.org> wrote:
: i am recieving the following stack trace:
:
: JVMDUMP013I Processed Dump Event "uncaught", detail "java/lang/OutOfMemoryError".
: Exception in thread "main" java.lang.OutOfMemoryError
: at org.apache.lucene.index.TermInfosReader.readIndex(TermInfosReader.java:82)
is it possible that parts of your application are eating up all of the
heap in your JVM before this exception is encountered? Possibly by
opening a the index many times without closing it?
More specifically, if you write a 4 line app that does nothing by open
your index and then close it again, do you get an OOM? ...
public class Main {
public static void main(String[] args) throws Exception {
Searcher s = new IndexSearcher("/your/index/path");
s.close();
}
}
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------
Yahoo! Mail
Bring photos to life! New PhotoMail makes sharing a breeze.
Re: 1.4.3 and 64bit support? out of memory??
Posted by Chris Hostetter <ho...@fucit.org>.
: i am recieving the following stack trace:
:
: JVMDUMP013I Processed Dump Event "uncaught", detail "java/lang/OutOfMemoryError".
: Exception in thread "main" java.lang.OutOfMemoryError
: at org.apache.lucene.index.TermInfosReader.readIndex(TermInfosReader.java:82)
is it possible that parts of your application are eating up all of the
heap in your JVM before this exception is encountered? Possibly by
opening a the index many times without closing it?
More specifically, if you write a 4 line app that does nothing by open
your index and then close it again, do you get an OOM? ...
public class Main {
public static void main(String[] args) throws Exception {
Searcher s = new IndexSearcher("/your/index/path");
s.close();
}
}
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org