lucene and large (2GB+) indexes using RAMDirectory

[Jonathan Hager <jh...@gmail.com>]

Nate Denning encountered the following error when trying to load a
large (greater than 2147483647 bytes) index into a RAMDirectory.  The
server has 12GB of memory, so loading it into memory should not be a
problem.

Exception in thread "main" java.lang.NegativeArraySizeException
        at
org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:63)
        at
org.apache.lucene.store.RAMDirectory.<init>(RAMDirectory.java:89)
        at org.search.lucene.SearchEngine.search(SearchEngine.java:105)
        at org.search.lucene.SearchEngine.main(SearchEngine.java:52)
        
I helped modify the RAMDirectory the current dev branch as follows to
fix the error:

[Added]
private static final int MAX_BUFFER_SIZE = Integer.MAX_VALUE;

[Replaced]
int len = (int) is.length();
byte[] buf = new byte[len];
is.readBytes(buf, 0, len);
os.writeBytes(buf, len);

[with]
long len = is.length();
byte[] buf = len < MAX_BUFFER_SIZE ? new byte[(int) len] : new
byte[MAX_BUFFER_SIZE];
long bytesRemaining = len;
while (bytesRemaining > 0) {
       int bytesToRead = bytesRemaining < buf.length ? (int)
bytesRemaining : buf.length;
       is.readBytes(buf, 0, bytesToRead);
       os.writeBytes(buf, bytesToRead);
       bytesRemaining -= bytesToRead;
}

I first tested it with a smaller MAX_BUFFER_SIZE to make sure 
that it will work.  It still loaded a small index.

Nate is still trying to get it to load the large index, but is now
encountering memory allocation issues when trying to allocate a heap
larger than 2580MB.  I'll keep you updated, if we make any more
progress using lucene and a RAMDirectory for large indexes.

Jonathan

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org

Re: lucene and large (2GB+) indexes using RAMDirectory

[Michael Wechner <mi...@wyona.com>]

Doug Cutting wrote:

> Jonathan Hager wrote:
>
>> Nate Denning encountered the following error when trying to load a
>> large (greater than 2147483647 bytes) index into a RAMDirectory.  The
>> server has 12GB of memory, so loading it into memory should not be a
>> problem.
>
>
> Have you instead tried copying the index to a ramfs ('mount -t 
> ramfs'), then opening it with a normal FSDirectory?  This forces the 
> entire index into RAM without forcing it into Java's heap.  In my 
> experience, huge Java heaps are problematic.


another possibility might be to actually split up the index itself and 
run in more than one JVM and use a dispatcher in front of them, but this 
depends on your actual requirements on your index.

Michi

>
> Doug
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org

Re: lucene and large (2GB+) indexes using RAMDirectory

[Doug Cutting <cu...@apache.org>]

Jonathan Hager wrote:
> Nate Denning encountered the following error when trying to load a
> large (greater than 2147483647 bytes) index into a RAMDirectory.  The
> server has 12GB of memory, so loading it into memory should not be a
> problem.

Have you instead tried copying the index to a ramfs ('mount -t ramfs'), 
then opening it with a normal FSDirectory?  This forces the entire index 
into RAM without forcing it into Java's heap.  In my experience, huge 
Java heaps are problematic.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org