Re: Most efficient way to index 14M documents (out of memory/file

[ma...@yahoo.co.uk]

Would it make more sense to use a parameter defining RAM size for the cache rather than  minMergeDocs?
Tuning RAM usage is the real issue here and controlling this by guessing the number of docs you can
squeeze into RAM is not the most helpful approach. How about a "setMaxCacheSize(int megabytes)" method?
I've found monitoring RAM usage with the following doesn't incur too much overhead:

public int getRAMSize() throws IOException
{
  String []segs=bulkLoadRAMDir.list();
  int totalSize=0;
  for(int i=0;i<segs.length;i++)
  {
    totalSize+=bulkLoadRAMDir.fileLength(segs[i]);
  }
  return totalSize;
}


Cheers
Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org