Directory objects for index

[Mark Kristensson <ma...@smartsheet.com>]

While implementing a solution for keeping warmed indexReaders around for our various indexes (so users don't have to wait while we open an indexReader for our one slow index), I've run into some serious problems trying to track down all of the outstanding file handles to my indexes.

For a while, I thought I was not properly closing my indexReader handles, however, I'm confident that I'm now doing that. So, the next most likely culprit are the directory objects. FYI, we use FSDirectory with a SingleInstanceLockFactory for all of our indexes right now.

I'm trying to decipher the behavior that I'm seeing, but the behavior seems to differ between my box (Mac OS X 10.6) and our lab machines (Linux) with how they handle and close the file handles from the Directory objects. 

Right now, we have several places in our code where we get a Directory object from the Java File object, including multiple times with each pass of our indexing service (which runs twice per minute). I'm wondering if I should instead use the static indexReader objects I'm now holding in a singleton and just call indexReader.directory() in each instance where we need a directory object (e.g. opening an indexWriter, checking to see if an index is locked, etc.).

A few elementary questions about the Directory (specifically FSDirectory) object:
1) Do I need to explicitly close directory objects? 
2) If I need to close them, when do I do it? Just after handing to IndexReader.open() or new IndexWriter()? Or should I close them  at the same time I close the IndexReader and IndexWriter objects?
3) Has anyone seem behavior that differs radically across OSes regarding references to open (and unclosed) FSDirectory objects?


Thanks,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Directory objects for index

[Michael McCandless <lu...@mikemccandless.com>]

The FSDirectory does not itself hold any open files.

All that its close method actually does is disable future operations against it.

Still, it is a good practice to close it (after you've closed all IRs
and IWs) since this can help you catch bugs in your app (ie, if you
hit AlreadyClosedException you'll know you're using a "stale" FSDir).

There are behavior differences eg b/w Windows & Unix, but OS X and
Linux ought to behave similarly.  Can you describe more details on
what you're seeing?

Mike

On Mon, Nov 29, 2010 at 8:57 PM, Mark Kristensson
<ma...@smartsheet.com> wrote:
> While implementing a solution for keeping warmed indexReaders around for our various indexes (so users don't have to wait while we open an indexReader for our one slow index), I've run into some serious problems trying to track down all of the outstanding file handles to my indexes.
>
> For a while, I thought I was not properly closing my indexReader handles, however, I'm confident that I'm now doing that. So, the next most likely culprit are the directory objects. FYI, we use FSDirectory with a SingleInstanceLockFactory for all of our indexes right now.
>
> I'm trying to decipher the behavior that I'm seeing, but the behavior seems to differ between my box (Mac OS X 10.6) and our lab machines (Linux) with how they handle and close the file handles from the Directory objects.
>
> Right now, we have several places in our code where we get a Directory object from the Java File object, including multiple times with each pass of our indexing service (which runs twice per minute). I'm wondering if I should instead use the static indexReader objects I'm now holding in a singleton and just call indexReader.directory() in each instance where we need a directory object (e.g. opening an indexWriter, checking to see if an index is locked, etc.).
>
> A few elementary questions about the Directory (specifically FSDirectory) object:
> 1) Do I need to explicitly close directory objects?
> 2) If I need to close them, when do I do it? Just after handing to IndexReader.open() or new IndexWriter()? Or should I close them  at the same time I close the IndexReader and IndexWriter objects?
> 3) Has anyone seem behavior that differs radically across OSes regarding references to open (and unclosed) FSDirectory objects?
>
>
> Thanks,
> Mark
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org