Keeping RAMDirectory and filesystem index in sync

["Satuluri, Venu_Madhav" <Ve...@deshaw.com>]

Hi,

Is there an elegant way to keep RAMDirectory and my file-system based
index in sync? I have a java class that is periodically started up by
crond that checks for modified documents and then reindexes them onto
the filesystem. However, for searching I want to use RAMDirectory (for
the performance benefits). The process that searches is alive all the
time (its in the backend of a webapp), and is different from the process
that indexes. 

Thanks,
Venu

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Keeping RAMDirectory and filesystem index in sync

[Chris Hostetter <ho...@fucit.org>]

: The Searching process then would have to re-open it's RAMDirectory.

the key to all of this being that there are constructors for RAMDirectory
that make it very easy to load in the contents of an FSDirectory.

: Or you check the version of the fs-based index from time to time, to see
: when it has changed. Using IndexReader.getCurrentVersion (path_to_index)
: this shouldn't be too expensive. But there's the risk that you re-open
: your RAMDirectory in the middle of an index update, since the version
: changes on every single modification done to the index.

I'm not certain if you are correct that the version changes on every
modification -- but even if it does, opening an index will only get the
changes as of the last "commit".  As long as you throttle how frequently
you re-open your RAMDirectory with some minimum "int timeSinceLastOpen",
and how often you check the version with some "int sleepBetweenVersionChecks"
this approach should work fine.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Keeping RAMDirectory and filesystem index in sync

[Jens Kraemer <kr...@webit.de>]

On Mon, Mar 13, 2006 at 06:23:10PM +0530, Satuluri, Venu_Madhav wrote:
> Hi,
> 
> Is there an elegant way to keep RAMDirectory and my file-system based
> index in sync? I have a java class that is periodically started up by
> crond that checks for modified documents and then reindexes them onto
> the filesystem. However, for searching I want to use RAMDirectory (for
> the performance benefits). The process that searches is alive all the
> time (its in the backend of a webapp), and is different from the process
> that indexes. 

You could somehow (HTTP, Database flag, some messaging system) ping your 
backend from your Indexing process, once it's finished. 
The Searching process then would have to re-open it's RAMDirectory. 

Or you check the version of the fs-based index from time to time, to see
when it has changed. Using IndexReader.getCurrentVersion (path_to_index)
this shouldn't be too expensive. But there's the risk that you re-open
your RAMDirectory in the middle of an index update, since the version 
changes on every single modification done to the index.

I don't know of any more elegant ways.

Jens



-- 
webit! Gesellschaft für neue Medien mbH          www.webit.de
Dipl.-Wirtschaftsingenieur Jens Krämer       kraemer@webit.de
Schnorrstraße 76                         Tel +49 351 46766  0
D-01069 Dresden                          Fax +49 351 46766 66

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org