use lucene as datastore?

[argh <ar...@v0.net>]

Hi,

I'm adding Lucene to an existing project where a daemon monitors a
frequently updated file system tree containing lots of expensive-to-parse
files for changes in order to keep cached metadata up to date about each
file.  (File writes unfortunately cannot be routed to allow for more
efficient change detection.)

Metadata is currently stored in a mirror directory tree as individual files
that are a trivial XML serialization of the same data that will soon be
indexed by Lucene.

I'm thus curious about the possibility eliminating the XML files altogether
and just using Lucene to store the metadata.  It seems like it could be a
big win on the complexity front.  My main concern lies with the time and
space efficiency of switching from implicit filename lookups to the
search-based model of "find the one document with the path field containing
/some/really/long/pathname".

This seems like a really common type of problem, but my searching didn't
turn up anything useful.  Pointers?  Thoughts?

Thanks...

-rg



-- 
View this message in context: http://www.nabble.com/use-lucene-as-datastore--tf4637962.html#a13246220
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: use lucene as datastore?

[Chris Lu <ch...@gmail.com>]

No experience on this. But there are two points I can think of:
1) you can use compressed field to store the text
2) use the hash code of the path as the key

-- 
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes

On 10/16/07, argh <ar...@v0.net> wrote:
>
>
> Hi,
>
> I'm adding Lucene to an existing project where a daemon monitors a
> frequently updated file system tree containing lots of expensive-to-parse
> files for changes in order to keep cached metadata up to date about each
> file.  (File writes unfortunately cannot be routed to allow for more
> efficient change detection.)
>
> Metadata is currently stored in a mirror directory tree as individual
> files
> that are a trivial XML serialization of the same data that will soon be
> indexed by Lucene.
>
> I'm thus curious about the possibility eliminating the XML files
> altogether
> and just using Lucene to store the metadata.  It seems like it could be a
> big win on the complexity front.  My main concern lies with the time and
> space efficiency of switching from implicit filename lookups to the
> search-based model of "find the one document with the path field
> containing
> /some/really/long/pathname".
>
> This seems like a really common type of problem, but my searching didn't
> turn up anything useful.  Pointers?  Thoughts?
>
> Thanks...
>
> -rg
>
>
>
> --
> View this message in context:
> http://www.nabble.com/use-lucene-as-datastore--tf4637962.html#a13246220
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>