You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Grant Ingersoll <gs...@syr.edu> on 2003/12/05 15:48:36 UTC
Index and Field.Text
Hi,
I have seen the example SAX based XML processing in the Lucene sandbox (thanks to the authors for contributing!) and have successfully adapted this approach for my application. The one thing that does not sit well with me is the fact that I am using the method Field.Text(String, String) instead of the Field.Text(String, Reader) version, which means I am storing the contents in the index.
Some questions:
1. Should I care? What is the cost of storing the contents of these files versus using the Reader based method. Presumably, the index size is going to be larger, but will it adversaly effect search time? If yes, how much so (relatively speaking)?
2. If storing the content is going to adversaly effect searching, has anyone written an XMLReader that extends java.io.Reader. I guess it would need to take in the name of the tag(s) that you want the reader to retrieve and then extend all of the java.io.Reader results to return values based on just the tag values that I am interested in. Has anyone taken this approach? If not, does it at least seem like a valid approach?
Thanks for your help!
-Grant Ingersoll
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: FSDIrectory.create doesn't tolerate subdirectories
Posted by Morus Walter <mo...@tanto-xipolis.de>.
Erik Hatcher writes:
> On Sunday, December 7, 2003, at 08:21 PM, Esmond Pitt wrote:
> > I'm not clear whether this is a 'yes' or a 'no'.
>
> I think other committers would need to weigh in on it. I'm fine with
> making a change to check isDirectory as well and not deleting them
> since Lucene (currently) does not work with sub-directories. But I'm
> only one vote.
>
How do you grant, that a subdirectory does not (and will never) have the
same name as a file of the lucene index?
This may be unlikely, but if it happens, it will be a serious problem
if lucene cannot write a file, because there is some directory.
> > For application reasons I
> > would like to use a directory structure for indexes that mirrors the
> > collection structure of the site, so that there is a master index for
> > the
> > whole site and subindexes in subdirectories for each virtual site (and
> > so on
> > recursively while I am building subsubindexes for later consolidation).
>
You can still have your structure by adding a lucene-index dir at each
node.
So if your site has
/root/foo
/root/foo/bar
the index for documents in /root/foo goes to /indices/foo/lucene-index
and for /root/foo/bar to /indices/foo/bar/lucene-index.
I don't see why this is more complicated.
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: FSDIrectory.create doesn't tolerate subdirectories
Posted by Doug Cutting <cu...@lucene.com>.
I agree. One should provide Lucene with a unique path in the
filesystem, one that is not intended to be used for any other purpose.
All access to that path should be through Lucene's API. The fact that
Lucene decides to create a directory there rather than a single file is
an implementation detail.
This is an interesting case to consider. If Lucene created a single
file at the provided path which stored the entire index, would one then
be entitled to complain that one cannot make subdirectories of this file
for other purposes? Clearly not. Someday we may in fact wish to
implement Lucene that way.
Doug
Otis Gospodnetic wrote:
> I am against making the suggested Lucene modification.
> Lucene index structure may change in the future. It is possible that
> one day Lucene developers will need to use a hierarchy of directories
> to implement some feature.
>
> Therefore, Lucene users should be discouraged from creating
> sub-directories under the Lucene index directory, and Lucene should
> 'reserve the right' to change the index structure in the future.
>
> Otis
>
>
>
> --- Esmond Pitt <es...@motile.net> wrote:
>
>>Erik
>>
>>I'm not clear whether this is a 'yes' or a 'no'. For application
>>reasons I
>>would like to use a directory structure for indexes that mirrors the
>>collection structure of the site, so that there is a master index for
>>the
>>whole site and subindexes in subdirectories for each virtual site
>>(and so on
>>recursively while I am building subsubindexes for later
>>consolidation).
>>
>>I can't see any reason why Lucene should adopt a policy of preventing
>>me
>>from doing this & of forcing me into something more complicated.
>>Especially
>>when in reality this policy is merely a bug & very easily fixable.
>>
>>EJP
>>----- Original Message -----
>>From: "Erik Hatcher" <er...@ehatchersolutions.com>
>>To: "Lucene Users List" <lu...@jakarta.apache.org>
>>Sent: Monday, December 08, 2003 10:47 AM
>>Subject: Re: FSDIrectory.create doesn't tolerate subdirectories
>>
>>
>>
>>>On Sunday, December 7, 2003, at 06:17 PM, Esmond Pitt wrote:
>>>
>>>>When creating an index, FSDirectory assumes that the directory
>>
>>has no
>>
>>>>subdirectories. If a non-empty subdirectory is present,
>>>>FSDirectory.create
>>>>fails to delete it and throws an IOException. As the subdirectory
>>
>>is
>>
>>>>not a
>>>>Lucene index file (although in my case it is a Lucene sub-index),
>>
>>the
>>
>>>>method
>>>>actually has no business attempting to delete it at all. Can this
>>>>behaviour
>>>>please be changed so that it doesn't attempt to delete
>>
>>subdirectories
>>
>>>>in an
>>>>index location at all?
>>>
>>>This seems like an almost reasonable request, and easy enough to
>>>implement in FSDirectory.create. Lucene has no business deleting
>>
>>other
>>
>>>files in that directory that it doesn't use either, although that
>>
>>would
>>
>>>be a bit more difficult to implement I think.
>>>
>>>But really, it seems that users of Lucene should view the index
>>>directory as a black box and think of it as a single entity that
>>
>>Lucene
>>
>>>manages by itself. Why do you need to put things into that
>>
>>directory?
>>
>>>If you want to create multiple indexes, then I'd recommend you
>>
>>create a
>>
>>>parent directory to hold them in which the outside world to your
>>>application should view as a single entity that should not be
>>
>>touched
>>
>>>either.
>>>
>>>This is why the terminology in Lucene is FS*Directory* - let Lucene
>>
>>own
>>
>>>it I say.
>>>
>>>Erik
>>>
>>>
>>>
>>
>>---------------------------------------------------------------------
>>
>>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>>For additional commands, e-mail:
>>
>>lucene-user-help@jakarta.apache.org
>>
>>>
>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
>>For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>>
>
>
>
> __________________________________
> Do you Yahoo!?
> New Yahoo! Photos - easier uploading and sharing.
> http://photos.yahoo.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: FSDIrectory.create doesn't tolerate subdirectories
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I am against making the suggested Lucene modification.
Lucene index structure may change in the future. It is possible that
one day Lucene developers will need to use a hierarchy of directories
to implement some feature.
Therefore, Lucene users should be discouraged from creating
sub-directories under the Lucene index directory, and Lucene should
'reserve the right' to change the index structure in the future.
Otis
--- Esmond Pitt <es...@motile.net> wrote:
> Erik
>
> I'm not clear whether this is a 'yes' or a 'no'. For application
> reasons I
> would like to use a directory structure for indexes that mirrors the
> collection structure of the site, so that there is a master index for
> the
> whole site and subindexes in subdirectories for each virtual site
> (and so on
> recursively while I am building subsubindexes for later
> consolidation).
>
> I can't see any reason why Lucene should adopt a policy of preventing
> me
> from doing this & of forcing me into something more complicated.
> Especially
> when in reality this policy is merely a bug & very easily fixable.
>
> EJP
> ----- Original Message -----
> From: "Erik Hatcher" <er...@ehatchersolutions.com>
> To: "Lucene Users List" <lu...@jakarta.apache.org>
> Sent: Monday, December 08, 2003 10:47 AM
> Subject: Re: FSDIrectory.create doesn't tolerate subdirectories
>
>
> >
> > On Sunday, December 7, 2003, at 06:17 PM, Esmond Pitt wrote:
> > > When creating an index, FSDirectory assumes that the directory
> has no
> > > subdirectories. If a non-empty subdirectory is present,
> > > FSDirectory.create
> > > fails to delete it and throws an IOException. As the subdirectory
> is
> > > not a
> > > Lucene index file (although in my case it is a Lucene sub-index),
> the
> > > method
> > > actually has no business attempting to delete it at all. Can this
> > > behaviour
> > > please be changed so that it doesn't attempt to delete
> subdirectories
> > > in an
> > > index location at all?
> >
> > This seems like an almost reasonable request, and easy enough to
> > implement in FSDirectory.create. Lucene has no business deleting
> other
> > files in that directory that it doesn't use either, although that
> would
> > be a bit more difficult to implement I think.
> >
> > But really, it seems that users of Lucene should view the index
> > directory as a black box and think of it as a single entity that
> Lucene
> > manages by itself. Why do you need to put things into that
> directory?
> > If you want to create multiple indexes, then I'd recommend you
> create a
> > parent directory to hold them in which the outside world to your
> > application should view as a single entity that should not be
> touched
> > either.
> >
> > This is why the terminology in Lucene is FS*Directory* - let Lucene
> own
> > it I say.
> >
> > Erik
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
__________________________________
Do you Yahoo!?
New Yahoo! Photos - easier uploading and sharing.
http://photos.yahoo.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: FSDIrectory.create doesn't tolerate subdirectories
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sunday, December 7, 2003, at 08:21 PM, Esmond Pitt wrote:
> I'm not clear whether this is a 'yes' or a 'no'.
I think other committers would need to weigh in on it. I'm fine with
making a change to check isDirectory as well and not deleting them
since Lucene (currently) does not work with sub-directories. But I'm
only one vote.
> For application reasons I
> would like to use a directory structure for indexes that mirrors the
> collection structure of the site, so that there is a master index for
> the
> whole site and subindexes in subdirectories for each virtual site (and
> so on
> recursively while I am building subsubindexes for later consolidation).
Certainly seems reasonable. But I could also argue that you are
abusing Lucene a little since Lucene *owns* the directory, not you
directly.
> I can't see any reason why Lucene should adopt a policy of preventing
> me
> from doing this & of forcing me into something more complicated.
> Especially
> when in reality this policy is merely a bug & very easily fixable.
Careful now. Given the above argument, it is not a bug. Lucene views
the directory as an index. I doubt very seriously if there is anything
documented within Lucene that says that directory can be used for other
purposes.
I'm on the fence here. I don't view it as a "bug" in that it is
behavior against the design. It is working as designed, is my guess.
But I also agree with you that it would be nice if Lucene was kind
enough to allow you to work within that directory in areas it is not
(currently) concerned with. But, what about the future.... should
Lucene reserve the right to own that directory to make subdirectories
for a future use?
It is not a black and white area here, I don't think.
Feel free to formally file a bug report though - that is the official
way to make an issue known to the committers.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: FSDIrectory.create doesn't tolerate subdirectories
Posted by Esmond Pitt <es...@motile.net>.
Erik
I'm not clear whether this is a 'yes' or a 'no'. For application reasons I
would like to use a directory structure for indexes that mirrors the
collection structure of the site, so that there is a master index for the
whole site and subindexes in subdirectories for each virtual site (and so on
recursively while I am building subsubindexes for later consolidation).
I can't see any reason why Lucene should adopt a policy of preventing me
from doing this & of forcing me into something more complicated. Especially
when in reality this policy is merely a bug & very easily fixable.
EJP
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Monday, December 08, 2003 10:47 AM
Subject: Re: FSDIrectory.create doesn't tolerate subdirectories
>
> On Sunday, December 7, 2003, at 06:17 PM, Esmond Pitt wrote:
> > When creating an index, FSDirectory assumes that the directory has no
> > subdirectories. If a non-empty subdirectory is present,
> > FSDirectory.create
> > fails to delete it and throws an IOException. As the subdirectory is
> > not a
> > Lucene index file (although in my case it is a Lucene sub-index), the
> > method
> > actually has no business attempting to delete it at all. Can this
> > behaviour
> > please be changed so that it doesn't attempt to delete subdirectories
> > in an
> > index location at all?
>
> This seems like an almost reasonable request, and easy enough to
> implement in FSDirectory.create. Lucene has no business deleting other
> files in that directory that it doesn't use either, although that would
> be a bit more difficult to implement I think.
>
> But really, it seems that users of Lucene should view the index
> directory as a black box and think of it as a single entity that Lucene
> manages by itself. Why do you need to put things into that directory?
> If you want to create multiple indexes, then I'd recommend you create a
> parent directory to hold them in which the outside world to your
> application should view as a single entity that should not be touched
> either.
>
> This is why the terminology in Lucene is FS*Directory* - let Lucene own
> it I say.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: FSDIrectory.create doesn't tolerate subdirectories
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Sunday, December 7, 2003, at 06:17 PM, Esmond Pitt wrote:
> When creating an index, FSDirectory assumes that the directory has no
> subdirectories. If a non-empty subdirectory is present,
> FSDirectory.create
> fails to delete it and throws an IOException. As the subdirectory is
> not a
> Lucene index file (although in my case it is a Lucene sub-index), the
> method
> actually has no business attempting to delete it at all. Can this
> behaviour
> please be changed so that it doesn't attempt to delete subdirectories
> in an
> index location at all?
This seems like an almost reasonable request, and easy enough to
implement in FSDirectory.create. Lucene has no business deleting other
files in that directory that it doesn't use either, although that would
be a bit more difficult to implement I think.
But really, it seems that users of Lucene should view the index
directory as a black box and think of it as a single entity that Lucene
manages by itself. Why do you need to put things into that directory?
If you want to create multiple indexes, then I'd recommend you create a
parent directory to hold them in which the outside world to your
application should view as a single entity that should not be touched
either.
This is why the terminology in Lucene is FS*Directory* - let Lucene own
it I say.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
FSDIrectory.create doesn't tolerate subdirectories
Posted by Esmond Pitt <es...@motile.net>.
When creating an index, FSDirectory assumes that the directory has no
subdirectories. If a non-empty subdirectory is present, FSDirectory.create
fails to delete it and throws an IOException. As the subdirectory is not a
Lucene index file (although in my case it is a Lucene sub-index), the method
actually has no business attempting to delete it at all. Can this behaviour
please be changed so that it doesn't attempt to delete subdirectories in an
index location at all?
Applies to 1.3RC3.
EJP
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Index and Field.Text
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Friday 05 December 2003 10:45, Doug Cutting wrote:
> Tatu Saloranta wrote:
> > Also, shouldn't there be at least 3 methods that take Readers; one for
> > Text-like handling, another for UnStored, and last for UnIndexed.
>
> How do you store the contents of a Reader? You'd have to double-buffer
> it, first reading it into a String to store, and then tokenizing the
> StringReader. A key feature of Reader values is that they're streamed:
Not really, you can pass Reader to tokenizer, which then reads and tokenizes
directly (I think that's the way code also works). This because internally
String is read using StringReader, so passing a String looks more like a
convenience feature?
> the entire value is never in RAM. Storing a Reader value would remove
> that advantage. The current API makes this explicit: when you want
> something streamed, you pass in a Reader, when you're willing to have
> the entire value in memory, pass in a String.
I guess for things that are both tokenized and stored, passing a Reader can't
really help a lot; if one wants to reduce mem usage, text needs to be read
twice, or analyzer needs to help in writing output; or, text needs to be read
in-memory much like what happens now. It'd simplify application code a bit,
but wouldn't do much more.
So.... I guess I need to downgrade my suggestion to require just 2
Reader-taking factory methods? :-)
I still think that index-only and store-only version would both make sense. In
latter case, storing could be done in fully streaming fashion; in former
tokenization can be done?
> Yes, it is a bit confusing that Text(String, String) stores its value,
> while Text(String, Reader) does not, but it is at least well documented.
> And we cannot change it: that would break too many applications. But
> we can put this on the list for Lucene 2.0 cleanups.
Yes, I understand that. It'd not be reasonable to do such a change. But how
about adding more intuitive factory method (UnStored(String, Reader))?
> When I first wrote these static methods I meant for them to be
> constructor-like. I wanted to have multiple Field(String, String)
> constructors, but that's not possible, so I used capitalized static
> methods instead. I've never seen anyone else do this (capitalize any
> method but a real constructor) so I guess I didn't start a fad! This
:-)
> should someday too be cleaned up. Lucene was the first Java program
> that I ever wrote, and thus its style is in places non-standard. Sorry.
Best standards are created by people doing things others use, follow or
imitate... so it was worth a try! :-)
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Index and Field.Text
Posted by Doug Cutting <cu...@lucene.com>.
Tatu Saloranta wrote:
> Also, shouldn't there be at least 3 methods that take Readers; one for
> Text-like handling, another for UnStored, and last for UnIndexed.
How do you store the contents of a Reader? You'd have to double-buffer
it, first reading it into a String to store, and then tokenizing the
StringReader. A key feature of Reader values is that they're streamed:
the entire value is never in RAM. Storing a Reader value would remove
that advantage. The current API makes this explicit: when you want
something streamed, you pass in a Reader, when you're willing to have
the entire value in memory, pass in a String.
Yes, it is a bit confusing that Text(String, String) stores its value,
while Text(String, Reader) does not, but it is at least well documented.
And we cannot change it: that would break too many applications. But
we can put this on the list for Lucene 2.0 cleanups.
When I first wrote these static methods I meant for them to be
constructor-like. I wanted to have multiple Field(String, String)
constructors, but that's not possible, so I used capitalized static
methods instead. I've never seen anyone else do this (capitalize any
method but a real constructor) so I guess I didn't start a fad! This
should someday too be cleaned up. Lucene was the first Java program
that I ever wrote, and thus its style is in places non-standard. Sorry.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Index and Field.Text
Posted by Tatu Saloranta <ta...@hypermall.net>.
On Friday 05 December 2003 08:22, Erik Hatcher wrote:
> On Friday, December 5, 2003, at 09:48 AM, Grant Ingersoll wrote:
...
> > Field.Text(String, String) instead of the Field.Text(String, Reader)
> > version, which means I am storing the contents in the index.
>
> So use Field.UnStored(String, String) then. It is the same as
> Field.Text(String, Reader).
>
> The static "factory" methods on Field are merely for convenience. You
> can control all the flags yourself using the constructor:
I think it's almost a bug that they act differently, although having same
method name. I don't think method should be called Text() if it behaves like
UnStored()? Additionally, implementation for non-public constructor relies on
default values for isIndexed, isStored and isTokenized; it probably should
take those from static method for clarity?
Also, shouldn't there be at least 3 methods that take Readers; one for
Text-like handling, another for UnStored, and last for UnIndexed. It's
probably ok not to have one for keywords. For other types, though, it's often
more convenient to just pass in Reader.
(internally difference between passing in a Reader or String is not huge, as
String will be accessed via StringReader).
-+ Tatu +-
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Index and Field.Text
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Friday, December 5, 2003, at 09:48 AM, Grant Ingersoll wrote:
> I have seen the example SAX based XML processing in the Lucene sandbox
> (thanks to the authors for contributing!) and have successfully
> adapted this approach for my application. The one thing that does not
> sit well with me is the fact that I am using the method
> Field.Text(String, String) instead of the Field.Text(String, Reader)
> version, which means I am storing the contents in the index.
So use Field.UnStored(String, String) then. It is the same as
Field.Text(String, Reader).
The static "factory" methods on Field are merely for convenience. You
can control all the flags yourself using the constructor:
public Field(String name, String string,
boolean store, boolean index, boolean token)
> 2. If storing the content is going to adversaly effect searching, has
> anyone written an XMLReader that extends java.io.Reader.
You could always use a StringReader wrapper :))
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org