You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Bi...@sungard.com on 2008/08/18 16:28:17 UTC
Testing for field existence
Hello,
I am creating fields for documents like this:
String name = ...
String value = ...
doc.add(new Field(name, value, Field.Store.NO,
Field.Index.UN_TOKENIZED));
On the query side, sometimes I want to want to search for documents for
which a given field, say 'foo' is equal to a given value, say 'bar'.
E.g.
+foo:bar
This works.
But other times I just want to query to find all docuements for which a
given field exists regardless of its value. Is there a query that I can
do for this?
One thing I tried is to change my field creation statement like this:
doc.add(new Field(name, "XXXX" + value, Field.Store.NO,
Field.Index.UN_TOKENIZED));
then do a query like this:
+foo:XXXX*
I thought the "XXXX" at the beginning would be necessary since I read
that you cannot query with "*" at the beginning of your query string.
Anyway, it did not work. Is this because I'm storing the field
"UN_TOKENIZED"? Do I need to tokenize it to to be able to query against
it with wildcards, etc.?
Maybe there is a better more canonical way to do this? I'm still
pretty new to Lucene. Any help appreciated.
regards,
Bill
Re: Testing for field existence
Posted by Erick Erickson <er...@gmail.com>.
Karsten's saying that prefix and wildcard queries require a bunch of work.
Specifically, Lucene assembles a list of all terms that match the query and
then, conceptually at least, forms a huge OR query with one clause for
each term. Say, for instance, you have the following values indexed in a
field
in any number of documents.
aaaaa
aaaa
aaa
aa
a
Now you search for a*. Underneath the covers, Lucene behaves as if
you submitted a query
aaaaa OR aaaa OR aaa OR aa OR a
which can rapidly grow to a query of thousands of clauses. Avoiding this
is a good thing <G>.
Best
Erick
On Mon, Aug 18, 2008 at 12:58 PM, <Bi...@sungard.com> wrote:
>
> Karsten,
>
> Thanks for the feedback. Not sure I understand the reasoning behind not
> using the "XXXX" prefix (do you have a link possibly?). But I see what
> you are getting at with the additional field. I'll give it a try.
> Thanks for the help.
>
> regards,
>
> Bill
>
>
> -----Original Message-----
> From: Karsten F. [mailto:karsten-lucene@fiz-technik.de]
> Sent: Monday, August 18, 2008 11:45 AM
> To: java-user@lucene.apache.org
> Subject: Re: Testing for field existence
>
>
> Hi Bill,
>
> you should not use prefix-query (XXXX*), because in first step lucene
> would
> generate a list of all terms in this field, and than search for all this
> terms. Which is senceless.
>
> I would suggest to insert a new field "myFields" which contains as value
> the
> names of all fields for this document.
> So instead of
> +foo:XXXX*
> you would search for
> +myFields:foo
>
> This would be very fast.
>
> Best regards
>
> Karsten
>
>
> Bill.Chesky wrote:
> >
> > Hello,
> >
> > I am creating fields for documents like this:
> >
> >
> >
> > String name = ...
> >
> > String value = ...
> >
> > doc.add(new Field(name, value, Field.Store.NO,
> > Field.Index.UN_TOKENIZED));
> > ...
> >
> > One thing I tried is to change my field creation statement like this:
> >
> >
> > doc.add(new Field(name, "XXXX" + value, Field.Store.NO,
> > Field.Index.UN_TOKENIZED));
> >
> >
> > then do a query like this:
> >
> >
> > +foo:XXXX*
> >
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Testing-for-field-existence-tp19032939p19034359.ht
> ml<http://www.nabble.com/Testing-for-field-existence-tp19032939p19034359.html>
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Testing for field existence
Posted by Daniel Noll <da...@nuix.com>.
Karsten F. wrote:
> Hi Bill,
>
> you should not use prefix-query (XXXX*), because in first step lucene would
> generate a list of all terms in this field, and than search for all this
> terms. Which is senceless.
That's not quite an accurate description of what it does as it nowhere
near as slow as doing a query for each term one by one, but yes, it's
inefficient. You can use a PrefixFilter to do much the same thing in a
somewhat quicker fashion, although it still takes a little more time
than having a new field.
This may help if adding a new field is impractical (e.g. if you have to
support older text indexes.)
Daniel
--
Daniel Noll
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Testing for field existence
Posted by Bi...@sungard.com.
Karsten,
Thanks for the feedback. Not sure I understand the reasoning behind not
using the "XXXX" prefix (do you have a link possibly?). But I see what
you are getting at with the additional field. I'll give it a try.
Thanks for the help.
regards,
Bill
-----Original Message-----
From: Karsten F. [mailto:karsten-lucene@fiz-technik.de]
Sent: Monday, August 18, 2008 11:45 AM
To: java-user@lucene.apache.org
Subject: Re: Testing for field existence
Hi Bill,
you should not use prefix-query (XXXX*), because in first step lucene
would
generate a list of all terms in this field, and than search for all this
terms. Which is senceless.
I would suggest to insert a new field "myFields" which contains as value
the
names of all fields for this document.
So instead of
+foo:XXXX*
you would search for
+myFields:foo
This would be very fast.
Best regards
Karsten
Bill.Chesky wrote:
>
> Hello,
>
> I am creating fields for documents like this:
>
>
>
> String name = ...
>
> String value = ...
>
> doc.add(new Field(name, value, Field.Store.NO,
> Field.Index.UN_TOKENIZED));
> ...
>
> One thing I tried is to change my field creation statement like this:
>
>
> doc.add(new Field(name, "XXXX" + value, Field.Store.NO,
> Field.Index.UN_TOKENIZED));
>
>
> then do a query like this:
>
>
> +foo:XXXX*
>
>
>
>
--
View this message in context:
http://www.nabble.com/Testing-for-field-existence-tp19032939p19034359.ht
ml
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Testing for field existence
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Bill,
A simpler suggestion, assuming you need to test for the existence of just one particular field: rather than adding a field containing a list of all indexed fields for a particular document, as Karsten suggested, you could just add a field with a constant value when the field you want to test for is present for a particular document.
For example, if "foo" is a field you want to test for, and a document contains:
foo:bar
you could add a new field "foo.present", with a constant value, to the document:
foo.present:true
and then when querying:
+foo.present:true
Steve
On 08/18/2008 at 11:45 AM, Karsten F. wrote:
> Hi Bill,
>
> you should not use prefix-query (XXXX*), because in first
> step lucene would generate a list of all terms in this field,
> and than search for all this terms. Which is senceless.
>
> I would suggest to insert a new field "myFields" which
> contains as value the names of all fields for this document.
> So instead of
> +foo:XXXX*
> you would search for
> +myFields:foo
>
> This would be very fast.
>
> Best regards
>
> Karsten
>
> Bill.Chesky wrote:
> >
> > Hello,
> >
> > I am creating fields for documents like this:
> >
> > String name = ...
> >
> > String value = ...
> >
> > doc.add(new Field(name, value, Field.Store.NO, Field.Index.UN_TOKENIZED));
> > ...
> >
> > One thing I tried is to change my field creation statement like this:
> >
> > doc.add(new Field(name, "XXXX" + value, Field.Store.NO,
> > Field.Index.UN_TOKENIZED));
> >
> > then do a query like this:
> >
> >
> > +foo:XXXX*
> >
>
> -- View this message in context:
> http://www.nabble.com/Testing-for-field-existence-tp19032939p1
> 9034359.html Sent from the Lucene - Java Users mailing list archive at
> Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Testing for field existence
Posted by "Karsten F." <ka...@fiz-technik.de>.
Hi Bill,
you should not use prefix-query (XXXX*), because in first step lucene would
generate a list of all terms in this field, and than search for all this
terms. Which is senceless.
I would suggest to insert a new field "myFields" which contains as value the
names of all fields for this document.
So instead of
+foo:XXXX*
you would search for
+myFields:foo
This would be very fast.
Best regards
Karsten
Bill.Chesky wrote:
>
> Hello,
>
> I am creating fields for documents like this:
>
>
>
> String name = ...
>
> String value = ...
>
> doc.add(new Field(name, value, Field.Store.NO,
> Field.Index.UN_TOKENIZED));
> ...
>
> One thing I tried is to change my field creation statement like this:
>
>
> doc.add(new Field(name, "XXXX" + value, Field.Store.NO,
> Field.Index.UN_TOKENIZED));
>
>
> then do a query like this:
>
>
> +foo:XXXX*
>
>
>
>
--
View this message in context: http://www.nabble.com/Testing-for-field-existence-tp19032939p19034359.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org