You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Erick Erickson <er...@gmail.com> on 2015/04/10 02:30:03 UTC

DocValues="true" and indexed="false"

So I was a bit embarrassed to be asked whether there was a use-case
for this. I did some simple tests on a field (with a magnificent total
of 32 docs indexed) and, with the exception of an error that
facet.mincount=0 doesn't work if a field isn't indexed (see
SOLR-5260), everything I tried worked fine. Searching by wildcards,
searching with the term query parser, faceting, grouping, whatever. I
admit I didn't spend very much time looking.

As I understand it DocValues are basically a serialized UnInverted
field so I'm wondering how searches work at all. Or, more
specifically, whether this works fine on small numbers of docs but
wouldn't scale. Or does a search on a DocValues field build an
inverted field?

Or anything else I should know. Is the rule simply 'if you search on
it, or use it in fq clauses, set indexed="true", and if you facet,
group etc. set indexed="true" '. So it would make sense to set
docValues="true" and indexed="false" on a field that's never searched
or used in an fq clause but used for faceting & etc.

So my mental model is "some operations need inverted fields, and some
need uninverted fields" and that docValues provide a way to store
uninverted fields on disk just like indexed="true" allows you to store
inverted fields on disk. Assuming you need both and set both
indexed="true" and docValues="true",  the _total_ memory requirements
for Solr are the same. What's NOT the same is that the docValues make
use of MMapDirectory where uninverting a field doesn't (this last is a
total guess).

I'm preparing a Google Doc that I'll certainly permit to anyone who
wants to add to it. I'll then add the results into the Reference
Guide.

Anyway, you can see I'm confused, but if I ask enough silly questions
eventually my questions get less silly.

Erick

Re: DocValues="true" and indexed="false"

Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Thu, 2015-04-09 at 17:30 -0700, Erick Erickson wrote:
> So I was a bit embarrassed to be asked whether there was a use-case
> for this. I did some simple tests on a field (with a magnificent total
> of 32 docs indexed) and, with the exception of an error that
> facet.mincount=0 doesn't work if a field isn't indexed (see
> SOLR-5260), everything I tried worked fine. Searching by wildcards,
> searching with the term query parser, faceting, grouping, whatever. I
> admit I didn't spend very much time looking.

The current fine-count for distributed faceting uses a lot of
mini-searches for facet:term, which requires the terms to be indexed (or
at least: Up until 20 minutes ago, I thought it required the term to be
indexed). However, as that implementation is quite slow, it would be
nice to change anyway.

I tried creating a small test and could not reproduce your findings: Not
indexed meant no search, DocValues or not. But you error message
indicates that you your field is indeed not indexed. Strange. Maybe you
can upload your test code somewhere?

> So my mental model is "some operations need inverted fields, and some
> need uninverted fields" and that docValues provide a way to store
> uninverted fields on disk just like indexed="true" allows you to store
> inverted fields on disk.

I like the idea of optional indexing of DV-fields.

- Toke Eskildsen, State and University Library, Denmark



Re: DocValues="true" and indexed="false"

Posted by "david.w.smiley@gmail.com" <da...@gmail.com>.
Yes, surprisingly enough, if indexed=false, docValues=true — you can still
search.  I’ve seen the code behind it; it’s interesting.  Rob wrote it.
I’m not sure how scalable it is compared to the inverted index.  I suspect
it wouldn’t do well for a lot of distinct values but will fine for a small
number of them.  What definition of “small” though… I don’t know.  I’d love
to see benchmarks of such a comparison.

~ David Smiley
Freelance Apache Lucene/Solr Search Consultant/Developer
http://www.linkedin.com/in/davidwsmiley

On Thu, Apr 9, 2015 at 8:30 PM, Erick Erickson <er...@gmail.com>
wrote:

> So I was a bit embarrassed to be asked whether there was a use-case
> for this. I did some simple tests on a field (with a magnificent total
> of 32 docs indexed) and, with the exception of an error that
> facet.mincount=0 doesn't work if a field isn't indexed (see
> SOLR-5260), everything I tried worked fine. Searching by wildcards,
> searching with the term query parser, faceting, grouping, whatever. I
> admit I didn't spend very much time looking.
>
> As I understand it DocValues are basically a serialized UnInverted
> field so I'm wondering how searches work at all. Or, more
> specifically, whether this works fine on small numbers of docs but
> wouldn't scale. Or does a search on a DocValues field build an
> inverted field?
>
> Or anything else I should know. Is the rule simply 'if you search on
> it, or use it in fq clauses, set indexed="true", and if you facet,
> group etc. set indexed="true" '. So it would make sense to set
> docValues="true" and indexed="false" on a field that's never searched
> or used in an fq clause but used for faceting & etc.
>
> So my mental model is "some operations need inverted fields, and some
> need uninverted fields" and that docValues provide a way to store
> uninverted fields on disk just like indexed="true" allows you to store
> inverted fields on disk. Assuming you need both and set both
> indexed="true" and docValues="true",  the _total_ memory requirements
> for Solr are the same. What's NOT the same is that the docValues make
> use of MMapDirectory where uninverting a field doesn't (this last is a
> total guess).
>
> I'm preparing a Google Doc that I'll certainly permit to anyone who
> wants to add to it. I'll then add the results into the Reference
> Guide.
>
> Anyway, you can see I'm confused, but if I ask enough silly questions
> eventually my questions get less silly.
>
> Erick
>