You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zahra Aminolroaya <z....@gmail.com> on 2018/08/12 10:39:01 UTC
Docvalue v.s. invert index
Could we say that docvalue technique is better for sorting and faceting and
inverted index one is better for searching?
Will I lose anything if I only use docvalue?
Does docvalue technique have better performance?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Docvalue v.s. invert index
Posted by Shawn Heisey <el...@elyograg.org>.
On 8/13/2018 12:02 AM, Zahra Aminolroaya wrote:
> Thanks Erick, Shawn and Tomoko for complete answers.
>
> If I set both docvalue and indexed "true" in a field, will Solr understand
> to use which technique for faceting or searching? Or Is there any way to
> inform Solr to use which technique?
Solr will check the schema and the index to see what's available, and
use the best option it can find for whatever it's trying to do. As far
as I know, Solr doesn't offer any direct ways to force a particular data
structure for a given operation. There might be some *indirect* ways for
some things that I'm not aware of.
Thanks,
Shawn
Re: Docvalue v.s. invert index
Posted by Mikhail Khludnev <mk...@apache.org>.
org.apache.lucene.search.IndexOrDocValuesQuery might be also relevant.
On Mon, Aug 13, 2018 at 9:02 AM Zahra Aminolroaya <z....@gmail.com>
wrote:
> Thanks Erick, Shawn and Tomoko for complete answers.
>
> If I set both docvalue and indexed "true" in a field, will Solr understand
> to use which technique for faceting or searching? Or Is there any way to
> inform Solr to use which technique?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
--
Sincerely yours
Mikhail Khludnev
Re: Docvalue v.s. invert index
Posted by Erick Erickson <er...@gmail.com>.
"My expectation is that scanning Doc Values might be faster than inverted
index if a query matches more than %25 of documents."
I seriously doubt it. Or my expectations are really off base, which is
always possible, I confess I've never measured though. At a high
level:
indexed:
find entry for term_im_searching_for, read the list of IDs it
satisfies immediately
DocValues
if (doc1.val.equals(term_im_searching_for)) { add it to the list }
And if there was more than one clause you'd be scanning the DocValues
for each term.....
It'd be interesting to actually measure though.
Erick
On Mon, Aug 13, 2018 at 1:24 AM, Mikhail Khludnev <mk...@apache.org> wrote:
> https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/schema/FieldType.java#L881
>
> https://github.com/apache/lucene-solr/blob/17eb8cd14d27d2680fe7c4b3871f3eb883542d34/solr/core/src/java/org/apache/solr/search/facet/FacetField.java#L106
>
>
> On Mon, Aug 13, 2018 at 9:02 AM Zahra Aminolroaya <z....@gmail.com>
> wrote:
>
>> Thanks Erick, Shawn and Tomoko for complete answers.
>>
>> If I set both docvalue and indexed "true" in a field, will Solr understand
>> to use which technique for faceting or searching? Or Is there any way to
>> inform Solr to use which technique?
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
Re: Docvalue v.s. invert index
Posted by Mikhail Khludnev <mk...@apache.org>.
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/schema/FieldType.java#L881
https://github.com/apache/lucene-solr/blob/17eb8cd14d27d2680fe7c4b3871f3eb883542d34/solr/core/src/java/org/apache/solr/search/facet/FacetField.java#L106
On Mon, Aug 13, 2018 at 9:02 AM Zahra Aminolroaya <z....@gmail.com>
wrote:
> Thanks Erick, Shawn and Tomoko for complete answers.
>
> If I set both docvalue and indexed "true" in a field, will Solr understand
> to use which technique for faceting or searching? Or Is there any way to
> inform Solr to use which technique?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
--
Sincerely yours
Mikhail Khludnev
Re: Docvalue v.s. invert index
Posted by Zahra Aminolroaya <z....@gmail.com>.
Thanks Erick, Shawn and Tomoko for complete answers.
If I set both docvalue and indexed "true" in a field, will Solr understand
to use which technique for faceting or searching? Or Is there any way to
inform Solr to use which technique?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Docvalue v.s. invert index
Posted by Mikhail Khludnev <mk...@apache.org>.
My expectation is that scanning Doc Values might be faster than inverted
index if a query matches more than %25 of documents.
On Sun, Aug 12, 2018 at 7:59 PM Erick Erickson <er...@gmail.com>
wrote:
> bq. I have been informed that the performance of such a search is
> absolutely terrible.
>
> Yep. Horrible.
>
> These two structures answer completely different questions
> indexed - "for this word, what docs contain it in field X?"
> DocValues - "for this document, what is the value of field X?"
>
> On my, my usual examples are going out of date. "phone book" and
> "dictionary". There used to be, in the old days, these book-like
> things that were printed on actual paper and you could use them to
> find people's phone number and address, or what the meaning of a word
> was. Siiiiggghhhh.
>
> Well, get a paper phone book from somewhere off the shelf and consider
> each entry a "document", and the phone number and address the "text"
>
> DocValues answers "for person X, what is the phone number" easily, the
> whole thing is alphabetically arranged. But to answer the question
> "Who lives on Maple street" you have to read _everything_ in the
> entire phone book. Think "table scan".
>
> To answer the question "Who lives on Maple street", you want to index
> all the text.
>
> The whole point of docValues was that the structure that was used to
> answer the first question was built in the heap at runtime, consuming
> memory and CPU cycles. DocValues serialized that structure to disk at
> index time where it is
> 1> easily read as memory pages
> 2> almost entirely kept in MMapDirectory space, see:
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Best,
> Erick
>
>
> On Sun, Aug 12, 2018 at 8:56 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> > On 8/12/2018 4:39 AM, Zahra Aminolroaya wrote:
> >>
> >> Could we say that docvalue technique is better for sorting and faceting
> >> and
> >> inverted index one is better for searching?
> >
> >
> > Yes. That is how things work.
> >
> > If docValues do not exist, then an equivalent data structure must be
> built
> > in heap memory *from* the inverted index in order for faceting or
> sorting to
> > take place. When docValues are present, Solr can just read the data
> > directly instead of generating it. If there is plenty of spare memory
> for
> > the OS to cache data, this is faster. It also uses less Java heap
> memory.
> >
> >> Will I lose anything if I only use docvalue?
> >>
> >> Does docvalue technique have better performance?
> >
> >
> > From what I understand, it actually is possible to search when docValues
> are
> > present but the inverted index isn't, assuming that what you're searching
> > for is the full value of the field, not an individual word. I have been
> > informed that the performance of such a search is absolutely terrible.
> >
> > Thanks,
> > Shawn
> >
>
--
Sincerely yours
Mikhail Khludnev
Re: Docvalue v.s. invert index
Posted by Erick Erickson <er...@gmail.com>.
bq. I have been informed that the performance of such a search is
absolutely terrible.
Yep. Horrible.
These two structures answer completely different questions
indexed - "for this word, what docs contain it in field X?"
DocValues - "for this document, what is the value of field X?"
On my, my usual examples are going out of date. "phone book" and
"dictionary". There used to be, in the old days, these book-like
things that were printed on actual paper and you could use them to
find people's phone number and address, or what the meaning of a word
was. Siiiiggghhhh.
Well, get a paper phone book from somewhere off the shelf and consider
each entry a "document", and the phone number and address the "text"
DocValues answers "for person X, what is the phone number" easily, the
whole thing is alphabetically arranged. But to answer the question
"Who lives on Maple street" you have to read _everything_ in the
entire phone book. Think "table scan".
To answer the question "Who lives on Maple street", you want to index
all the text.
The whole point of docValues was that the structure that was used to
answer the first question was built in the heap at runtime, consuming
memory and CPU cycles. DocValues serialized that structure to disk at
index time where it is
1> easily read as memory pages
2> almost entirely kept in MMapDirectory space, see:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Best,
Erick
On Sun, Aug 12, 2018 at 8:56 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 8/12/2018 4:39 AM, Zahra Aminolroaya wrote:
>>
>> Could we say that docvalue technique is better for sorting and faceting
>> and
>> inverted index one is better for searching?
>
>
> Yes. That is how things work.
>
> If docValues do not exist, then an equivalent data structure must be built
> in heap memory *from* the inverted index in order for faceting or sorting to
> take place. When docValues are present, Solr can just read the data
> directly instead of generating it. If there is plenty of spare memory for
> the OS to cache data, this is faster. It also uses less Java heap memory.
>
>> Will I lose anything if I only use docvalue?
>>
>> Does docvalue technique have better performance?
>
>
> From what I understand, it actually is possible to search when docValues are
> present but the inverted index isn't, assuming that what you're searching
> for is the full value of the field, not an individual word. I have been
> informed that the performance of such a search is absolutely terrible.
>
> Thanks,
> Shawn
>
Re: Docvalue v.s. invert index
Posted by Shawn Heisey <ap...@elyograg.org>.
On 8/12/2018 4:39 AM, Zahra Aminolroaya wrote:
> Could we say that docvalue technique is better for sorting and faceting and
> inverted index one is better for searching?
Yes. That is how things work.
If docValues do not exist, then an equivalent data structure must be
built in heap memory *from* the inverted index in order for faceting or
sorting to take place. When docValues are present, Solr can just read
the data directly instead of generating it. If there is plenty of spare
memory for the OS to cache data, this is faster. It also uses less Java
heap memory.
> Will I lose anything if I only use docvalue?
>
> Does docvalue technique have better performance?
From what I understand, it actually is possible to search when
docValues are present but the inverted index isn't, assuming that what
you're searching for is the full value of the field, not an individual
word. I have been informed that the performance of such a search is
absolutely terrible.
Thanks,
Shawn
Re: Docvalue v.s. invert index
Posted by Tomoko Uchida <to...@gmail.com>.
> Could we say that docvalue technique is better for sorting and faceting
and inverted index one is better for searching?
The short answer is yes.
In addition, there are several special data structures for numeric/date
range/geo spatial search.
https://lucene.apache.org/solr/guide/7_4/field-types-included-with-solr.html
> Will I lose anything if I only use docvalue?
> Does docvalue technique have better performance?
I guess no one can answer to such too general question. If you have any
concrete problems/concerns, you should specify more details of that to get
good advices.
Regards,
Tomoko
2018年8月12日(日) 19:39 Zahra Aminolroaya <z....@gmail.com>:
> Could we say that docvalue technique is better for sorting and faceting and
> inverted index one is better for searching?
>
> Will I lose anything if I only use docvalue?
>
> Does docvalue technique have better performance?
>
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
--
Tomoko Uchida