You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marc Sturlese <ma...@gmail.com> on 2010/06/18 18:45:52 UTC
performance sorting multivalued field
hey there!
can someone explain me how impacts to have multivalued fields when sorting?
I have read in other threads how does it affect when faceting but couldn't
find any info of the impact when sorting
Thanks in advance
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p905943.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: performance sorting multivalued field
Posted by Ya-Wen Hsu <yh...@eline.com>.
Hi,
I have sort on multivalued field with field collapse plugin. Solr always use the first value it gets from the search result when sorting multivalued fileds. I might be wrong but I vaguely remember it's the smallest value.
Wen
-----Original Message-----
From: Erik Hatcher [mailto:erik.hatcher@gmail.com]
Sent: Friday, June 18, 2010 10:32 AM
To: solr-user@lucene.apache.org
Subject: Re: performance sorting multivalued field
do you mean sorting facets? or sorting search results? you can't
sort search results by a multivalued field - which value would it use?
Erik
On Jun 18, 2010, at 12:45 PM, Marc Sturlese wrote:
>
> hey there!
> can someone explain me how impacts to have multivalued fields when
> sorting?
> I have read in other threads how does it affect when faceting but
> couldn't
> find any info of the impact when sorting
> Thanks in advance
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p905943.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance sorting multivalued field
Posted by Marc Sturlese <ma...@gmail.com>.
>>*There are lot's of docs with the same value, I mention that because I
supose that same value has nothing to do with the number of un-inverted term
instances.
It has to do, I've been able to reproduce teh error by setting different
values to each field:
HTTP Status 500 - there are more terms than documents in field "date", but
it's impossible to sort on tokenized fields java.lang.RuntimeException:
there are more terms than documents in field "id", but it's impossible to
sort on tokenized fields at
org.apache.lucene.search.FieldCacheImpl$StringIndexCache.createValue(FieldCacheImpl.java:706)...
But, it's already fixed for Lucene 2.9.4, 3.0.3, 3.1, 4.0 versions:
https://issues.apache.org/jira/browse/LUCENE-2142
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p921752.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance sorting multivalued field
Posted by Marc Sturlese <ma...@gmail.com>.
Thanks, that's very useful info. However can't reproduce the error. I've
created and index where all documents have a multivalued date field and each
document have a minimum of one value in that field. (most of the docs have 2
or 3). So, the number of un-inverted term instances is greater than
the number of documents.
*There are lot's of docs with the same value, I mention that because I
supose that same value has nothing to do with the number of un-inverted term
instances.
Never get the error explained here:
http://lucene.472066.n3.nabble.com/Different-sort-behavior-on-same-code-td503761.html
Could be that solr 1.4 or lucene 2.9.1 handle this avoiding the error?
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p920464.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance sorting multivalued field
Posted by wojtekpia <wo...@hotmail.com>.
Chris Hostetter-3 wrote:
>
> sorting on a multivalued is defined to have un-specified behavior. it
> might fail with an error, or it might fail silently.
>
I learned this the hard way, it failed silently for a long time until it
failed with an error:
http://lucene.472066.n3.nabble.com/Different-sort-behavior-on-same-code-td503761.html
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p920012.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance sorting multivalued field
Posted by Chris Hostetter <ho...@fucit.org>.
: I just like play with things. First checked the behavior of sorting on
: multiValued field and what I noticed was, let's say you have docs with field
sorting on a multivalued is defined to have un-specified behavior. it
might fail with an error, or it might fail silently.
fundementally solr can't sort on a multivvalued field, no matter how much
you might want it to, because if a doc contains the values "a" and "z"
then there is no deterministic way to decide where that document
should appear in an alphabetical list.
-Hoss
Re: performance sorting multivalued field
Posted by Erick Erickson <er...@gmail.com>.
Curiosity is good <G>. Do be aware, though, that the behavior is not
guaranteed,
it's just "how things happen to work" and may change without warning....
Erick
On Tue, Jun 22, 2010 at 4:01 AM, Marc Sturlese <ma...@gmail.com>wrote:
>
> >>Well, sorting requires that all the unique values in the target field
> >>get loaded into memory
> That's what I tought, thanks.
>
> >>But a larger question is whether what your doing is worthwhile
> >>even as just a measurement. You say
> >>"This is good for me, I don't care for my tests". I claim that
> >>you do care
> I just like play with things. First checked the behavior of sorting on
> multiValued field and what I noticed was, let's say you have docs with
> field
> called 'num':
> doc1->num:2;doc2->num:1,num:4;doc3->num:5
> Sorting by the field num what you get is:
> After sorting asc I get: doc2,doc1,doc3.
> The behavior seems to be always the same (I am not saying it works like
> that
> but it's what I've seen in my examples)
> After seeing that I just decided to check the performance. The point is
> simply curiosity.
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p913626.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: performance sorting multivalued field
Posted by Marc Sturlese <ma...@gmail.com>.
>>Well, sorting requires that all the unique values in the target field
>>get loaded into memory
That's what I tought, thanks.
>>But a larger question is whether what your doing is worthwhile
>>even as just a measurement. You say
>>"This is good for me, I don't care for my tests". I claim that
>>you do care
I just like play with things. First checked the behavior of sorting on
multiValued field and what I noticed was, let's say you have docs with field
called 'num':
doc1->num:2;doc2->num:1,num:4;doc3->num:5
Sorting by the field num what you get is:
After sorting asc I get: doc2,doc1,doc3.
The behavior seems to be always the same (I am not saying it works like that
but it's what I've seen in my examples)
After seeing that I just decided to check the performance. The point is
simply curiosity.
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p913626.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance sorting multivalued field
Posted by Erick Erickson <er...@gmail.com>.
Well, sorting requires that all the unique values in the target field
get loaded into memory, so it's possible that's the culprit. This
would likely only show up if you have a relatively small test set,
or the multivalued entries that you added are new to your corpus.
To test this stab in the dark, I'd start by counting the number of
unique terms in the field for the multi-valued and non-multi-valued
versions....
But a larger question is whether what your doing is worthwhile
even as just a measurement. You say
"This is good for me, I don't care for my tests". I claim that
you do care. You really can't trust measurements
of something when the behavior is undefined. Which the
behavior of sorting on mutivalued fields is. And it's measuring
something that you had better not be doing anyway in a real
application, so what's the point?
Best
Erick
On Sat, Jun 19, 2010 at 7:30 AM, Marc Sturlese <ma...@gmail.com>wrote:
>
> Hey Erik,
> I am currently sorting by a multiValued. It apears a feature tha't you may
> not know wich of the fields of the multiValued field makes the document be
> in that position. This is good for me, I don't care for my tests.
> What I need to know if there is any performance issue in all of this.
> Thanks
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p907502.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: performance sorting multivalued field
Posted by Marc Sturlese <ma...@gmail.com>.
Hey Erik,
I am currently sorting by a multiValued. It apears a feature tha't you may
not know wich of the fields of the multiValued field makes the document be
in that position. This is good for me, I don't care for my tests.
What I need to know if there is any performance issue in all of this.
Thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p907502.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance sorting multivalued field
Posted by Marc Sturlese <ma...@gmail.com>.
I mean sorting the query results, not facets.
I am asking because I have added a multivalued field that has as much 10
values. But 70% of the docs has just 1 or 2 fields of this multiValued
field. I am not doing faceting.
Since I have added the multiValued field, "java old gen" seems to get full
more quick and GC are happening more often.
I don't see why multiValued can use more memory querying by normal
relevance. That's why I think maybe it's sort queries fault...
Any explanation or advice?
Thanks in advance
--
View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p906115.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: performance sorting multivalued field
Posted by Erik Hatcher <er...@gmail.com>.
do you mean sorting facets? or sorting search results? you can't
sort search results by a multivalued field - which value would it use?
Erik
On Jun 18, 2010, at 12:45 PM, Marc Sturlese wrote:
>
> hey there!
> can someone explain me how impacts to have multivalued fields when
> sorting?
> I have read in other threads how does it affect when faceting but
> couldn't
> find any info of the impact when sorting
> Thanks in advance
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/performance-sorting-multivalued-field-tp905943p905943.html
> Sent from the Solr - User mailing list archive at Nabble.com.