How to find the ordinal for a numeric doc value

[tedsolr <ts...@sciquest.com>]

I'm trying to upgrade my custom post filter from Solr 4.9 to 5.2. This filter
collapses documents based on a user chosen field set. The key to the whole
thing is determining document uniqueness based on a fixed int array of field
value ordinals. In 4.9 this worked regardless of the field type. In the
collect() method of my delegating collector:

for (SortedDocValues vals : values) {
   ords[i++] = vals.getOrd(globalDoc);
}

The values come from FieldCache:

SortedDocValues docValues =
FieldCache.DEFAULT.getTermsIndex(searcher.getAtomicReader(), field);

I was not indexing using doc values. Now I'm trying to use doc values, and I
find that for non string fields I can't recover the ordinal for a field
value. At least I don't know how to get at it. So in my upgrade I index with
doc values, and the above code looks like:

SortedDocValues docValues =
searcher.getLeafReader().getSortedDocValues(field);

However this is not equivalent to the old 4.9 approach. My TrieDateField and
TrieDoubleField fields are being ignored (all field value ordinals are
treated as the same). I'm sure there is a good reason why SortedDocValues
exposes the backing dictionary and [Sorted]NumericDocValues does not. How do
I get to the ordinal for number and date fields?

I assume my fallback is to not index with doc values, and use an uninverting
reader to get the field data. Is there a better approach?



--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-find-the-ordinal-for-a-numeric-doc-value-tp4224018.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to find the ordinal for a numeric doc value

[tedsolr <ts...@sciquest.com>]

I see. The UninvertingReader even throws an IllegalStateException if you try
read a numeric field as a sorted doc values. I may have to index extra
fields to support my document collapsing scheme. Thanks for responding.



--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-find-the-ordinal-for-a-numeric-doc-value-tp4224018p4224255.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: How to find the ordinal for a numeric doc value

[Toke Eskildsen <te...@statsbiblioteket.dk>]

tedsolr <ts...@sciquest.com> wrote:
> I'm sure there is a good reason why SortedDocValues exposes
> the backing dictionary and [Sorted]NumericDocValues does not.

There is: Numerics does not have a backing dictionary. Instead of storing the values via the intermediate ordinals-map (aka by reference), they are stored directly. Overall it takes up less space and makes a lot of things easier (and a few things harder, as you have found out).

> How do I get to the ordinal for number and date fields?

You don't really. You will have to assign ordinals yourself by sorting all the existing values upon first call.

> I assume my fallback is to not index with doc values, and use an uninverting
> reader to get the field data. Is there a better approach?

You could index your integers as DocValued Strings, prefixed with zeroes to ensure same length and proper integer sort.

- Toke Eskildsen

Re: How to find the ordinal for a numeric doc value

[Mikhail Khludnev <mk...@griddynamics.com>]

Hello,
Giving the code
https://github.com/apache/lucene-solr/blob/trunk/solr/core/src/java/org/apache/solr/schema/TrieField.java#L727
it creates NumericDocValuesField only.
try to define field as multivalued, giving the code it creates
SortedSetDocValuesField.

On Wed, Aug 19, 2015 at 11:13 PM, tedsolr <ts...@sciquest.com> wrote:

> One error (others perhaps?) in my statement ... the code
>
> searcher.getLeafReader().getSortedDocValues(field)
>
> just returns null for numeric and date fields. That is why they appear to
> be
> ignored, not that the ordinals are all absent or equivalent. But my
> question
> is still valid I think!
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-find-the-ordinal-for-a-numeric-doc-value-tp4224018p4224037.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

<http://www.griddynamics.com>
<mk...@griddynamics.com>

Re: How to find the ordinal for a numeric doc value

[tedsolr <ts...@sciquest.com>]

One error (others perhaps?) in my statement ... the code

searcher.getLeafReader().getSortedDocValues(field)

just returns null for numeric and date fields. That is why they appear to be
ignored, not that the ordinals are all absent or equivalent. But my question
is still valid I think!



--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-find-the-ordinal-for-a-numeric-doc-value-tp4224018p4224037.html
Sent from the Solr - User mailing list archive at Nabble.com.