You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Gimantha Bandara <gi...@wso2.com> on 2016/04/21 09:43:00 UTC
Re: Lucene 5.0.0 - StringField and Sorting
Hi Torsten,
Did you find a solution for this? I am having the same issue.. I am
planning to create a custom Field with DocValueType.SORTED. Is there any
other way to do that without creating a custom Field?
On Fri, Mar 6, 2015 at 3:34 PM, Torsten Krah <kr...@gmail.com> wrote:
> Hi,
>
> looking at the JavaDoc of StringField it says:
>
> /** A field that is indexed but not tokenized: the entire
> * String value is indexed as a single token. For example
> * this might be used for a 'country' field or an 'id'
> * field, or any field that you intend to use for sorting
> * or access through the field cache. */
>
> So i intend to use some StringFields for sorting.
> However trying to sort on them fails with:
>
> java.lang.IllegalStateException: unexpected docvalues type NONE for
> field 'NAME_KEYWORD' (expected=SORTED).
>
> Was indexed as StringField and Store.YES.
>
> So is the JavaDoc wrong here or is it correct and StringField should
> set:
>
> TYPE.setDocValuesType(DocValuesType.SORTED);
>
> so it would work?
>
> kind regards
>
> Torsten
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Gimantha Bandara
Software Engineer
WSO2. Inc : http://wso2.com
Mobile : +94714961919
RE: Lucene 5.0.0 - StringField and Sorting
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi,
> - Alternatively you can use UninvertingReader from the misc module and
> wrap your index reader. This "emulates" the DocValues APIs on indexes
> without DocValues. Internally it uses the old FieldCache code to do this. This
> has the same problems as earlier Lucene versions: Heavy heap usage and
> slow uninversion of field values. To use it you must wrap every segment of
> your index separately (e.g. by implementing FilterDirectoryReader) using
> https://lucene.apache.org/core/5_0_0/misc/org/apache/lucene/uninverting
> /UninvertingReader.html
Of course there is also the static factory UninvertingReader.wrap(DirectoryReader) to wrap any DirectoryReader for convenience.
Uwe
> > -----Original Message-----
> > From: Torsten Krah [mailto:krah.tm@gmail.com]
> > Sent: Monday, April 25, 2016 6:25 PM
> > To: Erick Erickson <er...@gmail.com>
> > Cc: java-user <ja...@lucene.apache.org>
> > Subject: Re: Lucene 5.0.0 - StringField and Sorting
> >
> > Hi Erick,
> >
> > i guess you've muddled the lists - this is lucenes one, not solr. I know
> > how to define it in solr but that wasn't the question as i am using pure
> > lucene and it did not work as expected from the javadocs there.
> >
> > Cheers
> >
> > Torsten
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene 5.0.0 - StringField and Sorting
Posted by Uwe Schindler <uw...@thetaphi.de>.
In Lucene you have to use the new DocValues fields (available since 4.0) to sort on.
For backwards compatibility, you can still use the older "uninversion" hack (also known as "Field Cache"). In Lucene 5 and 6 this part was removed from Lucene Core. To sort, you have 2 options:
- Reindex your stuff with DocValues fields enabled. This new field type is optimized for random access as used by sorting. This is the recommended approach.
- Alternatively you can use UninvertingReader from the misc module and wrap your index reader. This "emulates" the DocValues APIs on indexes without DocValues. Internally it uses the old FieldCache code to do this. This has the same problems as earlier Lucene versions: Heavy heap usage and slow uninversion of field values. To use it you must wrap every segment of your index separately (e.g. by implementing FilterDirectoryReader) using https://lucene.apache.org/core/5_0_0/misc/org/apache/lucene/uninverting/UninvertingReader.html
Erick: Solr has UninvertingReader used by default in SolrIndexSearcher.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Torsten Krah [mailto:krah.tm@gmail.com]
> Sent: Monday, April 25, 2016 6:25 PM
> To: Erick Erickson <er...@gmail.com>
> Cc: java-user <ja...@lucene.apache.org>
> Subject: Re: Lucene 5.0.0 - StringField and Sorting
>
> Hi Erick,
>
> i guess you've muddled the lists - this is lucenes one, not solr. I know
> how to define it in solr but that wasn't the question as i am using pure
> lucene and it did not work as expected from the javadocs there.
>
> Cheers
>
> Torsten
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 5.0.0 - StringField and Sorting
Posted by Torsten Krah <kr...@gmail.com>.
Hi Erick,
i guess you've muddled the lists - this is lucenes one, not solr. I know
how to define it in solr but that wasn't the question as i am using pure
lucene and it did not work as expected from the javadocs there.
Cheers
Torsten
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 5.0.0 - StringField and Sorting
Posted by Erick Erickson <er...@gmail.com>.
This works fine for me in a current Solr (5x). What version are you using?
Note that _any_ field you use for sorting _must_ be indexed="true" or, in
recent Solrs, docValues="true", just like any other field. You can't
sort on a field that
only has stored='"true" which is the only bit of the field definition
posted so far..
Best,
Erick
On Mon, Apr 25, 2016 at 6:13 AM, Torsten Krah <kr...@gmail.com> wrote:
> Hi,
>
> unfortunately i did not and no one did answered to this yet - although
> javadoc states it should work it does not.
> In the meantime i've used my own custom field but still i am interested
> on a solution to this.
>
> Lucene 6 still does have the same javadoc:
>
> https://lucene.apache.org/core/6_0_0/core/org/apache/lucene/document/StringField.html
>
> Cheers
>
> Torsten
>
>
> Am Donnerstag, den 21.04.2016, 13:13 +0530 schrieb Gimantha Bandara:
>> Hi Torsten,
>>
>>
>> Did you find a solution for this? I am having the same issue.. I am
>> planning to create a custom Field with DocValueType.SORTED. Is there
>> any other way to do that without creating a custom Field?
>>
>>
>> On Fri, Mar 6, 2015 at 3:34 PM, Torsten Krah <kr...@gmail.com>
>> wrote:
>> Hi,
>>
>> looking at the JavaDoc of StringField it says:
>>
>> /** A field that is indexed but not tokenized: the entire
>> * String value is indexed as a single token. For example
>> * this might be used for a 'country' field or an 'id'
>> * field, or any field that you intend to use for sorting
>> * or access through the field cache. */
>>
>> So i intend to use some StringFields for sorting.
>> However trying to sort on them fails with:
>>
>> java.lang.IllegalStateException: unexpected docvalues type
>> NONE for
>> field 'NAME_KEYWORD' (expected=SORTED).
>>
>> Was indexed as StringField and Store.YES.
>>
>> So is the JavaDoc wrong here or is it correct and StringField
>> should
>> set:
>>
>> TYPE.setDocValuesType(DocValuesType.SORTED);
>>
>> so it would work?
>>
>> kind regards
>>
>> Torsten
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>> java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail:
>> java-user-help@lucene.apache.org
>>
>>
>>
>>
>> --
>> Gimantha Bandara
>> Software Engineer
>> WSO2. Inc : http://wso2.com
>> Mobile : +94714961919
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 5.0.0 - StringField and Sorting
Posted by Torsten Krah <kr...@gmail.com>.
Hi,
unfortunately i did not and no one did answered to this yet - although
javadoc states it should work it does not.
In the meantime i've used my own custom field but still i am interested
on a solution to this.
Lucene 6 still does have the same javadoc:
https://lucene.apache.org/core/6_0_0/core/org/apache/lucene/document/StringField.html
Cheers
Torsten
Am Donnerstag, den 21.04.2016, 13:13 +0530 schrieb Gimantha Bandara:
> Hi Torsten,
>
>
> Did you find a solution for this? I am having the same issue.. I am
> planning to create a custom Field with DocValueType.SORTED. Is there
> any other way to do that without creating a custom Field?
>
>
> On Fri, Mar 6, 2015 at 3:34 PM, Torsten Krah <kr...@gmail.com>
> wrote:
> Hi,
>
> looking at the JavaDoc of StringField it says:
>
> /** A field that is indexed but not tokenized: the entire
> * String value is indexed as a single token. For example
> * this might be used for a 'country' field or an 'id'
> * field, or any field that you intend to use for sorting
> * or access through the field cache. */
>
> So i intend to use some StringFields for sorting.
> However trying to sort on them fails with:
>
> java.lang.IllegalStateException: unexpected docvalues type
> NONE for
> field 'NAME_KEYWORD' (expected=SORTED).
>
> Was indexed as StringField and Store.YES.
>
> So is the JavaDoc wrong here or is it correct and StringField
> should
> set:
>
> TYPE.setDocValuesType(DocValuesType.SORTED);
>
> so it would work?
>
> kind regards
>
> Torsten
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail:
> java-user-help@lucene.apache.org
>
>
>
>
> --
> Gimantha Bandara
> Software Engineer
> WSO2. Inc : http://wso2.com
> Mobile : +94714961919
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org