You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Olivier Binda <ol...@wanadoo.fr> on 2014/07/30 08:21:25 UTC
How to get the Terms from the TermsEnum of an IntField ?
Hello.
How do you get the terms for a TermsEnum of an IntField coded with
precisionStc = Integer.Max that you get with
MultiFields.getTerms(reader, intField).iterator(null) ?
I had mitigated success trying to get the terms out of this iterator
with NumericUtils.prefixCodedToInt
I tried
while (true) {
BytesRef ref = termsEnum?.next()
if (ref == null) break
int value = NumericUtils.prefixCodedToInt(ref)
}
But it doesn't work (reliably) because of the trie structure I guess
In an IntField with values 1,2,3,4,5 it worked
But in an Int Field with all values from 1 to 2500, I got exceptions :
lots of shifts aren't in the 0..31 range and it looks like there are
"Blocks" with :
first a term with shift 0 and value n
followed by lots of terms with shift that aren't in 0..31 but who share
the same prefix...
Best regards,
Olivier
Re: How to get the Terms from the TermsEnum of an IntField ?
Posted by Olivier Binda <ol...@wanadoo.fr>.
On 07/30/2014 11:31 AM, Michael McCandless wrote:
> If you indexed with precStep=Integer.MAX_VALUE then your index should
> only have the "shift 0" terms, I think.
>
> Can you boil the exception case (values 1 to 2500) down to a small test case?
The issue was the copyChars of the BytesRef class
/**
* Copies the UTF8 bytes for this string.
*
* @paramtextMust be well-formed unicode text, with no
* unpaired surrogates or invalid UTF16 code units.
*/
public voidcopyChars(CharSequence text) {
assertoffset==0;// TODO broken if offset != 0
UnicodeUtil.UTF16toUTF8(text,0,text.length(), this);
}
As it says in the code, it is broken when offset != 0.
Except that the assertion didn't work for me (I'm using Lucene from
Kotlin, there may be some inter-op issues as Kotlin is still cooking)
In Kotlin, I was reusing the BytesRef that was given to me by
termsEnum?.next()
to convert the BytesRef for the prefixed int into a BytesRef
representation of the int :
by doing
while (true) {
BytesRef ref = termsEnum.next()
if (ref == null) break
int shift = ref.bytes[ref.offset] -
NumericUtils.SHIFT_START_INT
if (shift > 31 || shift < 0) {} else {
ref.copyChars(NumericUtils.prefixCodedToInt(ref).toString())
}
}
Still, it would be nice if the copyChars method was fixed or at least if
the warning about copyChars being broken for offset != 0 was not only in
the code but also in the javaDoc.
Best regards,
Olivier
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, Jul 30, 2014 at 2:21 AM, Olivier Binda <ol...@wanadoo.fr> wrote:
>> Hello.
>>
>> How do you get the terms for a TermsEnum of an IntField coded with
>> precisionStc = Integer.Max that you get with
>> MultiFields.getTerms(reader, intField).iterator(null) ?
>>
>> I had mitigated success trying to get the terms out of this iterator with
>> NumericUtils.prefixCodedToInt
>>
>> I tried
>>
>> while (true) {
>> BytesRef ref = termsEnum?.next()
>> if (ref == null) break
>> int value = NumericUtils.prefixCodedToInt(ref)
>> }
>>
>> But it doesn't work (reliably) because of the trie structure I guess
>>
>> In an IntField with values 1,2,3,4,5 it worked
>> But in an Int Field with all values from 1 to 2500, I got exceptions :
>> lots of shifts aren't in the 0..31 range and it looks like there are
>> "Blocks" with :
>>
>> first a term with shift 0 and value n
>> followed by lots of terms with shift that aren't in 0..31 but who share the
>> same prefix...
>>
>> Best regards,
>> Olivier
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: How to get the Terms from the TermsEnum of an IntField ?
Posted by Olivier Binda <ol...@wanadoo.fr>.
On 07/30/2014 11:31 AM, Michael McCandless wrote:
> If you indexed with precStep=Integer.MAX_VALUE then your index should
> only have the "shift 0" terms, I think.
>
>
>
Damn :(
I was wondering if that was the case but I wasn't sure because of
(eventual compression and) the trie structure.
This means a bug somewhere (most likely in my code)
BTW, I'm indexing with 4.7.0 (im stuck on 4.7 because 4.8 require java 7
and try with resources which aren't (reliably) available before android 4.4)
> Can you boil the exception case (values 1 to 2500) down to a small test case?
I'll investigate my code and see what I can do (I'm building the index
after a huge build process with lots of computation) to bake a read only
index
Thanks for the quick answer.
Olivier
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, Jul 30, 2014 at 2:21 AM, Olivier Binda <ol...@wanadoo.fr> wrote:
>> Hello.
>>
>> How do you get the terms for a TermsEnum of an IntField coded with
>> precisionStc = Integer.Max that you get with
>> MultiFields.getTerms(reader, intField).iterator(null) ?
>>
>> I had mitigated success trying to get the terms out of this iterator with
>> NumericUtils.prefixCodedToInt
>>
>> I tried
>>
>> while (true) {
>> BytesRef ref = termsEnum?.next()
>> if (ref == null) break
>> int value = NumericUtils.prefixCodedToInt(ref)
>> }
>>
>> But it doesn't work (reliably) because of the trie structure I guess
>>
>> In an IntField with values 1,2,3,4,5 it worked
>> But in an Int Field with all values from 1 to 2500, I got exceptions :
>> lots of shifts aren't in the 0..31 range and it looks like there are
>> "Blocks" with :
>>
>> first a term with shift 0 and value n
>> followed by lots of terms with shift that aren't in 0..31 but who share the
>> same prefix...
>>
>> Best regards,
>> Olivier
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: How to get the Terms from the TermsEnum of an IntField ?
Posted by Michael McCandless <lu...@mikemccandless.com>.
If you indexed with precStep=Integer.MAX_VALUE then your index should
only have the "shift 0" terms, I think.
Can you boil the exception case (values 1 to 2500) down to a small test case?
Mike McCandless
http://blog.mikemccandless.com
On Wed, Jul 30, 2014 at 2:21 AM, Olivier Binda <ol...@wanadoo.fr> wrote:
> Hello.
>
> How do you get the terms for a TermsEnum of an IntField coded with
> precisionStc = Integer.Max that you get with
> MultiFields.getTerms(reader, intField).iterator(null) ?
>
> I had mitigated success trying to get the terms out of this iterator with
> NumericUtils.prefixCodedToInt
>
> I tried
>
> while (true) {
> BytesRef ref = termsEnum?.next()
> if (ref == null) break
> int value = NumericUtils.prefixCodedToInt(ref)
> }
>
> But it doesn't work (reliably) because of the trie structure I guess
>
> In an IntField with values 1,2,3,4,5 it worked
> But in an Int Field with all values from 1 to 2500, I got exceptions :
> lots of shifts aren't in the 0..31 range and it looks like there are
> "Blocks" with :
>
> first a term with shift 0 and value n
> followed by lots of terms with shift that aren't in 0..31 but who share the
> same prefix...
>
> Best regards,
> Olivier
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org