You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by MyCoy Z <my...@gmail.com> on 2024/05/31 19:30:25 UTC

Lucene9.10 seemingly incorrect NormValue read from the index

Hello, Lucene Community:

After upgrading to Lucene9.10 from L9.8, I find some of our
testcases starting to fail.
After some debugging, I think it is related to incorrectly NormValues read
from the index.


The following screenshot shows the situation while computing and recording
the NormValue using L9.*10*
Here, the calculated normValue is 13107*6* for docId:5
[image: L910_NormBeforeWriteToIndex.png]

The following screenshot shows the situation while reading the normValue
from the index for the same docId using L9.*10*
Here, the normValue is 13107*5* for docId:5
So it seems that the read value is one less than the initial written value.
[image: L910_NormReadFromIndex.png]




I've not noticed similar behavior (no related testcase failures) for L9.4,
9.5, 9.7 and 9.8
As a reference, the following screenshot shows the situation while reading
the normValue from the index for the same docId using L9.*8*   Here, the
read value is consistent as the written value.
[image: L98_NormReadFromIndex.png]


I've not tried L9.9.X  So I'm not sure when the potential issue was
introduced.
Our code has not been changed in any way. I work in a private company, and
cannot share the testcases. I'm not sure if this always happens or just
under some specific scenarios.
But this indeed looks like a potential bug to me.

Any comment is appreciated.

Thanks
MyCoy

Re: Lucene9.10 seemingly incorrect NormValue read from the index

Posted by MyCoy Z <my...@gmail.com>.
Thanks Patrick, just find some time to try it.
It turns out that the Consumer you pointed out actually has the same value
produced by the similarity function:
[image: image.png]



On Mon, Jun 10, 2024 at 11:29 AM Patrick Zhai <zh...@gmail.com> wrote:

> Hi MyCoy,
>
> I suspect it is mutated somewhere after the IndexChain#finish (where your
> screenshot is) and before Lucene90NormsConsumer#writeValues (where we write
> it into a file).
> To verify that, could you do me a favor and inspect this line:
> https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90NormsConsumer.java#L157
> and see whether the number is the same as what producer produce? (131075)
>
> Patrick
>
> On Fri, May 31, 2024 at 9:48 PM MyCoy Z <my...@gmail.com> wrote:
>
>> Thanks,
>> but we don't use that "LuceneTestCase"
>>
>>
>> Moreover, a quick update, I've tried *L9.9.2*, I've observed the same
>> issue.
>>
>>
>>
>> On Fri, May 31, 2024 at 9:31 PM Zhang Chao <80...@qq.com> wrote:
>>
>>> Hi,
>>>
>>> The only thing I can guess is that it might be related to reorder. if
>>> your tests extends `LuceneTestCase`, you can try to set your
>>> IndexWriterConfig to disable it:
>>>
>>> iwc.setMergePolicy(newMergePolicy(random(), false));
>>>
>>> --
>>> Zhang Chao
>>>
>>> 2024年6月1日 03:30,MyCoy Z <my...@gmail.com> 写道:
>>>
>>> Hello, Lucene Community:
>>>
>>> After upgrading to Lucene9.10 from L9.8, I find some of our
>>> testcases starting to fail.
>>> After some debugging, I think it is related to incorrectly NormValues
>>> read from the index.
>>>
>>>
>>> The following screenshot shows the situation while computing and
>>> recording the NormValue using L9.*10*
>>> Here, the calculated normValue is 13107*6* for docId:5
>>> [image: L910_NormBeforeWriteToIndex.png]
>>>
>>> The following screenshot shows the situation while reading the normValue
>>> from the index for the same docId using L9.*10*
>>> Here, the normValue is 13107*5* for docId:5
>>> So it seems that the read value is one less than the initial written
>>> value.
>>> [image: L910_NormReadFromIndex.png]
>>>
>>>
>>>
>>>
>>> I've not noticed similar behavior (no related testcase failures) for
>>> L9.4, 9.5, 9.7 and 9.8
>>> As a reference, the following screenshot shows the situation while
>>> reading the normValue from the index for the same docId using L9.*8*
>>>  Here, the read value is consistent as the written value.
>>> [image: L98_NormReadFromIndex.png]
>>>
>>>
>>> I've not tried L9.9.X  So I'm not sure when the potential issue was
>>> introduced.
>>> Our code has not been changed in any way. I work in a private company,
>>> and cannot share the testcases. I'm not sure if this always happens or just
>>> under some specific scenarios.
>>> But this indeed looks like a potential bug to me.
>>>
>>> Any comment is appreciated.
>>>
>>> Thanks
>>> MyCoy
>>>
>>>
>>>
>>>
>>>
>>>
>>>

Re: Lucene9.10 seemingly incorrect NormValue read from the index

Posted by Patrick Zhai <zh...@gmail.com>.
Hi MyCoy,

I suspect it is mutated somewhere after the IndexChain#finish (where your
screenshot is) and before Lucene90NormsConsumer#writeValues (where we write
it into a file).
To verify that, could you do me a favor and inspect this line:
https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90NormsConsumer.java#L157
and see whether the number is the same as what producer produce? (131075)

Patrick

On Fri, May 31, 2024 at 9:48 PM MyCoy Z <my...@gmail.com> wrote:

> Thanks,
> but we don't use that "LuceneTestCase"
>
>
> Moreover, a quick update, I've tried *L9.9.2*, I've observed the same
> issue.
>
>
>
> On Fri, May 31, 2024 at 9:31 PM Zhang Chao <80...@qq.com> wrote:
>
>> Hi,
>>
>> The only thing I can guess is that it might be related to reorder. if
>> your tests extends `LuceneTestCase`, you can try to set your
>> IndexWriterConfig to disable it:
>>
>> iwc.setMergePolicy(newMergePolicy(random(), false));
>>
>> --
>> Zhang Chao
>>
>> 2024年6月1日 03:30,MyCoy Z <my...@gmail.com> 写道:
>>
>> Hello, Lucene Community:
>>
>> After upgrading to Lucene9.10 from L9.8, I find some of our
>> testcases starting to fail.
>> After some debugging, I think it is related to incorrectly NormValues
>> read from the index.
>>
>>
>> The following screenshot shows the situation while computing and
>> recording the NormValue using L9.*10*
>> Here, the calculated normValue is 13107*6* for docId:5
>> [image: L910_NormBeforeWriteToIndex.png]
>>
>> The following screenshot shows the situation while reading the normValue
>> from the index for the same docId using L9.*10*
>> Here, the normValue is 13107*5* for docId:5
>> So it seems that the read value is one less than the initial written
>> value.
>> [image: L910_NormReadFromIndex.png]
>>
>>
>>
>>
>> I've not noticed similar behavior (no related testcase failures) for
>> L9.4, 9.5, 9.7 and 9.8
>> As a reference, the following screenshot shows the situation while
>> reading the normValue from the index for the same docId using L9.*8*
>>  Here, the read value is consistent as the written value.
>> [image: L98_NormReadFromIndex.png]
>>
>>
>> I've not tried L9.9.X  So I'm not sure when the potential issue was
>> introduced.
>> Our code has not been changed in any way. I work in a private company,
>> and cannot share the testcases. I'm not sure if this always happens or just
>> under some specific scenarios.
>> But this indeed looks like a potential bug to me.
>>
>> Any comment is appreciated.
>>
>> Thanks
>> MyCoy
>>
>>
>>
>>
>>
>>
>>

Re: Lucene9.10 seemingly incorrect NormValue read from the index

Posted by MyCoy Z <my...@gmail.com>.
Thanks,
but we don't use that "LuceneTestCase"


Moreover, a quick update, I've tried *L9.9.2*, I've observed the same
issue.



On Fri, May 31, 2024 at 9:31 PM Zhang Chao <80...@qq.com> wrote:

> Hi,
>
> The only thing I can guess is that it might be related to reorder. if your
> tests extends `LuceneTestCase`, you can try to set your IndexWriterConfig
> to disable it:
>
> iwc.setMergePolicy(newMergePolicy(random(), false));
>
> --
> Zhang Chao
>
> 2024年6月1日 03:30,MyCoy Z <my...@gmail.com> 写道:
>
> Hello, Lucene Community:
>
> After upgrading to Lucene9.10 from L9.8, I find some of our
> testcases starting to fail.
> After some debugging, I think it is related to incorrectly NormValues read
> from the index.
>
>
> The following screenshot shows the situation while computing and recording
> the NormValue using L9.*10*
> Here, the calculated normValue is 13107*6* for docId:5
> [image: L910_NormBeforeWriteToIndex.png]
>
> The following screenshot shows the situation while reading the normValue
> from the index for the same docId using L9.*10*
> Here, the normValue is 13107*5* for docId:5
> So it seems that the read value is one less than the initial written value.
> [image: L910_NormReadFromIndex.png]
>
>
>
>
> I've not noticed similar behavior (no related testcase failures) for L9.4,
> 9.5, 9.7 and 9.8
> As a reference, the following screenshot shows the situation while reading
> the normValue from the index for the same docId using L9.*8*   Here, the
> read value is consistent as the written value.
> [image: L98_NormReadFromIndex.png]
>
>
> I've not tried L9.9.X  So I'm not sure when the potential issue was
> introduced.
> Our code has not been changed in any way. I work in a private company, and
> cannot share the testcases. I'm not sure if this always happens or just
> under some specific scenarios.
> But this indeed looks like a potential bug to me.
>
> Any comment is appreciated.
>
> Thanks
> MyCoy
>
>
>
>
>
>
>

Re: Lucene9.10 seemingly incorrect NormValue read from the index

Posted by Zhang Chao <80...@qq.com.INVALID>.
Hi, 

The only thing I can guess is that it might be related to reorder. if your tests extends `LuceneTestCase`, you can try to set your IndexWriterConfig to disable it:

iwc.setMergePolicy(newMergePolicy(random(), false));

--
Zhang Chao

> 2024年6月1日 03:30,MyCoy Z <my...@gmail.com> 写道:
> 
> Hello, Lucene Community:
> 
> After upgrading to Lucene9.10 from L9.8, I find some of our testcases starting to fail.
> After some debugging, I think it is related to incorrectly NormValues read from the index.
> 
> 
> The following screenshot shows the situation while computing and recording the NormValue using L9.10  
> Here, the calculated normValue is 131076 for docId:5

> 
> The following screenshot shows the situation while reading the normValue from the index for the same docId using L9.10  
> Here, the normValue is 131075 for docId:5
> So it seems that the read value is one less than the initial written value.

> 
> 
> 
> 
> I've not noticed similar behavior (no related testcase failures) for L9.4, 9.5, 9.7 and 9.8
> As a reference, the following screenshot shows the situation while reading the normValue from the index for the same docId using L9.8   Here, the read value is consistent as the written value.

> 
> 
> I've not tried L9.9.X  So I'm not sure when the potential issue was introduced.
> Our code has not been changed in any way. I work in a private company, and cannot share the testcases. I'm not sure if this always happens or just under some specific scenarios.
> But this indeed looks like a potential bug to me.
> 
> Any comment is appreciated.
> 
> Thanks
> MyCoy
> 
> 
> 
> 
>