You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by st...@pattersoncompanies.com on 2011/03/21 18:09:53 UTC
RE: [Lucene.Net] Arithmetic Overflow
I'm not sure how to report this or if this is working as intended, however
using the recommended Search overload of IndexSearcher and passing
Int32.MaxValue, results in an arithmetic overflow exception in
PriorityQueue.Initialize(int maxSize).
I tested this out on several platforms, including Windows 7 - 32, 64,
Server 2003, Server 2008.
Perhaps the parameter that is passed to the Search method should be
updated to reflect a more realistic value?
Regards,
Steve Hoff
NOTICE: This email transmission and any attachments that accompany it may
contain information that is confidential or otherwise exempt from
disclosure under applicable law and is intended solely for the use of the
individual(s) to whom it was intended to be addressed. If you have
received this email by mistake, or you are not the intended recipient, any
disclosure, dissemination, distribution, copying or other use or retention
of this communication or its substance is prohibited. If you have received
this communication in error, please immediately report to the author via
email that you received this message by mistake and also permanently
destroy printed copies and delete the original and all copies of this
email and any attachments from your computer.
RE: [Lucene.Net] Arithmetic Overflow
Posted by Digy <di...@gmail.com>.
No, I don't think it is a bad design. It could be limited to some number,
but what would it be? 10,100,1000,1000000 or 100000000?
Besides that you have other alternatives to use while searching.
DIGY.
-----Original Message-----
From: steven.hoff@pattersoncompanies.com
[mailto:steven.hoff@pattersoncompanies.com]
Sent: Monday, March 21, 2011 9:16 PM
To: lucene-net-dev@lucene.apache.org
Subject: RE: [Lucene.Net] Arithmetic Overflow
While I understand that and thank you for taking the time to respond, my
concern was that passing a perfectly valid parameter leads to an
exception. So this is bad design.
Regards,
Steve Hoff
From: "Digy" <di...@gmail.com>
To: <lu...@lucene.apache.org>
Date: 03/21/2011 01:53 PM
Subject: RE: [Lucene.Net] Arithmetic Overflow
This is an expected behaviour. You get arithmetic overflow since an
array[MaxInt] is created in PriorityQueue to store the results.
(Lucene doesn't know the result-count before iterating over the results).
If you want to collect all results( say,not the top 10 or 100), you can
use
HitCollector.
DIGY
-----Original Message-----
From: steven.hoff@pattersoncompanies.com
[mailto:steven.hoff@pattersoncompanies.com]
Sent: Monday, March 21, 2011 7:10 PM
To: lucene-net-dev@lucene.apache.org
Subject: RE: [Lucene.Net] Arithmetic Overflow
I'm not sure how to report this or if this is working as intended, however
using the recommended Search overload of IndexSearcher and passing
Int32.MaxValue, results in an arithmetic overflow exception in
PriorityQueue.Initialize(int maxSize).
I tested this out on several platforms, including Windows 7 - 32, 64,
Server 2003, Server 2008.
Perhaps the parameter that is passed to the Search method should be
updated to reflect a more realistic value?
Regards,
Steve Hoff
NOTICE: This email transmission and any attachments that accompany it may
contain information that is confidential or otherwise exempt from
disclosure under applicable law and is intended solely for the use of the
individual(s) to whom it was intended to be addressed. If you have
received this email by mistake, or you are not the intended recipient, any
disclosure, dissemination, distribution, copying or other use or retention
of this communication or its substance is prohibited. If you have received
this communication in error, please immediately report to the author via
email that you received this message by mistake and also permanently
destroy printed copies and delete the original and all copies of this
email and any attachments from your computer.
NOTICE: This email transmission and any attachments that accompany it may
contain information that is confidential or otherwise exempt from
disclosure under applicable law and is intended solely for the use of the
individual(s) to whom it was intended to be addressed. If you have
received this email by mistake, or you are not the intended recipient, any
disclosure, dissemination, distribution, copying or other use or retention
of this communication or its substance is prohibited. If you have received
this communication in error, please immediately report to the author via
email that you received this message by mistake and also permanently
destroy printed copies and delete the original and all copies of this
email and any attachments from your computer.
Re: [Lucene.Net] Arithmetic Overflow
Posted by Troy Howard <th...@gmail.com>.
That's a good point. There are a lot of scenarios where exceptions
could still occur, even with parameter validation.
The difference is, in some situations, MaxInt/2 is a valid value.
However, Int32.MaxValue is never a valid value due to the internal
sorting logic requiring an additional array slot beyond the parameter
value. Another example is any negative value. Negative values are
always invalid for this parameter.
The parameter values should be checked for known 'always invalid'
conditions, and ArgumentException should be thrown explaining the
problem.
Thanks,
Troy
On Mon, Mar 21, 2011 at 12:58 PM, Digy <di...@gmail.com> wrote:
> Is -for ex- MaxInt/2 a valid value? You can get easily an OOM exception( and the worst case would be getting sporadic OOM when search fequency increases).
>
> DIGY
>
> -----Original Message-----
> From: Troy Howard [mailto:thoward37@gmail.com]
> Sent: Monday, March 21, 2011 9:43 PM
> To: lucene-net-dev@lucene.apache.org
> Subject: Re: [Lucene.Net] Arithmetic Overflow
>
> I agree that it should not throw an overflow exception. That's not
> very helpful for debugging.
>
> The valid range of that parameter is essentially [0...
> Int32.MaxValue-1].. That's because it needs one extra slot in it's
> internal array, beyond the max size for juggling values during sort.
> The exception is thrown when it tries to setup the array to be the
> parameter value +1 to accommodate that... So the best we could do
> would be to throw a nicer exception when someone passes an invalid
> value.
>
> Until a nicer exception can be worked into the code, the documentation
> should be updated to reflect that limit...
>
> Thanks,
> Troy
>
> On Mon, Mar 21, 2011 at 12:15 PM, <st...@pattersoncompanies.com> wrote:
>> While I understand that and thank you for taking the time to respond, my
>> concern was that passing a perfectly valid parameter leads to an
>> exception. So this is bad design.
>>
>>
>> Regards,
>> Steve Hoff
>>
>>
>>
>>
>> From: "Digy" <di...@gmail.com>
>> To: <lu...@lucene.apache.org>
>> Date: 03/21/2011 01:53 PM
>> Subject: RE: [Lucene.Net] Arithmetic Overflow
>>
>>
>>
>> This is an expected behaviour. You get arithmetic overflow since an
>> array[MaxInt] is created in PriorityQueue to store the results.
>> (Lucene doesn't know the result-count before iterating over the results).
>> If you want to collect all results( say,not the top 10 or 100), you can
>> use
>> HitCollector.
>>
>> DIGY
>>
>> -----Original Message-----
>> From: steven.hoff@pattersoncompanies.com
>> [mailto:steven.hoff@pattersoncompanies.com]
>> Sent: Monday, March 21, 2011 7:10 PM
>> To: lucene-net-dev@lucene.apache.org
>> Subject: RE: [Lucene.Net] Arithmetic Overflow
>>
>> I'm not sure how to report this or if this is working as intended, however
>>
>> using the recommended Search overload of IndexSearcher and passing
>> Int32.MaxValue, results in an arithmetic overflow exception in
>> PriorityQueue.Initialize(int maxSize).
>>
>> I tested this out on several platforms, including Windows 7 - 32, 64,
>> Server 2003, Server 2008.
>>
>> Perhaps the parameter that is passed to the Search method should be
>> updated to reflect a more realistic value?
>>
>> Regards,
>> Steve Hoff
>>
>> NOTICE: This email transmission and any attachments that accompany it may
>> contain information that is confidential or otherwise exempt from
>> disclosure under applicable law and is intended solely for the use of the
>> individual(s) to whom it was intended to be addressed. If you have
>> received this email by mistake, or you are not the intended recipient, any
>>
>> disclosure, dissemination, distribution, copying or other use or retention
>>
>> of this communication or its substance is prohibited. If you have received
>>
>> this communication in error, please immediately report to the author via
>> email that you received this message by mistake and also permanently
>> destroy printed copies and delete the original and all copies of this
>> email and any attachments from your computer.
>>
>>
>>
>> NOTICE: This email transmission and any attachments that accompany it may
>> contain information that is confidential or otherwise exempt from
>> disclosure under applicable law and is intended solely for the use of the
>> individual(s) to whom it was intended to be addressed. If you have
>> received this email by mistake, or you are not the intended recipient, any
>> disclosure, dissemination, distribution, copying or other use or retention
>> of this communication or its substance is prohibited. If you have received
>> this communication in error, please immediately report to the author via
>> email that you received this message by mistake and also permanently
>> destroy printed copies and delete the original and all copies of this
>> email and any attachments from your computer.
>>
>
>
RE: [Lucene.Net] Arithmetic Overflow
Posted by Digy <di...@gmail.com>.
Is -for ex- MaxInt/2 a valid value? You can get easily an OOM exception( and the worst case would be getting sporadic OOM when search fequency increases).
DIGY
-----Original Message-----
From: Troy Howard [mailto:thoward37@gmail.com]
Sent: Monday, March 21, 2011 9:43 PM
To: lucene-net-dev@lucene.apache.org
Subject: Re: [Lucene.Net] Arithmetic Overflow
I agree that it should not throw an overflow exception. That's not
very helpful for debugging.
The valid range of that parameter is essentially [0...
Int32.MaxValue-1].. That's because it needs one extra slot in it's
internal array, beyond the max size for juggling values during sort.
The exception is thrown when it tries to setup the array to be the
parameter value +1 to accommodate that... So the best we could do
would be to throw a nicer exception when someone passes an invalid
value.
Until a nicer exception can be worked into the code, the documentation
should be updated to reflect that limit...
Thanks,
Troy
On Mon, Mar 21, 2011 at 12:15 PM, <st...@pattersoncompanies.com> wrote:
> While I understand that and thank you for taking the time to respond, my
> concern was that passing a perfectly valid parameter leads to an
> exception. So this is bad design.
>
>
> Regards,
> Steve Hoff
>
>
>
>
> From: "Digy" <di...@gmail.com>
> To: <lu...@lucene.apache.org>
> Date: 03/21/2011 01:53 PM
> Subject: RE: [Lucene.Net] Arithmetic Overflow
>
>
>
> This is an expected behaviour. You get arithmetic overflow since an
> array[MaxInt] is created in PriorityQueue to store the results.
> (Lucene doesn't know the result-count before iterating over the results).
> If you want to collect all results( say,not the top 10 or 100), you can
> use
> HitCollector.
>
> DIGY
>
> -----Original Message-----
> From: steven.hoff@pattersoncompanies.com
> [mailto:steven.hoff@pattersoncompanies.com]
> Sent: Monday, March 21, 2011 7:10 PM
> To: lucene-net-dev@lucene.apache.org
> Subject: RE: [Lucene.Net] Arithmetic Overflow
>
> I'm not sure how to report this or if this is working as intended, however
>
> using the recommended Search overload of IndexSearcher and passing
> Int32.MaxValue, results in an arithmetic overflow exception in
> PriorityQueue.Initialize(int maxSize).
>
> I tested this out on several platforms, including Windows 7 - 32, 64,
> Server 2003, Server 2008.
>
> Perhaps the parameter that is passed to the Search method should be
> updated to reflect a more realistic value?
>
> Regards,
> Steve Hoff
>
> NOTICE: This email transmission and any attachments that accompany it may
> contain information that is confidential or otherwise exempt from
> disclosure under applicable law and is intended solely for the use of the
> individual(s) to whom it was intended to be addressed. If you have
> received this email by mistake, or you are not the intended recipient, any
>
> disclosure, dissemination, distribution, copying or other use or retention
>
> of this communication or its substance is prohibited. If you have received
>
> this communication in error, please immediately report to the author via
> email that you received this message by mistake and also permanently
> destroy printed copies and delete the original and all copies of this
> email and any attachments from your computer.
>
>
>
> NOTICE: This email transmission and any attachments that accompany it may
> contain information that is confidential or otherwise exempt from
> disclosure under applicable law and is intended solely for the use of the
> individual(s) to whom it was intended to be addressed. If you have
> received this email by mistake, or you are not the intended recipient, any
> disclosure, dissemination, distribution, copying or other use or retention
> of this communication or its substance is prohibited. If you have received
> this communication in error, please immediately report to the author via
> email that you received this message by mistake and also permanently
> destroy printed copies and delete the original and all copies of this
> email and any attachments from your computer.
>
Re: [Lucene.Net] Arithmetic Overflow
Posted by Troy Howard <th...@gmail.com>.
I agree that it should not throw an overflow exception. That's not
very helpful for debugging.
The valid range of that parameter is essentially [0...
Int32.MaxValue-1].. That's because it needs one extra slot in it's
internal array, beyond the max size for juggling values during sort.
The exception is thrown when it tries to setup the array to be the
parameter value +1 to accommodate that... So the best we could do
would be to throw a nicer exception when someone passes an invalid
value.
Until a nicer exception can be worked into the code, the documentation
should be updated to reflect that limit...
Thanks,
Troy
On Mon, Mar 21, 2011 at 12:15 PM, <st...@pattersoncompanies.com> wrote:
> While I understand that and thank you for taking the time to respond, my
> concern was that passing a perfectly valid parameter leads to an
> exception. So this is bad design.
>
>
> Regards,
> Steve Hoff
>
>
>
>
> From: "Digy" <di...@gmail.com>
> To: <lu...@lucene.apache.org>
> Date: 03/21/2011 01:53 PM
> Subject: RE: [Lucene.Net] Arithmetic Overflow
>
>
>
> This is an expected behaviour. You get arithmetic overflow since an
> array[MaxInt] is created in PriorityQueue to store the results.
> (Lucene doesn't know the result-count before iterating over the results).
> If you want to collect all results( say,not the top 10 or 100), you can
> use
> HitCollector.
>
> DIGY
>
> -----Original Message-----
> From: steven.hoff@pattersoncompanies.com
> [mailto:steven.hoff@pattersoncompanies.com]
> Sent: Monday, March 21, 2011 7:10 PM
> To: lucene-net-dev@lucene.apache.org
> Subject: RE: [Lucene.Net] Arithmetic Overflow
>
> I'm not sure how to report this or if this is working as intended, however
>
> using the recommended Search overload of IndexSearcher and passing
> Int32.MaxValue, results in an arithmetic overflow exception in
> PriorityQueue.Initialize(int maxSize).
>
> I tested this out on several platforms, including Windows 7 - 32, 64,
> Server 2003, Server 2008.
>
> Perhaps the parameter that is passed to the Search method should be
> updated to reflect a more realistic value?
>
> Regards,
> Steve Hoff
>
> NOTICE: This email transmission and any attachments that accompany it may
> contain information that is confidential or otherwise exempt from
> disclosure under applicable law and is intended solely for the use of the
> individual(s) to whom it was intended to be addressed. If you have
> received this email by mistake, or you are not the intended recipient, any
>
> disclosure, dissemination, distribution, copying or other use or retention
>
> of this communication or its substance is prohibited. If you have received
>
> this communication in error, please immediately report to the author via
> email that you received this message by mistake and also permanently
> destroy printed copies and delete the original and all copies of this
> email and any attachments from your computer.
>
>
>
> NOTICE: This email transmission and any attachments that accompany it may
> contain information that is confidential or otherwise exempt from
> disclosure under applicable law and is intended solely for the use of the
> individual(s) to whom it was intended to be addressed. If you have
> received this email by mistake, or you are not the intended recipient, any
> disclosure, dissemination, distribution, copying or other use or retention
> of this communication or its substance is prohibited. If you have received
> this communication in error, please immediately report to the author via
> email that you received this message by mistake and also permanently
> destroy printed copies and delete the original and all copies of this
> email and any attachments from your computer.
>
RE: [Lucene.Net] Arithmetic Overflow
Posted by st...@pattersoncompanies.com.
While I understand that and thank you for taking the time to respond, my
concern was that passing a perfectly valid parameter leads to an
exception. So this is bad design.
Regards,
Steve Hoff
From: "Digy" <di...@gmail.com>
To: <lu...@lucene.apache.org>
Date: 03/21/2011 01:53 PM
Subject: RE: [Lucene.Net] Arithmetic Overflow
This is an expected behaviour. You get arithmetic overflow since an
array[MaxInt] is created in PriorityQueue to store the results.
(Lucene doesn't know the result-count before iterating over the results).
If you want to collect all results( say,not the top 10 or 100), you can
use
HitCollector.
DIGY
-----Original Message-----
From: steven.hoff@pattersoncompanies.com
[mailto:steven.hoff@pattersoncompanies.com]
Sent: Monday, March 21, 2011 7:10 PM
To: lucene-net-dev@lucene.apache.org
Subject: RE: [Lucene.Net] Arithmetic Overflow
I'm not sure how to report this or if this is working as intended, however
using the recommended Search overload of IndexSearcher and passing
Int32.MaxValue, results in an arithmetic overflow exception in
PriorityQueue.Initialize(int maxSize).
I tested this out on several platforms, including Windows 7 - 32, 64,
Server 2003, Server 2008.
Perhaps the parameter that is passed to the Search method should be
updated to reflect a more realistic value?
Regards,
Steve Hoff
NOTICE: This email transmission and any attachments that accompany it may
contain information that is confidential or otherwise exempt from
disclosure under applicable law and is intended solely for the use of the
individual(s) to whom it was intended to be addressed. If you have
received this email by mistake, or you are not the intended recipient, any
disclosure, dissemination, distribution, copying or other use or retention
of this communication or its substance is prohibited. If you have received
this communication in error, please immediately report to the author via
email that you received this message by mistake and also permanently
destroy printed copies and delete the original and all copies of this
email and any attachments from your computer.
NOTICE: This email transmission and any attachments that accompany it may
contain information that is confidential or otherwise exempt from
disclosure under applicable law and is intended solely for the use of the
individual(s) to whom it was intended to be addressed. If you have
received this email by mistake, or you are not the intended recipient, any
disclosure, dissemination, distribution, copying or other use or retention
of this communication or its substance is prohibited. If you have received
this communication in error, please immediately report to the author via
email that you received this message by mistake and also permanently
destroy printed copies and delete the original and all copies of this
email and any attachments from your computer.
RE: [Lucene.Net] Arithmetic Overflow
Posted by Digy <di...@gmail.com>.
This is an expected behaviour. You get arithmetic overflow since an
array[MaxInt] is created in PriorityQueue to store the results.
(Lucene doesn't know the result-count before iterating over the results).
If you want to collect all results( say,not the top 10 or 100), you can use
HitCollector.
DIGY
-----Original Message-----
From: steven.hoff@pattersoncompanies.com
[mailto:steven.hoff@pattersoncompanies.com]
Sent: Monday, March 21, 2011 7:10 PM
To: lucene-net-dev@lucene.apache.org
Subject: RE: [Lucene.Net] Arithmetic Overflow
I'm not sure how to report this or if this is working as intended, however
using the recommended Search overload of IndexSearcher and passing
Int32.MaxValue, results in an arithmetic overflow exception in
PriorityQueue.Initialize(int maxSize).
I tested this out on several platforms, including Windows 7 - 32, 64,
Server 2003, Server 2008.
Perhaps the parameter that is passed to the Search method should be
updated to reflect a more realistic value?
Regards,
Steve Hoff
NOTICE: This email transmission and any attachments that accompany it may
contain information that is confidential or otherwise exempt from
disclosure under applicable law and is intended solely for the use of the
individual(s) to whom it was intended to be addressed. If you have
received this email by mistake, or you are not the intended recipient, any
disclosure, dissemination, distribution, copying or other use or retention
of this communication or its substance is prohibited. If you have received
this communication in error, please immediately report to the author via
email that you received this message by mistake and also permanently
destroy printed copies and delete the original and all copies of this
email and any attachments from your computer.