You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Yuji Uchiyama (Created) (JIRA)" <ji...@apache.org> on 2011/12/04 01:42:39 UTC
[jira] [Created] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
------------------------------------------------------------------------------------------
Key: MATH-718
URL: https://issues.apache.org/jira/browse/MATH-718
Project: Commons Math
Issue Type: Bug
Affects Versions: 2.2, 3.0
Reporter: Yuji Uchiyama
The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
{{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
This returns 499525, though it should be 499999.
I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Sébastien Brisard (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252722#comment-13252722 ]
Sébastien Brisard commented on MATH-718:
----------------------------------------
Hi James,
thanks for your interest. STEM is a very interesting project!
I will try and find a fix for this issue as soon as possible. Any ideas are welcome!
Sébastien
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MATH-718)
inverseCumulativeProbability of BinomialDistribution returns wrong value
for large trials.
Posted by "Sébastien Brisard (Issue Comment Edited JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162326#comment-13162326 ]
Sébastien Brisard edited comment on MATH-718 at 12/5/11 6:53 AM:
-----------------------------------------------------------------
Hi Yuji,
thanks for reporting this. For version 3.0, we are currently reshaping the package distribution, and this will probably get resolved once we are over with MATH-692.
Best regards,
Sébastien
was (Author: celestin):
Hi Yuji,
thanks for reporting this. For version 3.0, we are currently reshaping the package distribution, and the next item on the list was inverseCumulativeProbability for discrete distributions (see MATH-699). So I'll gladly look into the reported bug.
As for version 2.2, I don't really know how to include the correction (if possible).
Sébastien
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Sébastien Brisard (Commented JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162326#comment-13162326 ]
Sébastien Brisard commented on MATH-718:
----------------------------------------
Hi Yuji,
thanks for reporting this. For version 3.0, we are currently reshaping the package distribution, and the next item on the list was inverseCumulativeProbability for discrete distributions (see MATH-699). So I'll gladly look into the reported bug.
As for version 2.2, I don't really know how to include the correction (if possible).
Sébastien
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279875#comment-13279875 ]
Thomas Neidhart commented on MATH-718:
--------------------------------------
I further looked into this with relation to MATH-785. First of all, in the original bug report, the reporter mentions that the expected result should be 499999 which is wrong, imho it should be 500000.
After implementing the modified Lentz-Thompson algorithm, the results for the BinomialDistribution of large trials show correct results.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253360#comment-13253360 ]
Thomas Neidhart commented on MATH-718:
--------------------------------------
The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:
{noformat}
public double cumulativeProbability(int x) {
double ret;
if (x < 0) {
ret = 0.0;
} else {
ret = Beta.regularizedBeta(probabilityOfSuccess,
numberOfSuccesses, x + 1);
}
return ret;
}
{noformat}
when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "James Kaufman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13252598#comment-13252598 ]
James Kaufman commented on MATH-718:
------------------------------------
I just wanted to let you know that our open source project (http://www.eclipse.org/stem/) needs this function and we are eagerly awaiting the update. We are experiencing the issue of wrong values for large trials.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MATH-718)
inverseCumulativeProbability of BinomialDistribution returns wrong value
for large trials.
Posted by "Thomas Neidhart (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253360#comment-13253360 ]
Thomas Neidhart edited comment on MATH-718 at 4/13/12 1:25 PM:
---------------------------------------------------------------
The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:
{noformat}
public double cumulativeProbability(int x) {
double ret;
if (x < 0) {
ret = 0.0;
} else {
ret = Beta.regularizedBeta(probabilityOfSuccess,
numberOfSuccesses, x + 1);
}
return ret;
}
{noformat}
when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, so it should be changed to something like "1L + x" to enforce a long addition.
Edit: Similar things happen btw also in other Distribution implementations, so it should be fixed also there, e.g. BinomialDistribution
was (Author: tn):
The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:
{noformat}
public double cumulativeProbability(int x) {
double ret;
if (x < 0) {
ret = 0.0;
} else {
ret = Beta.regularizedBeta(probabilityOfSuccess,
numberOfSuccesses, x + 1);
}
return ret;
}
{noformat}
when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.
Edit: Similar things happen btw also in other Distribution implementations, so it should be fixed also there, e.g. BinomialDistribution
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Neidhart updated MATH-718:
---------------------------------
Attachment: MATH-718.diff
The attached diff files shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.
Need to go through all failing tests and check the reason why they fail with this patch.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
> Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Neidhart updated MATH-718:
---------------------------------
Attachment: (was: MATH-718.diff)
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
> Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Neidhart updated MATH-718:
---------------------------------
Attachment: MATH-718.diff
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
> Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253381#comment-13253381 ]
Thomas Neidhart commented on MATH-718:
--------------------------------------
The problem is not only related to the Beta function, also the ContinuedFraction.evaluate is numerically unstable.
The reason the cumulativeProbability returns infinity instead of NaN is because the evaluate return 0.0 when called from Beta.regularizedBeta, which leads to a division by zero. The used default epsilon of 10e-15 seems also quite restrictive, when relaxing the epsilon I got much better results (e.g. with 10e-5 I got a result of 499997).
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Neidhart resolved MATH-718.
----------------------------------
Resolution: Fixed
Fix Version/s: (was: 4.0)
Fixed in r1341171.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Fix For: 3.1
>
> Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (MATH-718)
inverseCumulativeProbability of BinomialDistribution returns wrong value
for large trials.
Posted by "Thomas Neidhart (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253360#comment-13253360 ]
Thomas Neidhart edited comment on MATH-718 at 4/13/12 1:22 PM:
---------------------------------------------------------------
The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:
{noformat}
public double cumulativeProbability(int x) {
double ret;
if (x < 0) {
ret = 0.0;
} else {
ret = Beta.regularizedBeta(probabilityOfSuccess,
numberOfSuccesses, x + 1);
}
return ret;
}
{noformat}
when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.
Edit: Similar things happen btw also in other Distribution implementations, so it should be fixed also there, e.g. BinomialDistribution
was (Author: tn):
The problem Christian described wrt the PascalDistribution is a simple integer overflow in the class itself:
{noformat}
public double cumulativeProbability(int x) {
double ret;
if (x < 0) {
ret = 0.0;
} else {
ret = Beta.regularizedBeta(probabilityOfSuccess,
numberOfSuccesses, x + 1);
}
return ret;
}
{noformat}
when x = Integer.MAX_VALUE, adding 1 to it will result in an overflow. As the parameter of regularizedBeta is anyway a double, it should be cast to long/double before the addition.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Sébastien Brisard (Reopened JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sébastien Brisard reopened MATH-718:
------------------------------------
Sorry, I only meant to postpone this issue.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Sébastien Brisard (Resolved JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sébastien Brisard resolved MATH-718.
------------------------------------
Resolution: Later
Fix Version/s: 4.0
3.1
Assignee: (was: Sébastien Brisard)
As rightly pointed out by Christian, this issue is strongly related to MATH-738.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Fix For: 3.1, 4.0
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279877#comment-13279877 ]
Thomas Neidhart edited comment on MATH-718 at 5/20/12 10:05 PM:
----------------------------------------------------------------
The attached diff file shows the (preliminary) implementation of the modified Lentz-Thompson algorithm.
Edit: re-uploaded the diff file as it was broken.
Edit2: the failing unit tests I mentioned before were due to a wrong loop, the latest diff shows no unit test errors.
was (Author: tn):
The attached diff file shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.
Need to go through all failing tests and check the reason why they fail with this patch.
Edit: re-uploaded the diff file as it was broken.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
> Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Thomas Neidhart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279877#comment-13279877 ]
Thomas Neidhart edited comment on MATH-718 at 5/20/12 9:59 PM:
---------------------------------------------------------------
The attached diff file shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.
Need to go through all failing tests and check the reason why they fail with this patch.
Edit: re-uploaded the diff file as it was broken.
was (Author: tn):
The attached diff files shows the (preliminary) implementation of the modified Lentz-Thompson algorithm. There are in fact several unit tests that fail now, but this may be related to more accurate results. Relevant tests that use data from R or NIST datasets run successfully.
Need to go through all failing tests and check the reason why they fail with this patch.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
> Assignee: Sébastien Brisard
> Fix For: 3.1, 4.0
>
> Attachments: MATH-718.diff
>
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-718) inverseCumulativeProbability of
BinomialDistribution returns wrong value for large trials.
Posted by "Christian Winter (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170599#comment-13170599 ]
Christian Winter commented on MATH-718:
---------------------------------------
There seem to be stability problems with Beta.regularizedBeta(...) when using extreme parameters. {{PascalDistribution.cumulativeProbability(Integer.MAX_VALUE)}} returns {{NaN}} instead of 1. We should look for a way to avoid both infinite values and NaNs in the implementation of the regularized beta function.
> inverseCumulativeProbability of BinomialDistribution returns wrong value for large trials.
> ------------------------------------------------------------------------------------------
>
> Key: MATH-718
> URL: https://issues.apache.org/jira/browse/MATH-718
> Project: Commons Math
> Issue Type: Bug
> Affects Versions: 2.2, 3.0
> Reporter: Yuji Uchiyama
>
> The inverseCumulativeProbability method of the BinomialDistributionImpl class returns wrong value for large trials. Following code will be reproduce the problem.
> {{System.out.println(new BinomialDistributionImpl(1000000, 0.5).inverseCumulativeProbability(0.5));}}
> This returns 499525, though it should be 499999.
> I'm not sure how it should be fixed, but the cause is that the cumulativeProbability method returns Infinity, not NaN. As the result the checkedCumulativeProbability method doesn't work as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira