You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by "O'brien, Tim" <to...@transolutions.net> on 2003/05/26 16:29:51 UTC
Re: [math] Very legitimate reasons to draw up Presicison Unit Tests.
On Mon, 2003-05-26 at 18:27, Al Chou wrote:
> How about adding test cases for analytically exact scenarios that an algorithm
> should be able to get correct to within machine precision? I always had to
> start with those in my research.
I think we've found a good reference in the few certified results from
NIST. I'd be interested in knowing if there are other freely available
datasets. Al, you had previously submitted a list of resources, could
you do us the favor of submitting those as a patch for developer.xml?
Tim
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Very legitimate reasons to draw up Presicison Unit Tests.
Posted by "Mark R. Diggory" <md...@latte.harvard.edu>.
NIST is good for the Certified stats, but some things are not available
in the simple Univariate Cases, the sets I've seen only include mean,
std and an autocorrelation coef. These only cover a fraction of what
(y)our Univariate Implementations are capable of.
-Mark
O'brien, Tim wrote:
> On Mon, 2003-05-26 at 18:27, Al Chou wrote:
>
>>How about adding test cases for analytically exact scenarios that an algorithm
>>should be able to get correct to within machine precision? I always had to
>>start with those in my research.
>
>
> I think we've found a good reference in the few certified results from
> NIST. I'd be interested in knowing if there are other freely available
> datasets. Al, you had previously submitted a list of resources, could
> you do us the favor of submitting those as a patch for developer.xml?
>
> Tim
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Very legitimate reasons to draw up Presicison Unit Tests.
Posted by Phil Steitz <ph...@steitz.com>.
Mark R. Diggory wrote:
>
>
> Phil Steitz wrote:
>
>> O'brien, Tim wrote:
>>
>>> On Mon, 2003-05-26 at 18:27, Al Chou wrote:
>>>
>>>> How about adding test cases for analytically exact scenarios that an
>>>> algorithm
>>>> should be able to get correct to within machine precision? I always
>>>> had to
>>>> start with those in my research.
>>>
>>>
>>>
>>>
>>> I think we've found a good reference in the few certified results from
>>> NIST. I'd be interested in knowing if there are other freely available
>>> datasets. Al, you had previously submitted a list of resources, could
>>> you do us the favor of submitting those as a patch for developer.xml?
>>>
>>> Tim
>>
>>
>>
>> I agree that we need to pull the references list together. I will also
>> submit some patches for this.
>>
>> While I think that certified computations are good when we can do
>> them, I think the following are as important or more:
>>
>> * All implementations should be clearly documented, and wherever
>> possible we should use standard algorithms with well-known numerical
>> properties. The matrix LU decomposition discussion is an example of
>> this.
>>
>> * Support multiple algorithms when appropriate and let the users decide.
>> For example, we will likely want to add an alternative to Cholesky
>> decomposition and allow the user to choose.
>>
>> * Prefer verified computations to use in unit tests, but use whatever we
>> can get our hands on. It is 1000 times more important to verify that
>> the algorithm has been correctly implemented than to try to verify
>> empirically that our code + whatever the JDK does + special
>> characteristics of the data + our choice of algorithm = numbers
>> accurate to within some multiple of machine epsilon.
>>
>> Don't get me wrong -- I agree that numerical accuracy is very
>> important, but unfortumately numerical analysis is not an empirical
>> science. Analysis of algorithms, good functional testing and code
>> review are what will get us where we want to be, IMHO.
>>
>> Phil
>>
>
> I agree with everything you've said, but having datasets and expected
> results does make simply testing that the algorithm has been correctly
> implemented, yes they don't neccessarily need to be Certified. Just
> acurate to the current expectations of the algorithm.
>
Yeah, that's what I meant. My experience has been that most
implementation blunders result in *large* discrepancies which can be
caught with simple unit tests based on examples from books, web
examples, etc. or even examples computed using other software. I have
toyed with the idea of running some tests using R to get stats to
compare against.
> p.s. I am working on a Unit test for Univariate that should allow us to
> validate that algorithms for StoredUnivariate and Univariate.
>
This will be very useful, especially as we consider alternatives to the
simple formulas being used today.
> -Mark
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Very legitimate reasons to draw up Presicison Unit Tests.
Posted by "Mark R. Diggory" <md...@latte.harvard.edu>.
Phil Steitz wrote:
> O'brien, Tim wrote:
>
>> On Mon, 2003-05-26 at 18:27, Al Chou wrote:
>>
>>> How about adding test cases for analytically exact scenarios that an
>>> algorithm
>>> should be able to get correct to within machine precision? I always
>>> had to
>>> start with those in my research.
>>
>>
>>
>> I think we've found a good reference in the few certified results from
>> NIST. I'd be interested in knowing if there are other freely available
>> datasets. Al, you had previously submitted a list of resources, could
>> you do us the favor of submitting those as a patch for developer.xml?
>>
>> Tim
>
>
> I agree that we need to pull the references list together. I will also
> submit some patches for this.
>
> While I think that certified computations are good when we can do them,
> I think the following are as important or more:
>
> * All implementations should be clearly documented, and wherever
> possible we should use standard algorithms with well-known numerical
> properties. The matrix LU decomposition discussion is an example of
> this.
>
> * Support multiple algorithms when appropriate and let the users decide.
> For example, we will likely want to add an alternative to Cholesky
> decomposition and allow the user to choose.
>
> * Prefer verified computations to use in unit tests, but use whatever we
> can get our hands on. It is 1000 times more important to verify that
> the algorithm has been correctly implemented than to try to verify
> empirically that our code + whatever the JDK does + special
> characteristics of the data + our choice of algorithm = numbers
> accurate to within some multiple of machine epsilon.
>
> Don't get me wrong -- I agree that numerical accuracy is very important,
> but unfortumately numerical analysis is not an empirical science.
> Analysis of algorithms, good functional testing and code review are what
> will get us where we want to be, IMHO.
>
> Phil
>
I agree with everything you've said, but having datasets and expected
results does make simply testing that the algorithm has been correctly
implemented, yes they don't neccessarily need to be Certified. Just
acurate to the current expectations of the algorithm.
p.s. I am working on a Unit test for Univariate that should allow us to
validate that algorithms for StoredUnivariate and Univariate.
-Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Very legitimate reasons to draw up Presicison Unit Tests.
Posted by Phil Steitz <ph...@steitz.com>.
O'brien, Tim wrote:
> On Mon, 2003-05-26 at 18:27, Al Chou wrote:
>
>>How about adding test cases for analytically exact scenarios that an algorithm
>>should be able to get correct to within machine precision? I always had to
>>start with those in my research.
>
>
> I think we've found a good reference in the few certified results from
> NIST. I'd be interested in knowing if there are other freely available
> datasets. Al, you had previously submitted a list of resources, could
> you do us the favor of submitting those as a patch for developer.xml?
>
> Tim
>
I agree that we need to pull the references list together. I will also
submit some patches for this.
While I think that certified computations are good when we can do them,
I think the following are as important or more:
* All implementations should be clearly documented, and wherever
possible we should use standard algorithms with well-known numerical
properties. The matrix LU decomposition discussion is an example of
this.
* Support multiple algorithms when appropriate and let the users decide.
For example, we will likely want to add an alternative to Cholesky
decomposition and allow the user to choose.
* Prefer verified computations to use in unit tests, but use whatever we
can get our hands on. It is 1000 times more important to verify that
the algorithm has been correctly implemented than to try to verify
empirically that our code + whatever the JDK does + special
characteristics of the data + our choice of algorithm = numbers
accurate to within some multiple of machine epsilon.
Don't get me wrong -- I agree that numerical accuracy is very important,
but unfortumately numerical analysis is not an empirical science.
Analysis of algorithms, good functional testing and code review are what
will get us where we want to be, IMHO.
Phil
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Very legitimate reasons to draw up Presicison Unit Tests.
Posted by Al Chou <ho...@yahoo.com>.
--- "O'brien, Tim" <to...@transolutions.net> wrote:
> On Mon, 2003-05-26 at 18:27, Al Chou wrote:
> > How about adding test cases for analytically exact scenarios that an
> algorithm
> > should be able to get correct to within machine precision? I always had to
> > start with those in my research.
>
> I think we've found a good reference in the few certified results from
> NIST. I'd be interested in knowing if there are other freely available
> datasets. Al, you had previously submitted a list of resources, could
> you do us the favor of submitting those as a patch for developer.xml?
My wife needs the Mac where I've synch'ed to CVS tonight, but I'll try to synch
my work laptop via CygWin and submit the patch to developers.xml.
Al
=====
Albert Davidson Chou
Get answers to Mac questions at http://www.Mac-Mgrs.org/ .
__________________________________
Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org