You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by ps...@apache.org on 2004/01/14 08:36:35 UTC
cvs commit: jakarta-commons/math/xdocs tasks.xml
psteitz 2004/01/13 23:36:35
Modified: math/xdocs tasks.xml
Log:
Removed completed/obsolete tasks, added RealMatrix refactoring task.
Revision Changes Path
1.16 +8 -6 jakarta-commons/math/xdocs/tasks.xml
Index: tasks.xml
===================================================================
RCS file: /home/cvs/jakarta-commons/math/xdocs/tasks.xml,v
retrieving revision 1.15
retrieving revision 1.16
diff -u -r1.15 -r1.16
--- tasks.xml 15 Nov 2003 18:38:16 -0000 1.15
+++ tasks.xml 14 Jan 2004 07:36:35 -0000 1.16
@@ -34,11 +34,7 @@
<subsection name="Algorithm Development Tasks">
<p>These current tasks are planned and in need completion for the inital release.</p>
<dl>
- <dt>Add confidence intervals to Univariate implementations.</dt>
- <dd>Provide any comments on this task here.</dd>
- <dt>Distributions.</dt>
- <dd>Extend distribution framework to support discrete distributions and implement binomial and hypergeometric distributions.</dd>
- <dt>Analysis.</dt>
+ <dt>Root Finding.</dt>
<dd>
<ul>
<li>Rework unit tests for root finding and spline interpolation.</li>
@@ -64,6 +60,12 @@
</a></li>
</ul>
</dd>
+ <dt>RealMatrix.</dt>
+ <dd>
+ <ul>
+ <li>Refactor RealMatrix to minimally make solve() pluggable.</li>
+ </ul>
+ </dd>
</dl>
</subsection>
</section>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Re: cvs commit: jakarta-commons/math/xdocs tasks.xml
Posted by "Mark R. Diggory" <md...@latte.harvard.edu>.
Phil Steitz wrote:
> --- "Mark R. Diggory" <md...@latte.harvard.edu> wrote:
>
>>Hey Phil,
>>
>>I'm still planning to get around to some confidence interval features
>>(problem is determining the which approaches to determining conf are the
>>best to use in which situations).
>
>
> There is no generic solution that will work for arbitrary statistics,
> though bootstrap confidence intervals have wide applicability and might be
> OK to use as a default (though they are computationally expensive and would
> only work for the "stored" impls). I will think more about this.
>
Very true.
> Even if their not going into the
>
>>initial release, can we keep them around somewhere on the tasks or dev
>>page to give others ideas.
>
>
> Great. I removed the task that was there because it referred to adding
> confidence intervals to the Univariate interface, which no longer exists in
> its old form. Also, the ability to compute t-based confidence intervals
> exists in the code today.
>
> I will create a "post 0.1" section and add a task there if you don't do it
> first.
>
Sounds good to me too.
>
>>Also theres allot of solver framework stuff going on in the
>>"src/java/experimental" directory. This directory itself will not go
>>into the initial release, but some of the code in it is available as
>>possible approaches to pluggability of solvers/functions etc into the
>>rest of the api.
>>
>
>
> I would like to settle on a release plan for 0.1. Other than completing
> the user guide and refactoring RealMatrix, and a little code/test tidying,
> what else remains to be done? I will volunteer to play the release manager
> role if no one else wants to do it. In any case, I will help.
>
> Phil
>
From my other emails, you can see I was trying to generate
distributions using Maven. Robert pointed out some md5 and signature
stuff. I'll punt on that for a little while. I'm mostly interested in
getting the packaging process as automated as possible so the release
process is more "code oriented" and less of a packaging project.
-Mark
--
Mark Diggory
Software Developer
Harvard MIT Data Center
http://osprey.hmdc.harvard.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Re: bootstrap confidence intervals
Posted by Phil Steitz <ph...@steitz.com>.
Piotr,
Thanks for this patch. I like the simple boostrap impl, which could be
applied to any of our statistics. Pls do clean up and resubmit with
Apache License on each file. It might be easier to track if you attached
it to a Bugzilla enhancement request.
>
> Calculation of confidence intervals using bootstrap
> is a bit complicated, so I started with Standard Error -
> basically, one might calculate bootstrap CI using such SE,
> however this is often not good enough in practice.
Yes, but the bootstrap SE is valuable, nonetheless. I would like
eventually to include an impl of direct bootstrap confidence intervals,
which really means computing the bootstrap distribution.
>
> In order to calculate CI in a right way a fast algorithm
> for calculation of normal distribution function is
> needed, it is not present in commons-math (as far as
> I can remember I send some code doing this to
> a dev mailing list).
This depends on the statistic, of course. For means and regression
parameters, where the associated standard deviations are estimated from
the data, what you need is the t distribution, which we have. An example
is getSlopeConfidenceInterval in the BivariateRegression impl.
I will review the normal distribution patch, however, as I agree that this
would be a good addition in any case.
Phil
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[math] Re: bootstrap confidence intervals
Posted by Piotr Kochański <pi...@uw.edu.pl>.
Phil Steitz napisał(a):
> Excellent points. We would appreciate any other comments (or patches :-)
> that you have on the code or algorithms in [math].
Calculation of confidence intervals using bootstrap
is a bit complicated, so I started with Standard Error -
basically, one might calculate bootstrap CI using such SE,
however this is often not good enough in practice.
In order to calculate CI in a right way a fast algorithm
for calculation of normal distribution function is
needed, it is not present in commons-math (as far as
I can remember I send some code doing this to
a dev mailing list).
I attach classes, which calculate SE using bootstrap,
if you are interested in adding this code please let me know,
I will have to clean the code, javadoc and write tests.
All information about package organization, etc. would be
helpfull as well.
Here I put everything into o.a.c.math.stat.bootstrap
package.
The short description goes below:
Test class shows the usage of bootstrap to calculate
Standard Error.
Bootstrap class does actual resampling - it provides also
a method that returns an array of values of some
statistics calculated for every sample.
StandardError is an interface that is common for
all possible ways of calculating standard error.
MeanStandardError and BootstrapStandardError implements
this interface to calculate SE in a particular situation.
The usage is the follwing:
StandardError bse = new BootstrapStandardError();
((BootstrapStandardError)bse).setB(200);
bse.setStat(new Mean());
System.out.println("se for the mean (bootstrap): " +
bse.getStandardError(valSmall));
bse.setStat(new Median());
System.out.println("se for the median (bootstrap): " +
bse.getStandardError(valSmall));
StandardError se = new MeanStandardError();
se.setStat(new Mean());
System.out.println("se for the mean (standard formula): " +
se.getStandardError(valSmall));
System.out.println("---");
//this requieres patching the Mean class
Mean m = new Mean();
m.setStandardError(bse);
System.out.println("se for the mean (bootstrap thru Mean): " +
m.getStandardError(valSmall));
I would add Confidence Intervals calculation code in
a similar way, if this is ok. for you.
Regards
Piotr Kochanski
> --- Piotr Kochański <pi...@uw.edu.pl> wrote:
>
> > Another problem with bootstrap confidence intervals is that
> > they are non-parametric, and, inevitably, they provide less
> > power when doing statistical tests then any parametric method.
> >
> > Some people can be dissapointed with the fact that it is harder
> > to obtain significant results, so I think that usuall calculation method,
> > based on normal distribution assumption, should be also provided
> > (it can be done only for a few statistics though).
> >
> > There is also a "parametric" bootstrap, but this cannot be programmed
> > in a generic way, applicable in any situation.
> >
> > But still, bootstrap is very safe solution, given it's distribiution
> > independence
> > and ability to use it easyly for any statistics.
> >
> > Piotr Kochanski (pi@uw.edu.pl)
> >
>
>
>
>
> __________________________________
> Do you Yahoo!?
> Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
> http://hotjobs.sweepstakes.yahoo.com/signingbonus
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
Re: [math] bootstrap confidence intervals
Posted by Phil Steitz <st...@yahoo.com>.
Excellent points. We would appreciate any other comments (or patches :-)
that you have on the code or algorithms in [math].
--- Piotr Kocha�ski <pi...@uw.edu.pl> wrote:
> Another problem with bootstrap confidence intervals is that
> they are non-parametric, and, inevitably, they provide less
> power when doing statistical tests then any parametric method.
>
> Some people can be dissapointed with the fact that it is harder
> to obtain significant results, so I think that usuall calculation method,
> based on normal distribution assumption, should be also provided
> (it can be done only for a few statistics though).
>
> There is also a "parametric" bootstrap, but this cannot be programmed
> in a generic way, applicable in any situation.
>
> But still, bootstrap is very safe solution, given it's distribiution
> independence
> and ability to use it easyly for any statistics.
>
> Piotr Kochanski (pi@uw.edu.pl)
>
__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[math] bootstrap confidence intervals
Posted by Piotr Kochañski <pi...@uw.edu.pl>.
Hello,
Phil Steitz wrote:
> --- "Mark R. Diggory" <md...@latte.harvard.edu> wrote:
> > Hey Phil,
> >
> > I'm still planning to get around to some confidence interval features
> > (problem is determining the which approaches to determining conf are the
> > best to use in which situations).
>
> There is no generic solution that will work for arbitrary statistics,
> though bootstrap confidence intervals have wide applicability and might be
> OK to use as a default (though they are computationally expensive and would
> only work for the "stored" impls). I will think more about this.
Another problem with bootstrap confidence intervals is that
they are non-parametric, and, inevitably, they provide less
power when doing statistical tests then any parametric method.
Some people can be dissapointed with the fact that it is harder
to obtain significant results, so I think that usuall calculation method,
based on normal distribution assumption, should be also provided
(it can be done only for a few statistics though).
There is also a "parametric" bootstrap, but this cannot be programmed
in a generic way, applicable in any situation.
But still, bootstrap is very safe solution, given it's distribiution
independence
and ability to use it easyly for any statistics.
Piotr Kochanski (pi@uw.edu.pl)
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
Re: [math] Re: cvs commit: jakarta-commons/math/xdocs tasks.xml
Posted by Phil Steitz <st...@yahoo.com>.
--- "Mark R. Diggory" <md...@latte.harvard.edu> wrote:
> Hey Phil,
>
> I'm still planning to get around to some confidence interval features
> (problem is determining the which approaches to determining conf are the
> best to use in which situations).
There is no generic solution that will work for arbitrary statistics,
though bootstrap confidence intervals have wide applicability and might be
OK to use as a default (though they are computationally expensive and would
only work for the "stored" impls). I will think more about this.
Even if their not going into the
> initial release, can we keep them around somewhere on the tasks or dev
> page to give others ideas.
Great. I removed the task that was there because it referred to adding
confidence intervals to the Univariate interface, which no longer exists in
its old form. Also, the ability to compute t-based confidence intervals
exists in the code today.
I will create a "post 0.1" section and add a task there if you don't do it
first.
>
> Also theres allot of solver framework stuff going on in the
> "src/java/experimental" directory. This directory itself will not go
> into the initial release, but some of the code in it is available as
> possible approaches to pluggability of solvers/functions etc into the
> rest of the api.
>
I would like to settle on a release plan for 0.1. Other than completing
the user guide and refactoring RealMatrix, and a little code/test tidying,
what else remains to be done? I will volunteer to play the release manager
role if no one else wants to do it. In any case, I will help.
Phil
__________________________________
Do you Yahoo!?
Yahoo! Hotjobs: Enter the "Signing Bonus" Sweepstakes
http://hotjobs.sweepstakes.yahoo.com/signingbonus
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[math] Re: cvs commit: jakarta-commons/math/xdocs tasks.xml
Posted by "Mark R. Diggory" <md...@latte.harvard.edu>.
Hey Phil,
I'm still planning to get around to some confidence interval features
(problem is determining the which approaches to determining conf are the
best to use in which situations). Even if their not going into the
initial release, can we keep them around somewhere on the tasks or dev
page to give others ideas.
Also theres allot of solver framework stuff going on in the
"src/java/experimental" directory. This directory itself will not go
into the initial release, but some of the code in it is available as
possible approaches to pluggability of solvers/functions etc into the
rest of the api.
-Mark
psteitz@apache.org wrote:
> psteitz 2004/01/13 23:36:35
>
> Modified: math/xdocs tasks.xml
> Log:
> Removed completed/obsolete tasks, added RealMatrix refactoring task.
>
> Revision Changes Path
> 1.16 +8 -6 jakarta-commons/math/xdocs/tasks.xml
>
> Index: tasks.xml
> ===================================================================
> RCS file: /home/cvs/jakarta-commons/math/xdocs/tasks.xml,v
> retrieving revision 1.15
> retrieving revision 1.16
> diff -u -r1.15 -r1.16
> --- tasks.xml 15 Nov 2003 18:38:16 -0000 1.15
> +++ tasks.xml 14 Jan 2004 07:36:35 -0000 1.16
> @@ -34,11 +34,7 @@
> <subsection name="Algorithm Development Tasks">
> <p>These current tasks are planned and in need completion for the inital release.</p>
> <dl>
> - <dt>Add confidence intervals to Univariate implementations.</dt>
> - <dd>Provide any comments on this task here.</dd>
> - <dt>Distributions.</dt>
> - <dd>Extend distribution framework to support discrete distributions and implement binomial and hypergeometric distributions.</dd>
> - <dt>Analysis.</dt>
> + <dt>Root Finding.</dt>
> <dd>
> <ul>
> <li>Rework unit tests for root finding and spline interpolation.</li>
> @@ -64,6 +60,12 @@
> </a></li>
> </ul>
> </dd>
> + <dt>RealMatrix.</dt>
> + <dd>
> + <ul>
> + <li>Refactor RealMatrix to minimally make solve() pluggable.</li>
> + </ul>
> + </dd>
> </dl>
> </subsection>
> </section>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-dev-help@jakarta.apache.org
>
--
Mark Diggory
Software Developer
Harvard MIT Data Center
http://osprey.hmdc.harvard.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org