[jira] [Commented] (MATH-1670) Reuse "Commons Statistics"

["Alex Herbert (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MATH-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17939671#comment-17939671 ] 

Alex Herbert commented on MATH-1670:
------------------------------------

Commons Statistics (CS) can replace most of the functionality in the descriptive, inference and ranking packages of Commons Math (CM). A summary of each in order of complexity follows.
h1. ranking

The functionality of ranking is to provide an interface to rank a double[] with an implementation and some options. This has been transferred to Commons Statistics Ranking.

action: remove package from CM
h1. inference

All the inference tests have been updated and improved in Commons Statistics Inference.

There is no support for computing OneWayAnova statistics using a Collection<SummaryStatistics>. The ANOVA has to be computed using the raw data Collection<double[]>.

The InferenceTestUtils provides static calls to inference tests. This can be done in Commons Statistics using the current API:
{code:java}
double[] expected = ...
long[] observed = ...
double alpha = 0.01;

// CM
InferenceTestUtils.chiSquareTest(expected, observed, alpha)

// CS
ChiSquareTest.withDefaults().test(expected, observed).reject(alpha){code}
action: remove package from CM
h1. descriptive

The following classes have functionality that cannot be reproduced in CS:

WeightedEvaluation, implemented by
moment.Mean
moment.Variance
summary.Product

Note: Mean uses Sum but that does not implement this interface.

action: keep

StatisticalMultivariateSummary
MultivariateSummaryStatistics

CS has no support for computation of statistics on multi-dimensional data.

action: keep

SynchronizedDescriptiveStatistics
SynchronizedMultivariateSummaryStatistics
SynchronizedSummaryStatistics

These intefaces can be reproduced by CS summary statistic aggregators. But there is no support for synchronization.

action: keep

DescriptiveStatistics
- Allows a rolling window of statistics.
- Can return stored double[] of values.
- Computes stats on method call using the current array window.

action: keep and delegate to the Commons Statistics implementations.
h2. descriptive.moment:

VectorialMean
VectorialCovariance

action: keep
h2. descriptive.rank:

Contains a PivotingStrategy for use in Median and Percentile. These implementations are based on a single-pivot quickselect with a configurable pivot strategy. CS uses the selection API from Commons Numbers to compute the median and percentile. This API is backed by a double-pivot quickselect and a median-of-medians based single pivot quickselect. Neither of these require configuration of the pivoting strategy.

action: remove

 

I believe the remaining classes in CM descriptive can be removed as they are replaced by CS.
h2. Next steps

I have assessed the impact of removing classes. I suggest progressively removing the duplicated functionality in the CM stat package and fixing dependencies. The order should be by complexity:

Removal of the inference package should not impact other code.

Removal of the ranking package will affect the median/percentile classes and the correlation.SpearmansCorrelation class with some refactoring required.

Partial removal of the descriptive package will be the largest change with refactoring to use the CS API to replace removed code.

 

> Reuse "Commons Statistics"
> --------------------------
>
>                 Key: MATH-1670
>                 URL: https://issues.apache.org/jira/browse/MATH-1670
>             Project: Commons Math
>          Issue Type: Wish
>          Components: legacy
>    Affects Versions: 4.0-beta1
>            Reporter: Gilles Sadowski
>            Priority: Major
>              Labels: Obsolete, refactor
>             Fix For: 4.0-beta2
>
>
> As of version 1.1, ["Commons Statistics"|https://commons.apache.org/proper/commons-statistics] contains the following modules:
> * [{{org.apache.commons.statistics.descriptive}}|https://commons.apache.org/proper/commons-statistics/commons-statistics-docs/apidocs/org/apache/commons/statistics/descriptive/package-summary.html]
> * [{{org.apache.commons.statistics.distribution}}|https://commons.apache.org/proper/commons-statistics/commons-statistics-docs/apidocs/org/apache/commons/statistics/distribution/package-summary.html]
> * [{{org.apache.commons.statistics.inference}}|https://commons.apache.org/proper/commons-statistics/commons-statistics-docs/apidocs/org/apache/commons/statistics/inference/package-summary.html]
> * [{{org.apache.commons.statistics.ranking}}|https://commons.apache.org/proper/commons-statistics/commons-statistics-docs/apidocs/org/apache/commons/statistics/ranking/package-summary.html]
> Code in [Statistics] should be called from [Math] classes (replacing calls to similar functionality in packages below [{{org.apache.commons.math4.legacy.stat}}|https://commons.apache.org/proper/commons-math/javadocs/api-4.0-beta1/org/apache/commons/math4/legacy/stat/package-summary.html]).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)