You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Matthew Rowles (JIRA)" <ji...@apache.org> on 2009/08/21 07:34:14 UTC
[jira] Created: (MATH-287) Add support for weighted descriptive
statistics
Add support for weighted descriptive statistics
-----------------------------------------------
Key: MATH-287
URL: https://issues.apache.org/jira/browse/MATH-287
Project: Commons Math
Issue Type: New Feature
Affects Versions: 2.0
Reporter: Matthew Rowles
Priority: Minor
Add support for weighted descriptive statistics, such as weighted variance.
An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
- Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Phil Steitz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phil Steitz updated MATH-287:
-----------------------------
Fix Version/s: 2.1
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Assignee: Phil Steitz
> Priority: Minor
> Fix For: 2.1
>
> Attachments: weighted.patch, weightTests.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Matthew Rowles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew Rowles updated MATH-287:
--------------------------------
Attachment: weightTests.patch
Here is a patch that also has new tests for Mean, Variance, Product and Sum (as well as the changes in those classes).
The tests test that the expectedWeightedValue is what it's supposed to be and that expectedWeightedValue == expectedValue when all weights are either the same (Mean, Variance) or equal to 1 (Sum, Product).
I calculated the expected values using the definitional formulae with BigDecimals (and lots of decimal places). I rounded the answers to the number of decimal places that were already present in UnivariateStatisticAbstractTest.
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Priority: Minor
> Attachments: weighted.patch, weightTests.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Phil Steitz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phil Steitz updated MATH-287:
-----------------------------
Fix Version/s: 3.0
(was: 2.2)
Moving extensions to 3.0
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Assignee: Phil Steitz
> Priority: Minor
> Fix For: 3.0
>
> Attachments: weighted.patch, weightTests.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Phil Steitz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phil Steitz updated MATH-287:
-----------------------------
Fix Version/s: (was: 2.1)
2.2
As of r894705, all problems with the initial commit on this feature have been resolved. My recommendation is that we release the initial set of weighted stats in 2.1, but keep this issue open and target 2.2 for extension to other statistics and introduction of weighted increment methods for StorelessUnivariateStatistics.
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Assignee: Phil Steitz
> Priority: Minor
> Fix For: 2.2
>
> Attachments: weighted.patch, weightTests.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Phil Steitz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746466#action_12746466 ]
Phil Steitz commented on MATH-287:
----------------------------------
Thanks for the patch!
Looks good. Don't worry about the extraneous diffs in this case, as it looks to me like what is going on is that your editor is stripping trailing spaces, which is a good thing. These should not be in the source, so this is a (trivial) improvement.
Here are some comments.
1) unit tests. We need unit tests before we can apply this patch. See the src/tests directory for examples. Adding unit tests to the patch would be most appreciated. Once we have these, I can commit the code.
Before resolving this issue, we should consider
2) adding the capability to other statistics and to the "storeless" versions (i.e., add an increment(value, weight) method
3) once we have covered all statistics that they contain, add new addValue(-, weight) methods to the statistical aggregate classes (DescriptiveStatistics, SummaryStatistics)
Thanks again for the patch.
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Priority: Minor
> Attachments: weighted.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Phil Steitz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749356#action_12749356 ]
Phil Steitz commented on MATH-287:
----------------------------------
I committed a modified version of the patches above in r809448. Changes from the patch are:
* Clarified javadoc in test(values, weights, begin, end) added to AbstractUnivariateStatistic.
* Added requirements that weights cannot be infinite or NaN and not all weights can be zero.
* Added a test to UnivariateStatisticAbstractTest to verify that when weights are frequency counts, weighted stats behave as expected.
* Changed formula in weighted product to exponentiate rather than multiply the weights.
* Changed and documented the formula used in weighted variance.
More things to do on these (and others to add):
* Add evaluate(values, weights) - i.e., versions with no start, length parameters.
* Add tests to verify correct exception behavior on all bad parameter combinations.
* Replace naive loop in weighted variance with updating formula similar to unweighted. I committed naive version so the formula would be clear and we could argue about it ;)
Re the weighted variance computation: there is no standard formula, as what E(X - E(X))^2 means based on a weighted sample depends on how you look at the weights. The formula that I used is what the R hmisc package uses by default and is correct if the weights are frequency counts. The normalization in the javadoc gives the same result as R when normwt=true.
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Priority: Minor
> Attachments: weighted.patch, weightTests.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (MATH-287) Add support for weighted
descriptive statistics
Posted by "Matthew Rowles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745812#action_12745812 ]
Matthew Rowles edited comment on MATH-287 at 8/20/09 10:52 PM:
---------------------------------------------------------------
I've added weighted evaluate methods for Mean, Product, Sum and Variance. I've also added a test method to AbstractUnivariateStatistic.
When I run svn diff to generate a patch file, there are a lot of "changes" that I haven't done (lots of - a line followed by + the same line), and as a result, the file is about 27k. I don't know what I did wrong...
In my first pass at this problem, the test method tests that the weights array isn't null, is the same length as the values array, and checks each value to ensure they aren't negative. This last check could get a bit time consuming - weighted variance needs a weighted mean needs a weighted sum - all checking the same weights array... The check itself is necessary; I can't see any way around it other than making up a Weights class that knows that it is correct.
Additionally, the new evaluate methods essentially duplicate that of the existing methods, with only the bits that are affected by the weighting being changed. I had an idea that you could have a call to evaluate(values, begin, length) be sent off to evaluate(values, null, begin, length) and have appropriate null checks being done to decide whether or not to do a weighted/unweighted analysis, but I thought that may not be appropriate.
Feedback more than welcome...
was (Author: masher):
I've added weighted options for Mean, Product, Sum and Variance. I've also added a test method to AbstractUnivariateStatistic.
When I run svn diff to generate a patch file, there are a lot of "changes" that I haven't done (lots of - a line followed by + the same line), and as a result, the file is about 27k. I don't know what I did wrong...
In my first pass at this problem, the test method tests that the weights array isn't null, is the same length as the values array, and checks each value to ensure they aren't negative. This last check could get a bit time consuming - weighted variance needs a weighted mean needs a weighted sum - all checking the same weights array... The check itself is necessary; I can't see any way around it other than making up a Weights class that knows that it is correct.
Additionally, the new evaluate methods essentially duplicate that of the existing methods, with only the bits that are affected by the weighting being changed. I had an idea that you could have a call to evaluate(values, begin, length) be sent off to evaluate(values, null, begin, length) and have appropriate null checks being done to decide whether or not to do a weighted/unweighted analysis, but I thought that may not be appropriate.
Feedback more than welcome...
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Priority: Minor
> Attachments: weighted.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Matthew Rowles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew Rowles updated MATH-287:
--------------------------------
Attachment: weighted.patch
I've added weighted options for Mean, Product, Sum and Variance. I've also added a test method to AbstractUnivariateStatistic.
When I run svn diff to generate a patch file, there are a lot of "changes" that I haven't done (lots of - a line followed by + the same line), and as a result, the file is about 27k. I don't know what I did wrong...
In my first pass at this problem, the test method tests that the weights array isn't null, is the same length as the values array, and checks each value to ensure they aren't negative. This last check could get a bit time consuming - weighted variance needs a weighted mean needs a weighted sum - all checking the same weights array... The check itself is necessary; I can't see any way around it other than making up a Weights class that knows that it is correct.
Additionally, the new evaluate methods essentially duplicate that of the existing methods, with only the bits that are affected by the weighting being changed. I had an idea that you could have a call to evaluate(values, begin, length) be sent off to evaluate(values, null, begin, length) and have appropriate null checks being done to decide whether or not to do a weighted/unweighted analysis, but I thought that may not be appropriate.
Feedback more than welcome...
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Priority: Minor
> Attachments: weighted.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MATH-287) Add support for weighted descriptive
statistics
Posted by "Matthew Rowles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746521#action_12746521 ]
Matthew Rowles commented on MATH-287:
-------------------------------------
(1) Currently working on unit tests; should be done within the week. I need to figure out how they work first.
(2) I've started looking at adding the capability to increment(v,w) (and the like) but I wanted to put this patch up to make sure that I'm going in the right direction....
(3) A little while down the track.... :)
> Add support for weighted descriptive statistics
> -----------------------------------------------
>
> Key: MATH-287
> URL: https://issues.apache.org/jira/browse/MATH-287
> Project: Commons Math
> Issue Type: New Feature
> Affects Versions: 2.0
> Reporter: Matthew Rowles
> Priority: Minor
> Attachments: weighted.patch
>
>
> Add support for weighted descriptive statistics, such as weighted variance.
> An example method call would be Variance.evaluate(double[] values, double[] weights, int begin, int length)
> - Suggested by mickeydog@taosnet.com on the mailing list.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.