[GitHub] [spark] srowen commented on a change in pull request #24895: [MINOR][DOC] Fix python variance() documentation

[GitBox <gi...@apache.org>]

srowen commented on a change in pull request #24895: [MINOR][DOC] Fix python variance() documentation
URL: https://github.com/apache/spark/pull/24895#discussion_r294853405
 
 

 ##########
 File path: python/pyspark/sql/functions.py
 ##########
 @@ -215,7 +215,7 @@ def _():
                    ' the expression in a group.',
     'stddev_pop': 'Aggregate function: returns population standard deviation of' +
                   ' the expression in a group.',
-    'variance': 'Aggregate function: returns the population variance of the values in a group.',
+    'variance': 'Aggregate function: returns the unbiased variance of the values in a group.',
     'var_samp': 'Aggregate function: returns the unbiased variance of the values in a group.',
 
 Review comment:
   Just say that 'variance' is an alias for 'var_samp' here rather than duplicate anything.
   
   - `var_pop` in databases is the variance of the values (e.g. divided by n). It's 'correct' if the data it's given is the whole 'population', as it is simply the value of the statistic.
   - `var_samp` treats the data as a sample of a larger unknown population. It's still returning the _population_ variance, but an unbiased estimate of it (e.g. divided by n-1).
   
   Therefore the `var_samp` description is maybe best as "an unbiased estimate of the population variance given the values as a sample". "Unbiased sample variance" is OK too though I've never liked that description. The sample variance is just a statistic of given values and can't be 'biased'. It's only biased or not as an estimator of population variance.
   
   Anyway, I'm OK fixing all the docs accordingly, at least to read "unbiased sample variance".
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org