You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Yibo Cai (Jira)" <ji...@apache.org> on 2020/10/10 07:27:00 UTC
[jira] [Updated] (ARROW-10263) [C++][Compute] Improve numerical
stability of variances merging
[ https://issues.apache.org/jira/browse/ARROW-10263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yibo Cai updated ARROW-10263:
-----------------------------
Description:
For chunked array, variance kernel needs to merge variances.
Tested with two single value chunk, [40000008000000490], [40000008000000400]. The merged variance is 3872. If treated as single array with two values, the variance is 3904, same as numpy outputs.
So current merging method is not stable in extreme cases when chunks are very short and with approximate mean values.
was:
For chunked array, variance kernel needs to merge variances.
Tested with two single value chunk, [40000008000000490], [40000008000000400]. The merged variance is 3872. If treated as single array with two values, the variance is 3904, same as numpy outputs.
So current merging method is not stable in extreme cases.
> [C++][Compute] Improve numerical stability of variances merging
> ---------------------------------------------------------------
>
> Key: ARROW-10263
> URL: https://issues.apache.org/jira/browse/ARROW-10263
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Yibo Cai
> Assignee: Yibo Cai
> Priority: Major
>
> For chunked array, variance kernel needs to merge variances.
> Tested with two single value chunk, [40000008000000490], [40000008000000400]. The merged variance is 3872. If treated as single array with two values, the variance is 3904, same as numpy outputs.
> So current merging method is not stable in extreme cases when chunks are very short and with approximate mean values.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)