You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Krisztian Szucs (Jira)" <ji...@apache.org> on 2021/01/16 21:38:00 UTC

[jira] [Updated] (ARROW-10263) [C++][Compute] Improve numerical stability of variances merging

     [ https://issues.apache.org/jira/browse/ARROW-10263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Krisztian Szucs updated ARROW-10263:
------------------------------------
    Fix Version/s:     (was: 2.0.0)
                   3.0.0

> [C++][Compute] Improve numerical stability of variances merging
> ---------------------------------------------------------------
>
>                 Key: ARROW-10263
>                 URL: https://issues.apache.org/jira/browse/ARROW-10263
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>            Reporter: Yibo Cai
>            Assignee: Yibo Cai
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.0.0
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> For chunked array, variance kernel needs to merge variances.
> Tested with two single value chunk, [40000008000000490], [40000008000000400]. The merged variance is 3872. If treated as single array with two values, the variance is 3904, same as numpy outputs.
> So current merging method is not stable in extreme cases when chunks are very short and with approximate mean values.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)