You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@madlib.apache.org by "Frank McQuillan (JIRA)" <ji...@apache.org> on 2019/06/13 19:29:00 UTC
[jira] [Updated] (MADLIB-1301) Improve correlation and covariance
memory usage with large number of groups
[ https://issues.apache.org/jira/browse/MADLIB-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Frank McQuillan updated MADLIB-1301:
------------------------------------
Fix Version/s: (was: v2.0)
v1.17
> Improve correlation and covariance memory usage with large number of groups
> ---------------------------------------------------------------------------
>
> Key: MADLIB-1301
> URL: https://issues.apache.org/jira/browse/MADLIB-1301
> Project: Apache MADlib
> Issue Type: Improvement
> Components: Module: Descriptive Statistics
> Reporter: Frank McQuillan
> Priority: Major
> Fix For: v1.17
>
>
> When correlation and covariance are run with large number of groups (100's), can run out of memory. Increasing statement_mem helps, but this JIRA is to investigate and improve memory usage with large numbers of groups.
> Sample findings on correlation for 300K input data set:
> || #groups || statement mem 186M || statement mem 200M || statement mem 500M || statement mem 1000M ||
> | 6 | Success | Success | Success | - |
> | 127 | Success | Success | - | - |
> | 930 | Fail | Fail | Success | - |
> | 1213 | Fail | Fail | Success | - |
> | 4852 | Fail | Fail | Fail | Fail |
> The question is:
> * is this a bug?
> * or is it a limitation of current grouping design?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)