[jira] [Created] (SYSTEMML-945) Improve inter-procedure analysis to avoid more checkpoints

["Matthias Boehm (JIRA)" <ji...@apache.org>]

Matthias Boehm created SYSTEMML-945:
---------------------------------------

             Summary: Improve inter-procedure analysis to avoid more checkpoints
                 Key: SYSTEMML-945
                 URL: https://issues.apache.org/jira/browse/SYSTEMML-945
             Project: SystemML
          Issue Type: Task
            Reporter: Matthias Boehm


Unnecessary caching can lead to substantial performance issues if not all cached intermediates fit into memory. Our inter-procedural analysis already removes unnecessary checkpoints (e.g., checkpoint - update - checkpoint, where we remove the first checkpoint for simple chains of operations, potentially across multiple statement blocks). However, we should extend this by the treatment of other common cases. For example, recent GLM experiments on a small cluster of 4 nodes showed the following:
{code}
GLM_poisson_log train ict=0 on mbperftest/binomial/X10M_1k_dense: 127
GLM_poisson_log train ict=1 on mbperftest/binomial/X10M_1k_dense: 2996
GLM_poisson_log train ict=2 on mbperftest/binomial/X10M_1k_dense: 1971
{code} 

The reason was the additional append for intercept 1 and 2 which triggered a second checkpoint (before the outer loop after the append, in addition to the checkpoint after initial read). Note that the existing IPA removal of unnecessary checkpoints did not trigger because there was an additional sum(X^2) consumer breaking the simple update chain requirement.

Preliminary experiments with an improved handling of checkpoints showed reasonable performance of ~130s for both intercept 1 and 2.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)