[jira] [Closed] (SYSTEMML-2021) Perftest stratstats w/ codegen getting stuck in optimization

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm closed SYSTEMML-2021.
------------------------------------
       Resolution: Fixed
         Assignee: Matthias Boehm
    Fix Version/s: SystemML 1.0

> Perftest stratstats w/ codegen getting stuck in optimization
> ------------------------------------------------------------
>
>                 Key: SYSTEMML-2021
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2021
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> On stratstats 10K, the codegen optimizer gets stuck during plan enumeration of a DAG partition with 274,877,906,944 plans. During initial compilation, the same partition was processed in 2.8s as most candidate plans have been successfully pruned by cost or structure.
> After a detailed analysis, we can characterize the problem as follows: there is a fusion partition with 38 interesting points, resulting in the huge search space of 274 billion plans. Furthermore, all these materialization points are extremely small while at the same time there are large inputs and compute-intensive operations as well as many common subexpression, which renders the pruning by costs starting from the fuse-all heuristic ineffective. 
> We can address this as follows: First, we should run both heuristics (fuse-all and fuse-no-redundancy) which are the first and last plan upfront to quickly obtain a good lower bound for the costs. Second, we should only consider enumerating large partitions (say with more than 20 interesting points) if the opening heuristics show costs that is further than an epsilon (say 1%) away from the minimal static costs. Together these rules will make the optimizer significantly more robust without missing any meaningful plan choices.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)