[jira] [Resolved] (IMPALA-2651) codegen overhead can be high

["Michael Smith (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-2651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Smith resolved IMPALA-2651.
-----------------------------------
    Resolution: Fixed

This is no longer actionable on its own. Several linked tickets address specific ideas, and other efforts - async codegen, codegen caching, other improvements to codegen time - are also done or planned.

> codegen overhead can be high
> ----------------------------
>
>                 Key: IMPALA-2651
>                 URL: https://issues.apache.org/jira/browse/IMPALA-2651
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.2, Impala 2.3.0
>            Reporter: Silvius Rus
>            Assignee: Michael Smith
>            Priority: Minor
>              Labels: codegen, performance
>
> We received reports of excessive codegen compilation/optimization times for very large expressions generated by visualization tools.
> We should:
> # Expose codegen optimization levels as query options.  Currently there is only an all or nothing codegen query option.  It's likely that overly complex expressions such as hundreds of cascading conditions take very long and benefit very little from an O2 optimization level, but they could still run significantly faster even at O0 or O1 versus interpreted.
> # Consider dropping to O1 (or turn off riskier passes individually) automatically for very large expressions.
> # Consider parameterizing the compilation duration time limits and set a reasonable default, say 10 seconds.  Either disable codegen or reduce it to, say, O0 if compilation takes longer than the preset limit.
> *Workaround*
> In some cases disabling codegen can help.
> {code}
> SET disable_codegen=true;
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)