You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2020/12/23 20:47:00 UTC

[jira] [Resolved] (IMPALA-8509) Data load schema generation should lazily evaluate shell substitutions

     [ https://issues.apache.org/jira/browse/IMPALA-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe McDonnell resolved IMPALA-8509.
-----------------------------------
    Fix Version/s: Impala 3.4.0
       Resolution: Fixed

Most of the eval blocks were converted to use LOAD DATA statements here:

[https://github.com/apache/impala/commit/90ab610d3432a93248486ee38e10f58ec47908e6]

That improved the generate-schema-statements.py time significantly, so I'm going to close this as resolved.

> Data load schema generation should lazily evaluate shell substitutions
> ----------------------------------------------------------------------
>
>                 Key: IMPALA-8509
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8509
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Infrastructure
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Major
>             Fix For: Impala 3.4.0
>
>
> Some of the data loading commands (testescape_* tables in particular) use shell statements to execute. These are evaluated by the 'eval_section' function in generate-schema-statements.py. However, the 'eval_section' call was eagerly evaluated even if the table already existed, which makes 'generate-schema-statements' quite slow on every run.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)