[jira] [Updated] (IMPALA-5885) Parquet scanner does not free local allocations in filter contexts

["Alex Rodoni (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alex Rodoni updated IMPALA-5885:
--------------------------------
    Docs Text:   (was: Workarounds:
Either of the following can help:
* Reduce num_scanner_threads to reduce the scanner thread concurrency.
* Set runtime_filter_mode=0 to disable runtime filters)

> Parquet scanner does not free local allocations in filter contexts
> ------------------------------------------------------------------
>
>                 Key: IMPALA-5885
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5885
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0, Impala 2.9.0, Impala 2.10.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Major
>              Labels: resource-management
>             Fix For: Impala 2.11.0
>
>
> This problem can occur if runtime filter expressions that are evaluated in the scan allocate temporary memory - "local allocations". These accumulate for each scan range and are only 
> freed upon scan range completion. 
> A contrived query that exhibits the problem is the following. If I continue adding upper() and lower() to the expression the memory consumption of the scan node will continue to grow - up to 100MB for each extra function call!
> {code}
> set runtime_filter_wait_time_ms=1000000;
> select straight_join count(*) from tpch_parquet.lineitem l1 join tpch_parquet.lineitem l2 on upper(lower(upper(lower(l1.l_comment)))) = concat(l2.l_comment, 'foo');
> summary;
> {code}
> I think other conjuncts in the scanner may be affected by the same problem, e.g. the min_max conjuncts.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org