[jira] [Created] (IMPALA-12161) Add query hints for memory limits

["Peter Ebert (Jira)" <ji...@apache.org>]

Peter Ebert created IMPALA-12161:
------------------------------------

             Summary: Add query hints for memory limits
                 Key: IMPALA-12161
                 URL: https://issues.apache.org/jira/browse/IMPALA-12161
             Project: IMPALA
          Issue Type: New Feature
          Components: fe
    Affects Versions: Impala 4.1.2
            Reporter: Peter Ebert


Support for a query hint to specify memory limits would make implementing memory limits in applications easier where multiple queries need different memory limits but are run through the same connection, e.g.:  {{/* +mem_limit=4g */ select * from table}}

---

Session settings for mem_limit are critical for getting large queries to avoid OOM when using admission control pools (for multi tenancy) and the memory estimation happens to severely under-estimate (raising the minimum memory for the pool would also reduce concurrency).  However, for applications that run multiple queries through a single connection (e.g. data visualization tools) this pattern is cumbersome:
 # Set the memory limit
 # Run the large query
 # Reset the memory limit
 # Resume other queries with good estimates

Due to this, some data visualization tools only support session settings at the connection level.  If you need GBs for your worst query and MBs for your average query, this greatly limits the concurrency in that pool using a 'one size fits all' memory limit or it requires maintaining many connections of varying size memory limits to attempt to use memory efficiently.

In one production use case this would limit concurrency by 10x or more.

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org