You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@impala.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2018/05/25 17:39:00 UTC

[jira] [Created] (IMPALA-7077) Add a configuration for the maximum number of partitions to load

Todd Lipcon created IMPALA-7077:
-----------------------------------

             Summary: Add a configuration for the maximum number of partitions to load
                 Key: IMPALA-7077
                 URL: https://issues.apache.org/jira/browse/IMPALA-7077
             Project: IMPALA
          Issue Type: Improvement
          Components: Catalog
    Affects Versions: Impala 2.12.0
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon


The current design of the catalog server has some tables which are large enough that they cannot be serialized in Thrift under 2GB (eg IMPALA-3910, IMPALA-2648, etc). Anecdotally some users have some extremely large tables that they use via Hive that, if they reference them in an Impala query, will bring down the Impala cluster.

Long term we would like to support these without OOM, but a short term improvement is to add a flag that limits the number of partitions to load to some maximum. If the user references a table that is larger than this size, they should get an error message.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)