You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@impala.apache.org by "Vincent Tran (JIRA)" <ji...@apache.org> on 2018/05/15 19:52:00 UTC

[jira] [Created] (IMPALA-7034) Increase scalability of metadata handling

Vincent Tran created IMPALA-7034:
------------------------------------

             Summary: Increase scalability of metadata handling
                 Key: IMPALA-7034
                 URL: https://issues.apache.org/jira/browse/IMPALA-7034
             Project: IMPALA
          Issue Type: Improvement
    Affects Versions: Impala 2.13.0
            Reporter: Vincent Tran


Currently the practical limit for catalog topic update is in the neighborhood of 4GB - the fundamental limit of max thrift message size. This is an architectural limitation and not a resource limitation

Larger enterprise clusters with high file counts can easily surpass this with normal usage.
The high level ask here is for a more scalable implementation for metadata handling. The amount metadata that a cluster can handle should be proportional to the amount of hardware resource that a user is willing to allocate to it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)