[jira] [Created] (IMPALA-7224) UpdateCatalogMetrics very slow when there are many tables

["Todd Lipcon (JIRA)" <ji...@apache.org>]

Todd Lipcon created IMPALA-7224:
-----------------------------------

             Summary: UpdateCatalogMetrics very slow when there are many tables
                 Key: IMPALA-7224
                 URL: https://issues.apache.org/jira/browse/IMPALA-7224
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: Todd Lipcon


impalad calls UpdateCatalogMetrics after each statement which is considered a DDL. This includes statements like USE, SHOW TABLES, DESCRIBE, etc, which don't actually change the number of tables in the catalog, and therefore probably don't need to update metrics. That aside, even when the metrics _do_ need to be updated, the implementation is very slow. It calls getTableNames on each database, which results in (a) creating an array of all the names, (b) sorting that array and (c) encoding/decoding that whole array into Thrift. This is very expensive: on a use case with approximately 8M tables, each such call takes 10-12 seconds of CPU, most of which is spent in sorting and encoding. All that's really needed is a _count_ of tables, which could be fetched directly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)