[jira] [Resolved] (IMPALA-5500) Reduce catalog topic size when --compact_catalog_topic is enabled

["Dimitris Tsirogiannis (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dimitris Tsirogiannis resolved IMPALA-5500.
-------------------------------------------
    Resolution: Fixed

Change-Id: I2f725cd8596205e6101d5b56abf08125faa30b0a
Reviewed-on: http://gerrit.cloudera.org:8080/7268
Reviewed-by: Dimitris Tsirogiannis <dt...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-util.cc
M be/src/catalog/catalog-util.h
M be/src/service/impala-server.cc
A tests/custom_cluster/test_compact_catalog_updates.py
5 files changed, 136 insertions(+), 5 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Dimitris Tsirogiannis: Looks good to me, approved

> Reduce catalog topic size when --compact_catalog_topic is enabled
> -----------------------------------------------------------------
>
>                 Key: IMPALA-5500
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5500
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog
>    Affects Versions: Impala 2.9.0
>            Reporter: Dimitris Tsirogiannis
>            Assignee: Dimitris Tsirogiannis
>            Priority: Blocker
>
> IMPALA-4029 introduced the use of flatbuffers for storing file and block metadata. The flatbuffers are stored as binary fields in the THdfsPartition thrift struct. However, by doing so, we limit the ability of Thrift to compact catalog update messages sent by the statestore to the impalad nodes when --compact_catalog_topic is set to true. In particular, if that flag is set to true upgrading to Impala 2.9 will result in the size of catalog topic updates to increase by 2X, thereby increasing network and CPU utilization both at the statestore and the impalads. We should consider compressing the catalog update messages sent from the statestore to the impalad nodes when that flag is set to true.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)