[jira] [Created] (IMPALA-7501) Slim down metastore Partition objects in LocalCatalog cache

["Todd Lipcon (JIRA)" <ji...@apache.org>]

Todd Lipcon created IMPALA-7501:
-----------------------------------

             Summary: Slim down metastore Partition objects in LocalCatalog cache
                 Key: IMPALA-7501
                 URL: https://issues.apache.org/jira/browse/IMPALA-7501
             Project: IMPALA
          Issue Type: Sub-task
            Reporter: Todd Lipcon


I took a heap dump of an impalad running in LocalCatalog mode with a 2G limit after running a production workload simulation for a couple hours. It had 38.5M objects and 2.02GB heap (the vast majority of the heap is, as expected, in the LocalCatalog cache). Of this total footprint, 1.78GB and 34.6M objects are retained by 'Partition' objects. Drilling into those, 1.29GB and 33.6M objects are retained by FieldSchema, which, as far as I remember, are ignored on the partition level by the Impala planner. So, with a bit of slimming down of these objects, we could make a huge dent in effective cache capacity given a fixed budget. Reducing object count should also have the effect of improved GC performance (old gen GC is more closely tied to object count than size)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org