You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Zhihua Deng (Jira)" <ji...@apache.org> on 2026/07/10 02:51:00 UTC
[jira] [Assigned] (HIVE-29694) Reuse DB column descriptors when adding partitions in HMS
[ https://issues.apache.org/jira/browse/HIVE-29694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihua Deng reassigned HIVE-29694:
----------------------------------
Assignee: Zsolt Miskolczi (was: Stamatis Zampetakis)
> Reuse DB column descriptors when adding partitions in HMS
> ---------------------------------------------------------
>
> Key: HIVE-29694
> URL: https://issues.apache.org/jira/browse/HIVE-29694
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Reporter: Stamatis Zampetakis
> Assignee: Zsolt Miskolczi
> Priority: Major
> Labels: pull-request-available
>
> In most cases the partitions of the table have the same schema. However, there are use-cases where the partitions have a different schema from the table. The most straightforward example comes from schema evolution where individual columns can be [added/removed/renamed|https://hive.apache.org/docs/latest/language/languagemanual-ddl/#alter-column] at the table level or partition level.
> {code:sql}
> create table person(id string, fname string) partitioned by (birthyear string);
> alter table person add partition (birthyear='1987');
> alter table person add partition (birthyear='1988');
> alter table person add partition (birthyear='1989');
> alter table person add partition (birthyear='1990');
> alter table person add partition (birthyear='1991');
> alter table person partition (birthyear='1989') add columns (lname string);
> alter table person partition (birthyear='1990') add columns (lname string);
> alter table person partition (birthyear='1991') change column fname fullname string;{code}
> By checking the metastore content we can see that we are using the same column descriptor (CD_ID = 1) for the table (SD_ID=1) and the partitions 1987, 1988. However, the partitions 1989, 1990, 1991 each have its own column descriptor. Note that partitions 1989 and 1990 have exactly the same schema so it should be feasible to reuse the same column descriptor.
> {noformat}
> hivedb=# SELECT "SD_ID", SUBSTRING("LOCATION", 69) AS LOCATION, "CD_ID" FROM "SDS";
> SD_ID | location | CD_ID
> -------+---------------------------------+-------
> 1 | warehouse/person | 1
> 2 | warehouse/person/birthyear=1987 | 1
> 3 | warehouse/person/birthyear=1988 | 1
> 4 | warehouse/person/birthyear=1989 | 2
> 5 | warehouse/person/birthyear=1990 | 3
> 6 | warehouse/person/birthyear=1991 | 4
> {noformat}
> The benefit of reusing column descriptors becomes more apparent by inspecting "COLUMNS_V2" table notably the descriptor 2 and 3 where the column entries are identical.
> {noformat}
> hivedb=# SELECT * FROM "COLUMNS_V2" ;
> CD_ID | COMMENT | COLUMN_NAME | TYPE_NAME | INTEGER_IDX
> -------+---------+-------------+-----------+-------------
> 1 | | id | string | 0
> 1 | | fname | string | 1
> 2 | | id | string | 0
> 2 | | fname | string | 1
> 2 | | lname | string | 2
> 3 | | id | string | 0
> 3 | | fname | string | 1
> 3 | | lname | string | 2
> 4 | | id | string | 0
> 4 | | fullname | string | 1
> {noformat}
> At this tiny scale the reuse benefits may be hard to see but lets assume that we have 1K partitions with the same schema (20 columns). Now it could potentially lead to 1K entries in CDS table and 20K entries in COLUMNS_V2 table instead of 1 on each.
> The main driving motivation behind this improvement is replication scenarios with many tables and partitions. The replication tool calls the org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore.Iface#add_partition for each partition on the source cluster and when the partition schema differs from the table schema we end up having a distinct column descriptor for each partition entry.
> The goal is to add an option to be able to reuse column descriptors across partitions of the same table. Since replication is not a dominant use-case the reuse logic must be behind a configuration property (disabled by default) to avoid performance regressions on the main use-cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)