You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Zhihua Deng (Jira)" <ji...@apache.org> on 2025/06/03 03:10:00 UTC
[jira] [Resolved] (HIVE-28972) HMS performace degradation post HIVE-28909 for alter query
[ https://issues.apache.org/jira/browse/HIVE-28972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhihua Deng resolved HIVE-28972.
--------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Fix has been pushed to master. Thank you [~Aggarwal_Raghav] for reporting the issue and testing the PR, [~dkuzmenko] for the review!
> HMS performace degradation post HIVE-28909 for alter query
> ----------------------------------------------------------
>
> Key: HIVE-28972
> URL: https://issues.apache.org/jira/browse/HIVE-28972
> Project: Hive
> Issue Type: Bug
> Reporter: Raghav Aggarwal
> Assignee: Zhihua Deng
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
> Attachments: Beeline_Output_with_HIVE-28909.png, Beeline_Output_without_HIVE-28909.png, HMS_Heap_Graph.png, VisualVM_Graphs.png, create_tbl.sql, hivemetastore_with_HIVE-28909.log.tar.gz, hivemetastore_without_HIVE-28909.log, hiveserver2_with_HIVE-28909.log, hiveserver2_without_HIVE-28909.log
>
>
> ENV: Hive master branch (26e6880c9053717c17e5e6416451750e832d46c9) + JAVA 8 + Datanucleus 5.x, HMS mem: 8GB
> Decription: I have a table with 800 columns and 5000 partitions. I ran
> {code:java}
> alter table test_tbl add columns (col801 string) cascade; {code}
> and without HIVE-28909 it took 117.42 sec but with HIVE-28909 it is taking *986.065 sec*
>
> *Steps to reproduce:*
> # Create partitioned table with 800 columns (attaching the create table sql)
> # Create 5000 or so empty partitions in hdfs (I used [https://github.com/Aggarwal-Raghav/Concurrent-Partition-Gen] )
> # Run msck repair table test_tbl to load the partitions to HMS
> # alter table test_tbl add columns (col801 string) cascade;
> Attaching all the captured info i.e. HMS logs, HMS2, beeline screenshots with and without HIVE-28909
>
> The following log is coming: *4006601 times*
>
> {code:java}
> 2025-05-27T23:08:02,690 INFO [Metastore-Handler-Pool: Thread-64] DataNucleus.Persistence: Object "org.apache.hadoop.hive.metastore.model.MColumnDescriptor@68097e71" has a collection "org.apache.hadoop.hive.metastore.model.MColumnDescriptor.fields" yet element "org.apache.hadoop.hive.metastore.model.MColumn@43adb9db" doesnt have the owner set. Managing the relation and setting the owner.
> 2025-05-27T23:08:02,690 INFO [Metastore-Handler-Pool: Thread-64] DataNucleus.Persistence: Object "org.apache.hadoop.hive.metastore.model.MColumnDescriptor@68097e71" has a collection "org.apache.hadoop.hive.metastore.model.MColumnDescriptor.fields" yet element "org.apache.hadoop.hive.metastore.model.MColumn@29276cc5" doesnt have the owner set. Managing the relation and setting the owner.
> 2025-05-27T23:08:02,690 INFO [Metastore-Handler-Pool: Thread-64] DataNucleus.Persistence: Object "org.apache.hadoop.hive.metastore.model.MColumnDescriptor@68097e71" has a collection "org.apache.hadoop.hive.metastore.model.MColumnDescriptor.fields" yet element "org.apache.hadoop.hive.metastore.model.MColumn@109992db" doesnt have the owner set. Managing the relation and setting the owner.
> 2025-05-27T23:08:02,691 INFO [Metastore-Handler-Pool: Thread-64] DataNucleus.Persistence: Object "org.apache.hadoop.hive.metastore.model.MColumnDescriptor@68097e71" has a collection "org.apache.hadoop.hive.metastore.model.MColumnDescriptor.fields" yet element "org.apache.hadoop.hive.metastore.model.MColumn@69748f31" doesnt have the owner set. Managing the relation and setting the owner.
> 2025-05-27T23:08:02,691 INFO [Metastore-Handler-Pool: Thread-64] DataNucleus.Persistence: Object "org.apache.hadoop.hive.metastore.model.MColumnDescriptor@68097e71" has a collection "org.apache.hadoop.hive.metastore.model.MColumnDescriptor.fields" yet element "org.apache.hadoop.hive.metastore.model.MColumn@6ae74875" doesnt have the owner set. Managing the relation and setting the owner. {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)