[jira] [Updated] (HIVE-26024) Metastore RDS CPU spike in ALTER TABLE COMMAND

["Samrat Deb (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Samrat Deb updated HIVE-26024:
------------------------------
    Summary: Metastore RDS CPU spike in ALTER TABLE COMMAND  (was: Metastore RDS CPU spike in DDL commands ())

> Metastore RDS CPU spike in ALTER TABLE COMMAND
> ----------------------------------------------
>
>                 Key: HIVE-26024
>                 URL: https://issues.apache.org/jira/browse/HIVE-26024
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Samrat Deb
>            Priority: Major
>
> We are experiencing sudden spike in the cpu utilization of hive-metastore rds, whenever there is `ALTER TABLE ` queries in hive.
> There is sudden spike in the rds cpu (attached screenshot )
> This leads to non-responsive metastore service , and effects other jobs/queries
> Meta Information
> -> Hive version -  3.1.2
> -> Hive metastore version - 3.1.2
> -> Hadoop version - 3.2.2
> -> DB used for metastore - Postgres 42.2.18
> Observations :-
> 1) On running DDL queries like "ALTER TABLE " commands  it fails in around 120s approx with the error message
> ```while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. null```
> though it has returned error , we observed that
> even if the query has returned an error, the query still keeps running on the metastore via querying pg_stat_activity table .
> 2) the spike in metastore CPU is mostly related to partition stats collection, Tried running the queries via a hive cli in debug mode and noticed these logs when the CPU was spiking
> logs
> 	```
> 	2022-02-21T08:33:22,876 DEBUG [<object_id> main] metastore.ObjectStore: Open transaction: count = 3, isActive = true  at:
> 	org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:4232)
> 	org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4129)
> 	sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> 2022-02-21T08:33:22,876 DEBUG [<object_id> main] metastore.ObjectStore: execute removeUnusedColumnDescriptor
> 2022-02-21T08:33:24,482 DEBUG [70618618-a5a0-497a-abc3-6aacf390024e main] metastore.ObjectStore: Commit transaction: count = 2, isactive true  at:
> 	org.apache.hadoop.hive.metastore.ObjectStore.removeUnusedColumnDescriptor(ObjectStore.java:4245)
> 	org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4129)
> 	sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> 2022-02-21T08:33:24,482 DEBUG [<object_id> main] metastore.ObjectStore: successfully deleted a CD in removeUnusedColumnDescriptor
> 2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Commit transaction: count = 1, isactive true  at:
> 	org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4131)
> 	sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Open transaction: count = 2, isActive = true  at:
> 	org.apache.hadoop.hive.metastore.ObjectStore.alterPartition(ObjectStore.java:4127)
> 	sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> 	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-02-21T08:33:24,483 DEBUG [<object_id> main] metastore.ObjectStore: Open transaction: count = 3, isActive = true  at:```



--
This message was sent by Atlassian Jira
(v8.20.1#820001)