You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Adriano (Jira)" <ji...@apache.org> on 2022/09/26 09:58:00 UTC
[jira] [Updated] (HIVE-26562) HMS partitions quota
[ https://issues.apache.org/jira/browse/HIVE-26562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adriano updated HIVE-26562:
---------------------------
Description:
The Hive service always suffered on all the versions for the number of partitions that affected the performances, the resources used, and the JDK limit for the array size when it was trying to write the thrift answers.
A 'partition quota' on the HMS can help to prevent a number of issues, at least avoiding dealing with them when it is already too late to restructure the schema of the tables (because of the number of jobs built around it, clients, and so on).
We have the hive.limit.query.max.table.partition and hive.metastore.limit.partition.request (HIVE-13884/HIVE-23556), or hive.exec.max.dynamic.partitions (that's ok for a single execution, but we can't really limit the partitions in case of sequential dynamic inserts).
On HDFS we have a quota for files, but not for directories.
I would propose the Hive team evaluate the idea to have an upper bound directly on the HMS that can prevent a table to have the partitions growing indefinitely (e.g.: in case this limit is hit to abort/fail the operation). Maybe not solve all the issues, but most likely will be of help.
was:
The Hive service always suffered on all the versions for the number of partitions that affected the performances, the resources used, and the JDK limit for the array size when it was trying to write the thrift answers.
A 'partition quota' on the HMS can help to prevent a number of issues, at least avoiding dealing with them when it is already too late to restructure the schema of the tables (because of the number of jobs built around it, clients, and so on).
We have the hive.limit.query.max.table.partition and hive.metastore.limit.partition.request (HIVE-13884/HIVE-23556), or hive.exec.max.dynamic.partitions (that ok for a single execution, but we can't really limit the partitions in case of sequential dynamic inserts).
On HDFS we have a quota for files, but not for directories.
I would propose the Hive team evaluate the idea to have an upper bound directly on the HMS that can prevent a table to have the partitions growing indefinitely (e.g.: in case this limit is hit to abort/fail the operation). Maybe not solve all the issues, but most likely will be of help.
> HMS partitions quota
> --------------------
>
> Key: HIVE-26562
> URL: https://issues.apache.org/jira/browse/HIVE-26562
> Project: Hive
> Issue Type: New Feature
> Components: Metastore
> Environment:
> Reporter: Adriano
> Priority: Major
>
> The Hive service always suffered on all the versions for the number of partitions that affected the performances, the resources used, and the JDK limit for the array size when it was trying to write the thrift answers.
> A 'partition quota' on the HMS can help to prevent a number of issues, at least avoiding dealing with them when it is already too late to restructure the schema of the tables (because of the number of jobs built around it, clients, and so on).
> We have the hive.limit.query.max.table.partition and hive.metastore.limit.partition.request (HIVE-13884/HIVE-23556), or hive.exec.max.dynamic.partitions (that's ok for a single execution, but we can't really limit the partitions in case of sequential dynamic inserts).
> On HDFS we have a quota for files, but not for directories.
> I would propose the Hive team evaluate the idea to have an upper bound directly on the HMS that can prevent a table to have the partitions growing indefinitely (e.g.: in case this limit is hit to abort/fail the operation). Maybe not solve all the issues, but most likely will be of help.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)