You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Gopal V (JIRA)" <ji...@apache.org> on 2019/03/01 00:47:00 UTC
[jira] [Commented] (HIVE-21354) Lock The Entire Table If Majority
Of Partitions Are Locked
[ https://issues.apache.org/jira/browse/HIVE-21354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781140#comment-16781140 ]
Gopal V commented on HIVE-21354:
--------------------------------
bq. Does ACIDv2 apply to Parquet, Avro, JSON, etc?
Yes, it does - you can't UPDATE rows in those formats, but you still get atomic "insert overwrite" across multiple partitions.
> Lock The Entire Table If Majority Of Partitions Are Locked
> ----------------------------------------------------------
>
> Key: HIVE-21354
> URL: https://issues.apache.org/jira/browse/HIVE-21354
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
> Affects Versions: 4.0.0, 3.2.0
> Reporter: BELUGA BEHR
> Priority: Major
>
> One of the bottlenecks of any Hive query is the ZooKeeper locking mechanism. When a Hive query interacts with a table which has a lot of partitions, this may put a lot of stress on the ZK system.
> Please add a heuristic that works like this:
> # Count the number of partitions that a query is required to lock
> # Obtain the total number of partitions in the table
> # If the number of partitions accessed by the query is greater than or equal to half the total number of partitions, simply create one ZNode lock at the table level.
> This would improve performance of many queries, but in particular, a {{select count(1) from table}} ... or ... {{select * from table limit 5}} where the table has many partitions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)