You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Pravin Sinha (Jira)" <ji...@apache.org> on 2023/07/29 16:31:00 UTC
[jira] [Resolved] (HIVE-27505) Added batching with retry mechanism in Hive.getAllPartitionsOf API
[ https://issues.apache.org/jira/browse/HIVE-27505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pravin Sinha resolved HIVE-27505.
---------------------------------
Fix Version/s: 4.0.0
Release Note: Merged to master. Thanks for the patch Vikram Ahuja and review zhangbutao !!
Resolution: Fixed
> Added batching with retry mechanism in Hive.getAllPartitionsOf API
> ------------------------------------------------------------------
>
> Key: HIVE-27505
> URL: https://issues.apache.org/jira/browse/HIVE-27505
> Project: Hive
> Issue Type: Bug
> Reporter: Vikram Ahuja
> Assignee: Vikram Ahuja
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> Hive.getAllPartitionsOf() tries to list partitions in one go, it can cause memory related issues with wide tables(>2000 columns) with large amount of partitions when data is being transferred from HMS to HS2 using Thrift. Optimise the flow to use PartitionIterable so that partitions can be fetched in batches rather than one go.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)