You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2022/06/28 13:26:00 UTC

[jira] [Created] (IMPALA-11402) Add flag to limit the number of partitions in a single getPartialCatalogObject request

|  ![](cid:jira-generated-image-avatar-f6e60285-364c-484f-95d9-acb56f65dbe5) |
[Quanlong
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stigahuang)
**created** an issue  
---|---  
|  
---  
|  [IMPALA](https://issues.apache.org/jira/browse/IMPALA) /
[![Improvement](cid:jira-generated-image-avatar-
ac1e40df-4800-4d16-b10d-89e26c1f6538)](https://issues.apache.org/jira/browse/IMPALA-11402)
[IMPALA-11402](https://issues.apache.org/jira/browse/IMPALA-11402)  
---  
[Add flag to limit the number of partitions in a single
getPartialCatalogObject
request](https://issues.apache.org/jira/browse/IMPALA-11402)  
| Issue Type: |  ![Improvement](cid:jira-generated-image-avatar-
ac1e40df-4800-4d16-b10d-89e26c1f6538) Improvement  
---|---  
Assignee: |  [Quanlong
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stigahuang)  
Components: |  Catalog  
Created: |  28/Jun/22 13:25  
Priority: |  ![Major](cid:jira-generated-image-static-major-
dfdb0af7-1b6e-492c-9c66-971cd7e8a55b) Major  
Reporter: |  [Quanlong
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stigahuang)  
|

The response size of getPartialCatalogObject depends on the number of
partitions in the request. Even with the optimization of
[~~IMPALA-7501~~](https://issues.apache.org/jira/browse/IMPALA-7501 "Slim down
metastore Partition objects in LocalCatalog cache"), the response size could
still exceeds the 2GB byte array limit if requesting all partitions of a huge
table.

We should add flag to limit the number of partitions in a single
getPartialCatalogObject request. When more partitions are required, fetch them
in different batches.  
  
---  
|  |  [ ![Add Comment](cid:jira-generated-image-static-comment-icon-
da64aa65-0e6a-4ef3-b1e0-8377f795792f)
](https://issues.apache.org/jira/browse/IMPALA-11402#add-comment "Add
Comment") |  [Add
Comment](https://issues.apache.org/jira/browse/IMPALA-11402#add-comment "Add
Comment")  
---|---  
  
|  This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9) |  |
![Atlassian logo](https://issues.apache.org/jira/images/mail/atlassian-email-
logo.png)  
---