[jira] [Resolved] (IMPALA-4762) RECOVER PARTITIONS should send new partitions in small batches to HMS

["Zach Amsden (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zach Amsden resolved IMPALA-4762.
---------------------------------
    Resolution: Fixed


IMPALA-4762: RECOVER PARTITIONS should batch partition updates

Batch updates when doing a RECOVER PARTITIONS on over 500
partitions at a time to avoid HMS timeouts, possible OOM.

Testing: Expanded test coverage with a new python test
for this case.  Test takes ~18s to run.

Change-Id: I7f9334051b11ba8fa16159b7ca67ddc7f2392733
Reviewed-on: http://gerrit.cloudera.org:8080/6275
Reviewed-by: Dimitris Tsirogiannis <dt...@cloudera.com>
Tested-by: Impala Public Jenkins
Author	
Zach Amsden <za...@cloudera.com>
Mar 6, 2017 3:43 PM
Committer	
Impala Public Jenkins <im...@gerrit.cloudera.org>
Mar 13, 2017 6:38 PM
Commit	
eec8d6fd15dc9e914a773aab1390f95d97f515ae
Parent(s)	
d9602df71b6d686a8e8268d0e18e99f2a51d4e78
Change-Id	
I7f9334051b11ba8fa16159b7ca67ddc7f2392733

> RECOVER PARTITIONS should send new partitions in small batches to HMS
> ---------------------------------------------------------------------
>
>                 Key: IMPALA-4762
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4762
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Catalog
>    Affects Versions: Impala 2.5.0
>            Reporter: Juan Yu
>            Assignee: Zach Amsden
>              Labels: ramp-up
>
> HMS cannot handle more than 32k partitions in one call. When adding large amount of partitions, Impala should send the new partitions in smaller batches. otherwise HMS could go OOM.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)