You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hive.apache.org by "Peter Varga (Jira)" <ji...@apache.org> on 2021/01/20 14:29:00 UTC

[jira] [Commented] (HIVE-24669) Improve Filesystem usage in Hive::loadPartitionInternal

    [ https://issues.apache.org/jira/browse/HIVE-24669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268605#comment-17268605 ] 

Peter Varga commented on HIVE-24669:
------------------------------------

Local performance measurement results:
Doing insert select from unpartition table on S3 to a partitioned table on S3, creating 100 new dynamic partition.

Avg loadPartitionInternal before patch: 2433 ms
Avg loadPartitionInternal after patch: 1279 ms

> Improve Filesystem usage in Hive::loadPartitionInternal
> -------------------------------------------------------
>
>                 Key: HIVE-24669
>                 URL: https://issues.apache.org/jira/browse/HIVE-24669
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Peter Varga
>            Assignee: Peter Varga
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> * Use native recursive listing instead doing it on the Hive side
>  * Reuse the file list determined for writeNotificationlogs in quickstat generation



--
This message was sent by Atlassian Jira
(v8.3.4#803005)