You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hive.apache.org by "mahesh kumar behera (JIRA)" <ji...@apache.org> on 2018/09/11 04:35:00 UTC

[jira] [Updated] (HIVE-20534) File operation at target side during S3 replication slowing down the replication

     [ https://issues.apache.org/jira/browse/HIVE-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

mahesh kumar behera updated HIVE-20534:
---------------------------------------
    Description: 
1. Check is there during create partition for existence of partition location (in add partitions core method in metastore.java). It’s not required as we would have created the directory and copied the required files to it.
2. Creating qualified directory name (convertAddSpecToMetaPartition method in hive.java)– File system is access to check if the path provided is fully qualified or not. Not sure why it’s taking 1-2 seconds.

  was:Add notification for add partition and insert operation adds the list of files added by the operation. The file listing is done at target side for replication load. This takes 2-3 seconds as s3 is slow. This can be improved by using the file list from the event directory and same can be used to populate the notification table.


> File operation at target side during S3 replication slowing down the replication
> --------------------------------------------------------------------------------
>
>                 Key: HIVE-20534
>                 URL: https://issues.apache.org/jira/browse/HIVE-20534
>             Project: Hive
>          Issue Type: Sub-task
>          Components: repl
>    Affects Versions: 4.0.0
>            Reporter: mahesh kumar behera
>            Assignee: mahesh kumar behera
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.0.0
>
>
> 1. Check is there during create partition for existence of partition location (in add partitions core method in metastore.java). It’s not required as we would have created the directory and copied the required files to it.
> 2. Creating qualified directory name (convertAddSpecToMetaPartition method in hive.java)– File system is access to check if the path provided is fully qualified or not. Not sure why it’s taking 1-2 seconds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)