[jira] [Reopened] (HUDI-1268) Fix UpgradeDowngrade fs Rename issue for hdfs and aliyun oss

["liwei (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

liwei reopened HUDI-1268:
-------------------------

> Fix UpgradeDowngrade fs Rename issue for hdfs and aliyun oss
> ------------------------------------------------------------
>
>                 Key: HUDI-1268
>                 URL: https://issues.apache.org/jira/browse/HUDI-1268
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: Writer Core
>            Reporter: leesf
>            Assignee: liwei
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 0.6.1
>
>
> 一、issue：in UpgradeDowngrade.run()
> fs.rename(updatedPropsFilePath, propsFilePath); 
> the fs.rename have different action,such as:
> // a) for hdfs : if propsFilePath already exist,fs.rename will not replace propsFilePath, but just return false
> // b) for localfs: if propsFilePath already exist,fs.rename will replace propsFilePath, and return ture
> // c) for aliyun ossfs: if propsFilePath already exist,will throw FileAlreadyExistsException
> // so we should delete the old propsFilePath. also upgrade and downgrade is Idempotent
>  
> 二、for aliyun oss filesystem
> when using HoodieWriteClient API to write data to hudi with following config:
> ```
> Properties properties = new Properties();
>  properties.setProperty(HoodieTableConfig.HOODIE_TABLE_NAME_PROP_NAME, tableName);
>  properties.setProperty(HoodieTableConfig.HOODIE_TABLE_TYPE_PROP_NAME, tableType.name());
>  properties.setProperty(HoodieTableConfig.HOODIE_PAYLOAD_CLASS_PROP_NAME, OverwriteWithLatestAvroPayload.class.getName());
>  properties.setProperty(HoodieTableConfig.HOODIE_ARCHIVELOG_FOLDER_PROP_NAME, "archived");
>  return HoodieTableMetaClient.initTableAndGetMetaClient(hadoopConf, basePath, properties);
> ```
> the exception will be thrown with FileAlreadyExistsException in aliyun OSS, after debugging, it is the following code throws the exception.
>  
> ```
> // Rename the .updated file to hoodie.properties. This is atomic in hdfs, but not in cloud stores.
>  // But as long as this does not leave a partial hoodie.properties file, we are okay.
>  fs.rename(updatedPropsFilePath, propsFilePath);
> ```
> however, we would ignore the FileAlreadyExistsException since hoodie.properties already exists.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)