[jira] [Updated] (HIVE-28819) [HiveAcidReplication] copy gets stuck for hive.repl.retry.total.duration for FileNotFoundException

["Denys Kuzmenko (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-28819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Denys Kuzmenko updated HIVE-28819:
----------------------------------
    Labels: hive-4.2.0-must pull-request-available replication  (was: pull-request-available replication)

> [HiveAcidReplication] copy gets stuck for hive.repl.retry.total.duration for FileNotFoundException
> --------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-28819
>                 URL: https://issues.apache.org/jira/browse/HIVE-28819
>             Project: Hive
>          Issue Type: Bug
>          Components: repl
>    Affects Versions: 4.1.0
>            Reporter: Harshal Patel
>            Assignee: Harshal Patel
>            Priority: Major
>              Labels: hive-4.2.0-must, pull-request-available, replication
>
> in _*{{FileUtils.java}}*_  in function 
> {code:java}
> public static boolean copy(FileSystem srcFS, Path[] srcs, FileSystem dstFS, Path dst, boolean deleteSource, boolean overwrite, boolean preserveXAttr, Configuration conf) throws IOException 
> {code}
> there is this block which copies the files
>  
> {code:java}
> try { if (!copy(srcFS, srcFS.getFileStatus(src), dstFS, dst, deleteSource, overwrite, preserveXAttr, conf))
> { returnVal = false; }
> } catch (IOException var15) \{ gotException = true; exceptions.append(var15.getMessage()); exceptions.append("\n"); } } if (gotException) \{ throw new IOException(exceptions.toString()); } else \{ return returnVal; } } }
> {code}
> so if file gets deleted during copy operation then it *_FileNotFoundException_* will be wrapped in the *_IOException_* and that is causing an issue
> because of retryable function which is like this
>  
> {code:java}
> private <T> T retryableFxn(Callable<T> callable) throws IOException { Retryable retryable = Retryable.builder() .withHiveConf(hiveConf) .withRetryOnException(IOException.class).withFailOnParentExceptionList(failOnParentExceptionList).build(); try
> { return retryable.executeCallable(() -> callable.call()); }
> catch (Exception e) { if (failOnParentExceptionList.stream().anyMatch(k -> k.isAssignableFrom(e.getClass())))
> { throw new IOException(e); }
> throw new IOException(ErrorMsg.REPL_FILE_SYSTEM_OPERATION_RETRY.getMsg(), e); } }
> {code}
>  
>  
> {code:java}
> private List<Class<? extends Exception>> failOnParentExceptionList = Arrays.asList(org.apache.hadoop.fs.PathIOException.class, org.apache.hadoop.fs.UnsupportedFileSystemException.class, org.apache.hadoop.fs.InvalidPathException.class, org.apache.hadoop.fs.InvalidRequestException.class, org.apache.hadoop.fs.FileAlreadyExistsException.class, org.apache.hadoop.fs.ChecksumException.class, org.apache.hadoop.fs.ParentNotDirectoryException.class, org.apache.hadoop.hdfs.protocol.QuotaExceededException.class, FileNotFoundException.class);
> {code}
> Here, if you see, if it hits *_FileNotFoundException_* then it will not retry but in above case, that is getting wrapped into IOException and that's why it is getting stuck for 24 hours



--
This message was sent by Atlassian Jira
(v8.20.10#820010)