You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ayush Saxena (Jira)" <ji...@apache.org> on 2025/05/27 05:53:00 UTC

[jira] [Comment Edited] (HIVE-28963) Handle FNF when ListFiles with recursive fails

    [ https://issues.apache.org/jira/browse/HIVE-28963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954220#comment-17954220 ] 

Ayush Saxena edited comment on HIVE-28963 at 5/27/25 5:52 AM:
--------------------------------------------------------------

ListFiles failing is a Hadoop issue, it is fixed 3.3.6+ & we are at 3.4.1 I believe, So, I don't think this is an issue in our current code


was (Author: ayushtkn):
ListFiles failing is a Hadoop issue, it is fixed 3.3.6+ & we are at 3.4.1 I believe, So, I don't think this is an issue in our code

> Handle FNF when ListFiles with recursive fails
> ----------------------------------------------
>
>                 Key: HIVE-28963
>                 URL: https://issues.apache.org/jira/browse/HIVE-28963
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Kokila N
>            Assignee: Kokila N
>            Priority: Major
>
> Insert Overwrite query with hive.acid.direct.insert.enabled=true writes data directly to the target directory(actual table location) in the MOVE stage. 
> Say insert query is of writeid 5 and cleaner has got writeid 2 directories marked as obsolete
> {code:java}
> private static Path[] getDirectInsertDirectoryCandidatesRecursive(FileSystem fs,
>     Path path, int skipLevels, PathFilter filter) throws IOException {
>   String lastRelDir = null;
>   HashSet<Path> results = new HashSet<Path>();
>   String relRoot = Path.getPathWithoutSchemeAndAuthority(path).toString();
>   if (!relRoot.endsWith(Path.SEPARATOR)) {
>     relRoot += Path.SEPARATOR;
>   }
>   RemoteIterator<LocatedFileStatus> allFiles = fs.listFiles(path, true);
>   while (allFiles.hasNext()) {
>   LocatedFileStatus lfs = allFiles.next();
>   .
>   .
> }{code}
> _*fs.listFiles(path, true)*_
>     - This recursively lists {*}all files{*}, even those that may be obsolete or deleted during iteration. So if Hive's cleaner deletes {{base_000002}} _after_ it's discovered by {{listFiles()}} but _before_ {{hasNext()}} tries to access it (for metadata resolution), we get a {{{}FileNotFoundException{}}}.
> *Cleaner:*
>   There is no issue with the cleaner because it is deleting only the files/directories that are marked as obsolete. 
> There is a fix for this issue in Hadoop upstream HADOOP-18662
> This was discovered when run hadoop version without this fix.
> Considering that we can handle this issue and also optimize from hive side.
> This scenario is race condition but also should occur rarely as cleaner should delete those directories in move stage of the insert between {{listFiles()and}} {{{}hasNext(){}}}to trigger this issue. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)