You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2025/11/26 05:44:00 UTC
[jira] [Commented] (HDFS-17855) ViewFS with linkMergeSlash generates invalid paths during listStatus/listLocatedStatus operations, causing InvalidPathException or incorrect path resolution
[ https://issues.apache.org/jira/browse/HDFS-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18040764#comment-18040764 ]
ASF GitHub Bot commented on HDFS-17855:
---------------------------------------
koodin9 opened a new pull request, #8102:
URL: https://github.com/apache/hadoop/pull/8102
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g., 'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
ViewFS with linkMergeSlash generates invalid paths during listStatus/listLocatedStatus operations, causing InvalidPathException or incorrect path resolution.
### How was this patch tested?
Added test codes
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files?
> ViewFS with linkMergeSlash generates invalid paths during listStatus/listLocatedStatus operations, causing InvalidPathException or incorrect path resolution
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17855
> URL: https://issues.apache.org/jira/browse/HDFS-17855
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: viewfs
> Affects Versions: 2.10.2, 3.4.1
> Environment: * Hadoop version: 2.10.2
> * Configuration: ViewFS with linkMergeSlash enabled
> * Affected applications: JobHistoryServer, Hive, any application using ViewFS with linkMergeSlash
> Reporter: SeongHoon Ku
> Priority: Major
> Attachments: HADOOP-ViewFS-linkMergeSlash-fix.patch
>
>
> h1. Summary
> ViewFS with linkMergeSlash generates invalid paths during listStatus/listLocatedStatus operations, causing InvalidPathException or incorrect path resolution
> h1. Description
> When ViewFS is configured with {{linkMergeSlash}}, directory listing operations using *RemoteIterator* generate invalid paths, causing {{InvalidPathException}} errors in applications using the FileContext API.
> * Applications using *FileContext API (ViewFs)* with {{listLocatedStatus()}} or {{listStatusIterator()}}
> * Examples: JobHistoryServer, Hive/Tez applications
> * Specifically fails in {{ViewFs$WrappingRemoteIterator.next()}} method
> h2. Configuration Example
> {code:xml}
> <property>
> <name>fs.defaultFS</name>
> <value>viewfs://hadoop-cluster</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.hadoop-cluster.linkMergeSlash</name>
> <value>hdfs://hadoop-cluster</value>
> </property>
> {code}
> h2. Error Stack Trace
> *JobHistoryServer:*
> {noformat}
> org.apache.hadoop.fs.InvalidPathException: Invalid path name relative paths not allowed:
> hadoop-cluster/user/history/done/2021
> at org.apache.hadoop.fs.AbstractFileSystem.checkPath(AbstractFileSystem.java:370)
> at org.apache.hadoop.fs.AbstractFileSystem.makeQualified(AbstractFileSystem.java:428)
> at org.apache.hadoop.fs.viewfs.ViewFs$WrappingRemoteIterator.next(ViewFs.java:848)
> at org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:238)
> {noformat}
> *Hive (Tez):*
> {noformat}
> org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.io.IOException:
> cannot find dir = viewfs://hadoop-cluster/user/hive/hadoop-cluster/tmp/hive/...
> {noformat}
> *Observed pattern:*
> * Invalid path: {{viewfs://hadoop-cluster/user/hive/hadoop-cluster/tmp/hive/...}}
> * Correct path: {{viewfs://hadoop-cluster/tmp/hive/...}}
> * Working directory and cluster name are duplicated in the path
> ----
> h1. Root Cause
> h2. Technical Analysis
> When {{linkMergeSlash}} is configured, the ViewFS root node is created with its path name ({{fullPath}}) incorrectly set to {{mountTableName}} instead of {{"/"}}.
> *Bug location in {{InodeTree.java}}:*
> {code:java}
> // Current (buggy) code
> if (isMergeSlashConfigured) {
> root = new INodeLink<T>(mountTableName, ugi, // "hadoop-cluster" - BUG!
> initAndGetTargetFs(), mergeSlashTarget);
> mountPoints.add(new MountPoint<T>("/", (INodeLink<T>) root));
> rootFallbackLink = null;
> }
> {code}
> This causes {{root.fullPath}} to be set to the cluster name (e.g., {{"hadoop-cluster"}}) instead of {{"/"}}.
> h2. Impact Chain
> # During path resolution ({{InodeTree.java}}), {{root.fullPath}} is used as {{ResolveResult.resolvedPath}}:
> {code:java}
> if (root.isLink()) {
> ResolveResult<T> res = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
> getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
> // ^^^^^^^^^^^^^ Uses mountTableName!
> return res;
> }
> {code}
> # During path conversion in {{ViewFileSystem.getChrootedPath()}} (line 563):
> {code:java}
> return this.makeQualified(
> suffix.length() == 0 ? f : new Path(res.resolvedPath, suffix));
> // Creates: new Path("hadoop-cluster", "user/history/done")
> // Result: "hadoop-cluster/user/history/done" (RELATIVE PATH!)
> {code}
> # {{makeQualified()}} then prepends the working directory to this relative path:
> {noformat}
> Expected: viewfs://hadoop-cluster/user/history/done
> Actual: viewfs://hadoop-cluster/user/mapred/hadoop-cluster/user/history/done
> {noformat}
> h2. Why linkMergeSlash Should Use "/"
> {{linkMergeSlash}} is designed to merge the entire ViewFS root with a single target directory. Therefore:
> * ViewFS root ({{/}}) = Target directory specified by linkMergeSlash
> * The root node's {{fullPath}} should naturally be {{/}}
> * This maintains consistency with the {{MountPoint}} API which already returns {{/}}
> ----
> h1. Testing
> h2. Test Cases
> Added comprehensive test cases in {{TestViewFileSystemLinkMergeSlash.java}}:
> # *{{testListStatusReturnsCorrectPaths()}}*
> ** Verifies {{listStatus()}} returns proper ViewFS paths
> ** Checks scheme, authority, and path correctness
> # *{{testListLocatedStatusReturnsCorrectPaths()}}*
> ** Verifies {{listLocatedStatus()}} with RemoteIterator
> ** Ensures lazy evaluation works correctly
> # *{{testResolvedPathIsAbsolute()}}*
> ** Reproduces exact bug scenario (JobHistoryServer use case)
> ** Validates path resolution for {{/user/history/done/2021}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org