You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Uma Maheswara Rao G (Jira)" <ji...@apache.org> on 2020/09/23 06:39:00 UTC
[jira] [Comment Edited] (HDFS-15592) DistCP fails with ViewHDFS if
the actual target path is non HDFS
[ https://issues.apache.org/jira/browse/HDFS-15592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17200597#comment-17200597 ]
Uma Maheswara Rao G edited comment on HDFS-15592 at 9/23/20, 6:38 AM:
----------------------------------------------------------------------
{code:java}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt --> hdfs://ns1/OzoneTest/test/test.txt at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: java.lang.UnsupportedOperationException: This API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1 at org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402) at org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more
{code}
was (Author: umamaheswararao):
{noformat}
Error: java.io.IOException: File copy failed: hdfs://ns1/test/test.txt --> hdfs://ns1/OzoneTest/test/test.txt at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:262) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:219) at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:48) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168) Caused by: java.io.IOException: Couldn't run retriable-command: Copying hdfs://ns1/test/test.txt to hdfs://ns1/OzoneTest/test/test.txt at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101) at org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:258) ... 10 more Caused by: java.lang.UnsupportedOperationException: This API:create is specific to DFS. Can't run on other fs:o3fs://bucket.vol.ozone1 at org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:402) at org.apache.hadoop.hdfs.ViewDistributedFileSystem.create(ViewDistributedFileSystem.java:391) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.copyToFile(RetriableFileCopyCommand.java:201) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:143) at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:115) at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87) ... 11 more
{noformat}
> DistCP fails with ViewHDFS if the actual target path is non HDFS
> ----------------------------------------------------------------
>
> Key: HDFS-15592
> URL: https://issues.apache.org/jira/browse/HDFS-15592
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: viewfs, ViewHDFS
> Affects Versions: 3.4.0
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
>
> When we configure target path mount point with Ozone (or any other fs), distcp will fail.
> The reason is, if the src path having ec policy enabled, it will try to retain that properties.SO, in this case it is using DFS specific createFile API.
> But here we have to ensure, tareget path can from non hdfs in ViewHDFS case.
> In RetriayableFIleCopyCommand#copyToFile, we should fix the following piece of code.
>
> {code:java}
> if (preserveEC && sourceStatus.isErasureCoded()
> && sourceStatus instanceof HdfsFileStatus
> && targetFS instanceof DistributedFileSystem) {
> ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
> }{code}
>
> Here it's just checking targetFs instanceof DistributedFileSystem, but in ViewHDFS case, fs will be DFS only but actual target can point to mounted fs. So, to handle this case, we should use resolvePath API and check the resolved target path scheme is dfs or or not.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org