You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by we...@apache.org on 2019/05/17 17:28:23 UTC
[hadoop] branch branch-3.1 updated: HDFS-14323. Distcp fails in
Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file
path. Contributed by Srinivasu Majeti.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/hadoop.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new e599e3b HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti.
e599e3b is described below
commit e599e3b54617d29ed4536397481487225e5faa66
Author: Srinivasu Majeti <sm...@cloudera.com>
AuthorDate: Fri May 17 19:19:12 2019 +0200
HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti.
Signed-off-by: Wei-Chiu Chuang <we...@apache.org>
(cherry picked from commit 3e5e5b028ad7e199d08e524fe7cddeee5db51a6d)
(cherry picked from commit 85f3664bb84f393f8006d3ab781d8ce5ebb1ab62)
---
.../main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
index 6310d3e..35c49a7 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
@@ -609,7 +609,13 @@ public class WebHdfsFileSystem extends FileSystem
boolean pathAlreadyEncoded = false;
try {
fspathUriDecoded = URLDecoder.decode(fspathUri.getPath(), "UTF-8");
- pathAlreadyEncoded = true;
+ //below condition check added as part of fixing HDFS-14323 to make
+ //sure pathAlreadyEncoded is not set in the case the input url does
+ //not have any encoded sequence already.This will help pulling data
+ //from 2.x hadoop cluster to 3.x using 3.x distcp client operation
+ if(!fspathUri.getPath().equals(fspathUriDecoded)) {
+ pathAlreadyEncoded = true;
+ }
} catch (IllegalArgumentException ex) {
LOG.trace("Cannot decode URL encoded file", ex);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org