You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2021/05/06 03:24:11 UTC

[GitHub] [hadoop] zhuxiangyi opened a new pull request #2981: HDFS-16008. RBF: Initialize the ViewFS Maping tool to the Router

zhuxiangyi opened a new pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981


   [HDFS-16008](https://issues.apache.org/jira/browse/HDFS-16008)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-833227893


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 51s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | -1 :x: |  test4tests  |   0m  0s |  |  The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 37s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 22s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 38s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 50s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 16s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 58s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 28s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 28s |  |  the patch passed  |
   | -1 :x: |  blanks  |   0m  0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/1/artifact/out/blanks-eol.txt) |  The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   0m 15s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 30s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 45s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | -1 :x: |  spotbugs  |   1m 20s | [/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html) |  hadoop-hdfs-project/hadoop-hdfs-rbf generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)  |
   | +1 :green_heart: |  shadedclient  |  16m 52s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  22m  9s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 30s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 102m  6s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
   |  |  Dead store to i in org.apache.hadoop.hdfs.tools.federation.RouterAdmin.initViewFsToMountTable(String[], int)  At RouterAdmin.java:org.apache.hadoop.hdfs.tools.federation.RouterAdmin.initViewFsToMountTable(String[], int)  At RouterAdmin.java:[line 1062] |
   |  |  org.apache.hadoop.hdfs.tools.federation.RouterAdmin.initViewFsToMountTable(String[], int) makes inefficient use of keySet iterator instead of entrySet iterator  At RouterAdmin.java:of keySet iterator instead of entrySet iterator  At RouterAdmin.java:[line 1076] |
   | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/1/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 5422d6c1b45a 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / ef407a8065b4e6a0889d90b5cbbb1b81a563d107 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/1/testReport/ |
   | Max. process+thread count | 2271 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/1/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] goiri commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627891890



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());
+        continue;
+      }
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      boolean added = addMount(
+          mount[1], nss, path.toUri().getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path));
+      if (added) {
+        System.out.println("added mount point " + mount[1]);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  public ACLEntity getACLEntityFormHdfsPath(Path path){
+    String owner = null;
+    String group = null;
+    FsPermission mode = null;
+    try {
+      FileSystem fs = path.getFileSystem(getConf());
+      if (fs.exists(path)) {
+        FileStatus fileStatus = fs.getFileStatus(path);
+        owner = fileStatus.getOwner();
+        group = fileStatus.getGroup();
+        mode = fileStatus.getPermission();
+      }
+    } catch (IOException e) {
+      System.out.println("Exception encountered " + e);

Review comment:
       Yes, but it should go to System.err not System.out, right?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627882834



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());
+        continue;
+      }
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      boolean added = addMount(
+          mount[1], nss, path.toUri().getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path));
+      if (added) {
+        System.out.println("added mount point " + mount[1]);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  public ACLEntity getACLEntityFormHdfsPath(Path path){
+    String owner = null;
+    String group = null;
+    FsPermission mode = null;
+    try {
+      FileSystem fs = path.getFileSystem(getConf());
+      if (fs.exists(path)) {
+        FileStatus fileStatus = fs.getFileStatus(path);
+        owner = fileStatus.getOwner();
+        group = fileStatus.getGroup();
+        mode = fileStatus.getPermission();
+      }
+    } catch (IOException e) {
+      System.out.println("Exception encountered " + e);

Review comment:
       I think it fails to get FileStatus and can continue to use the default ACLEntity to add mapping.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-835241248


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 54s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 25s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 32s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 22s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 38s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 53s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 15s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 57s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 28s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 28s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 17s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 56s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 20s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  18m 37s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  24m 15s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 29s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 106m  5s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/5/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux 6fec1ae611c9 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 3be7673a4f31d82f0e2669a1e2d16d132b8d80ab |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/5/testReport/ |
   | Max. process+thread count | 2306 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/5/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r631549963



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1057,83 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * Initialize the ViewFS mount point to the Router,
+   * either to specify a cluster or to initialize it all.
+   * @param clusterName The specified cluster to initialize,
+   * AllCluster was then all clusters.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix;
+    if (clusterName.equals(ALL_CLUSTERS)) {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + ".*" +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    } else {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    }
+    final String rootPath = "/";
+    Map<String, String> viewFsMap = getConf().getValByRegex(
+        mountTablePrefix  + rootPath);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      URI destUri = path.toUri();
+      String mountKey = entry.getKey();
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount = mountKey.replaceAll(mountTablePrefix, "");
+      if (!destUri.getScheme().equals("hdfs")) {
+        System.out.println("Only supports HDFS, " +
+            "added Mount Point failed , " + mountKey);
+      }
+      if (!mount.startsWith(rootPath) ||
+          !destUri.getPath().startsWith(rootPath)) {
+        System.out.println("Added Mount Point failed " + mountKey);
+        continue;
+      }
+      String[] nss = new String[]{destUri.getAuthority()};
+      boolean added = addMount(
+          mount, nss, destUri.getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path, getConf()));
+      if (added) {
+        System.out.println("Added mount point " + mount);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  static public ACLEntity getACLEntityFormHdfsPath(
+      Path path, Configuration conf) {
+    String owner = null;
+    String group = null;
+    FsPermission mode = null;
+    try {
+      FileSystem fs = path.getFileSystem(conf);
+      if (fs.exists(path)) {
+        FileStatus fileStatus = fs.getFileStatus(path);
+        owner = fileStatus.getOwner();
+        group = fileStatus.getGroup();
+        mode = fileStatus.getPermission();
+      }
+    } catch (IOException e) {
+      System.err.println("Exception encountered " + e);

Review comment:
       When FileStatus cannot be obtained, I think the default ACLEntity should be used to add the mapping.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] goiri commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r631274370



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1057,83 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * Initialize the ViewFS mount point to the Router,
+   * either to specify a cluster or to initialize it all.
+   * @param clusterName The specified cluster to initialize,
+   * AllCluster was then all clusters.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix;
+    if (clusterName.equals(ALL_CLUSTERS)) {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + ".*" +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    } else {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    }
+    final String rootPath = "/";
+    Map<String, String> viewFsMap = getConf().getValByRegex(
+        mountTablePrefix  + rootPath);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      URI destUri = path.toUri();
+      String mountKey = entry.getKey();
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount = mountKey.replaceAll(mountTablePrefix, "");
+      if (!destUri.getScheme().equals("hdfs")) {
+        System.out.println("Only supports HDFS, " +
+            "added Mount Point failed , " + mountKey);
+      }
+      if (!mount.startsWith(rootPath) ||
+          !destUri.getPath().startsWith(rootPath)) {
+        System.out.println("Added Mount Point failed " + mountKey);
+        continue;
+      }
+      String[] nss = new String[]{destUri.getAuthority()};
+      boolean added = addMount(
+          mount, nss, destUri.getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path, getConf()));
+      if (added) {
+        System.out.println("Added mount point " + mount);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  static public ACLEntity getACLEntityFormHdfsPath(

Review comment:
       I think the order is usually public static.
   It could also be private.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r628667273



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
##########
@@ -241,6 +241,22 @@ Mount table permission can be set by following command:
 
 The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
 
+#### Init ViewFs To Router
+Router supports initializing the ViewFS mount point to the Router. The mapping directory protocol of ViewFS must be HDFS, and the initializer only supports one-to-one mapping.
+
+For example, use the following viewfs to configure the initial mount table to the router.
+
+    <configuration>
+      <property>
+        <name>fs.viewfs.mounttable.ClusterX.link./data</name>
+        <value>hdfs://nn1-clusterx.example.com:8020/data</value>
+      </property>
+    </configuration>
+
+The ViewFS mount table can be initialized to the Router by using the following command:
+
+    [hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable ClusterX

Review comment:
       Do you mean that there are mount points for multiple clusters, can the user import all of them without specifying the cluster name? For example, the following configuration.
     <configuration>
         <property>
           <name>fs.viewfs.mounttable.ClusterX.link./data</name>
           <value>hdfs://nn1-clusterx.example.com:8020/data</value>
         </property>
         <property>
           <name>fs.viewfs.mounttable.ClusterY.link./project</name>
           <value>hdfs://nn1-clusterx.example.com:8020/project</value>
         </property>
       </configuration>
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] goiri commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r628356897



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
##########
@@ -241,6 +241,22 @@ Mount table permission can be set by following command:
 
 The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
 
+#### Init ViewFs To Router
+Router supports initializing the ViewFS mount point to the Router. The mapping directory protocol of ViewFS must be HDFS, and the initializer only supports one-to-one mapping.
+
+For example, use the following viewfs to configure the initial mount table to the router.
+
+    <configuration>
+      <property>
+        <name>fs.viewfs.mounttable.ClusterX.link./data</name>
+        <value>hdfs://nn1-clusterx.example.com:8020/data</value>
+      </property>
+    </configuration>
+
+The ViewFS mount table can be initialized to the Router by using the following command:
+
+    [hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable ClusterX

Review comment:
       Can we also support an option to import the full table from ViewFs?

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
##########
@@ -852,6 +908,7 @@ public void testInvalidArgumentMessage() throws Exception {
         + " <quota in bytes or quota size string>]\n"
         + "\t[-clrQuota <path>]\n"
         + "\t[-clrStorageTypeQuota <path>]\n"
+        +"\t[-initViewFsToMountTable <clusterName>]\n"

Review comment:
       Space after +

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
##########
@@ -700,6 +706,49 @@ public void testAddMountTableIfParentExist() throws Exception {
     }
   }
 
+  @Test
+  public void testInitViewFsToMountTable() throws Exception {
+    // re-set system out for testing
+    System.setOut(new PrintStream(out));
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    String nnAddress = cluster.getRandomNamenode().
+        getNamenode().getHostAndPort();
+
+    String src = "/data";
+    Path destPath = new Path("hdfs://" + nnAddress + "/data");
+    String user = "user1";
+    String group = "group1";
+    String clusterName = "ClusterX";
+
+    // 0.mkdir destPath
+    hdfs.mkdirs(destPath);
+    // 1.set owner
+    hdfs.setOwner(destPath, user, group);
+    // 2.set viewFs mapping
+    admin.getConf().set(
+        "fs.viewfs.mounttable.ClusterX.link." + src, destPath.toString());
+    // 3.run initialization
+    String[] argv = new String[]{"-initViewFsToMountTable", clusterName};
+    assertEquals(0, ToolRunner.run(admin, argv));
+    // 4.gets the mount point entries
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    GetMountTableEntriesRequest getRequest = GetMountTableEntriesRequest
+        .newInstance(src);
+    GetMountTableEntriesResponse getResponse = client.getMountTableManager()
+        .getMountTableEntries(getRequest);
+    List<MountTable> mountTables = getResponse.getEntries();
+    // 5.check
+    assertEquals(1, mountTables.size());
+    MountTable mountTable = mountTables.get(0);
+    assertEquals(user, mountTable.getOwnerName());
+    assertEquals(group, mountTable.getGroupName());
+    assertEquals(destPath.toUri().getPath(), mountTable.

Review comment:
       Let's assert the size of mountTable.getDestinations() and extract the get(0)

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
##########
@@ -241,6 +241,22 @@ Mount table permission can be set by following command:
 
 The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
 
+#### Init ViewFs To Router
+Router supports initializing the ViewFS mount point to the Router. The mapping directory protocol of ViewFS must be HDFS, and the initializer only supports one-to-one mapping.

Review comment:
       Pointer to ViewFs doc?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] goiri commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r630388995



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1054,81 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.

Review comment:
       Mention the allClusters.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
##########
@@ -700,6 +707,117 @@ public void testAddMountTableIfParentExist() throws Exception {
     }
   }
 
+  @Test
+  public void testInitViewFsToMountTable() throws Exception {
+    // re-set system out for testing
+    System.setOut(new PrintStream(out));
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    String nnAddress = cluster.getRandomNamenode().
+        getNamenode().getHostAndPort();
+    String baseDir = "/initViewFs";
+    String src1 = baseDir + "/data1";
+    Path destPath1 = new Path("hdfs://" + nnAddress + src1);
+    String user1 = "user1";
+    String group1 = "group1";
+    String clusterName1 = "ClusterX";
+
+    String src2 = baseDir + "/data2";
+    String clusterName2 = "ClusterY";
+
+    String src3 = baseDir + "/inExistent";
+    Path destPath3 = new Path("hdfs://" + nnAddress + src3);
+    String clusterName3 = "ClusterZ";
+
+    // 0.mkdir destPath
+    hdfs.mkdirs(destPath1);
+    // 1.set owner
+    hdfs.setOwner(destPath1, user1, group1);
+    // 2.set viewFs mapping
+    // Use different clusterName and mount points
+    admin.getConf().set(CONFIG_VIEWFS_PREFIX + "." +
+        clusterName1 + ".link." + src1, destPath1.toString());
+    admin.getConf().set(CONFIG_VIEWFS_PREFIX + "." +
+        clusterName2 + ".link." + src2, destPath1.toString());
+
+    // 3.run initialization,Specify a ClusterName
+    String[] argv = new String[]{"-initViewFsToMountTable", clusterName1};
+    assertEquals(0, ToolRunner.run(admin, argv));
+    // 4.gets the mount point entries
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    GetMountTableEntriesRequest getRequest = GetMountTableEntriesRequest
+        .newInstance(src1);
+    GetMountTableEntriesResponse getResponse = client.getMountTableManager()
+        .getMountTableEntries(getRequest);
+    List<MountTable> mountTables = getResponse.getEntries();
+    // 5.Checking
+    assertEquals(1, mountTables.size());
+    MountTable mountTable = mountTables.get(0);
+    List<RemoteLocation> destinations = mountTable.getDestinations();
+    assertEquals(1, destinations.size());
+    assertEquals(user1, mountTable.getOwnerName());
+    assertEquals(group1, mountTable.getGroupName());
+    assertEquals(destPath1.toUri().getPath(), mountTable.
+        getDestinations().get(0).getDest());
+    assertEquals(nnAddress, mountTable.
+        getDestinations().get(0).getNameserviceId());
+    assertEquals(src1, mountTable.getSourcePath());
+
+    // Specify allCluster to initialize all mappings
+    argv = new String[]{"-rm", src1};
+    assertEquals(0, ToolRunner.run(admin, argv));
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    argv = new String[]{"-initViewFsToMountTable", "allClusters"};
+    assertEquals(0, ToolRunner.run(admin, argv));
+
+    stateStore.loadCache(MountTableStoreImpl.class, true);

Review comment:
       This test got massive can we refactor and split?

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1054,81 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix;
+    if (clusterName.equals("allClusters")) {

Review comment:
       Maybe make allClusters a constant.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1054,81 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * initViewFsToMountTable.

Review comment:
       Extend this a little.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-840386658


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 36s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  34m  6s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 42s |  |  trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |   0m 38s |  |  trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   0m 28s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 42s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 41s |  |  trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 57s |  |  trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   1m 17s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  14m 16s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 35s |  |  the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |   0m 35s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 30s |  |  the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |   0m 30s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 17s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/10/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 31s |  |  the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  |  the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   1m 16s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  14m 32s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  18m 37s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 33s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  94m 39s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/10/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux 4bd8c41b3b1f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 0204cdcfe14b729a8a189379b3e6e2cefc28f843 |
   | Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/10/testReport/ |
   | Max. process+thread count | 2665 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/10/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] goiri commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r629530054



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
##########
@@ -241,6 +241,22 @@ Mount table permission can be set by following command:
 
 The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
 
+#### Init ViewFs To Router
+Router supports initializing the ViewFS mount point to the Router. The mapping directory protocol of ViewFS must be HDFS, and the initializer only supports one-to-one mapping.
+
+For example, use the following viewfs to configure the initial mount table to the router.
+
+    <configuration>
+      <property>
+        <name>fs.viewfs.mounttable.ClusterX.link./data</name>
+        <value>hdfs://nn1-clusterx.example.com:8020/data</value>
+      </property>
+    </configuration>
+
+The ViewFS mount table can be initialized to the Router by using the following command:
+
+    [hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable ClusterX

Review comment:
       Correct.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-833235149


   @goiri Thank you for your detailed review. I will modify and add unit tests according to your comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627895751



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());
+        continue;
+      }
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      boolean added = addMount(
+          mount[1], nss, path.toUri().getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path));
+      if (added) {
+        System.out.println("added mount point " + mount[1]);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  public ACLEntity getACLEntityFormHdfsPath(Path path){
+    String owner = null;
+    String group = null;
+    FsPermission mode = null;
+    try {
+      FileSystem fs = path.getFileSystem(getConf());
+      if (fs.exists(path)) {
+        FileStatus fileStatus = fs.getFileStatus(path);
+        owner = fileStatus.getOwner();
+        group = fileStatus.getGroup();
+        mode = fileStatus.getPermission();
+      }
+    } catch (IOException e) {
+      System.out.println("Exception encountered " + e);

Review comment:
       Yes,Thank you for reminding me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r628667273



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
##########
@@ -241,6 +241,22 @@ Mount table permission can be set by following command:
 
 The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
 
+#### Init ViewFs To Router
+Router supports initializing the ViewFS mount point to the Router. The mapping directory protocol of ViewFS must be HDFS, and the initializer only supports one-to-one mapping.
+
+For example, use the following viewfs to configure the initial mount table to the router.
+
+    <configuration>
+      <property>
+        <name>fs.viewfs.mounttable.ClusterX.link./data</name>
+        <value>hdfs://nn1-clusterx.example.com:8020/data</value>
+      </property>
+    </configuration>
+
+The ViewFS mount table can be initialized to the Router by using the following command:
+
+    [hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable ClusterX

Review comment:
       Do you mean that there are mount points for multiple clusters, can the user import all of them without specifying the cluster name? For example, the following configuration.
   `  <configuration>
         <property>
           <name>fs.viewfs.mounttable.ClusterX.link./data</name>
           <value>hdfs://nn1-clusterx.example.com:8020/data</value>
         </property>
         <property>
           <name>fs.viewfs.mounttable.ClusterY.link./project</name>
           <value>hdfs://nn1-clusterx.example.com:8020/project</value>
         </property>
       </configuration>`
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r637659170



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1057,83 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * Initialize the ViewFS mount point to the Router,
+   * either to specify a cluster or to initialize it all.
+   * @param clusterName The specified cluster to initialize,
+   * AllCluster was then all clusters.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix;
+    if (clusterName.equals(ALL_CLUSTERS)) {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + ".*" +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    } else {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    }
+    final String rootPath = "/";
+    Map<String, String> viewFsMap = getConf().getValByRegex(
+        mountTablePrefix  + rootPath);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      URI destUri = path.toUri();
+      String mountKey = entry.getKey();
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount = mountKey.replaceAll(mountTablePrefix, "");
+      if (!destUri.getScheme().equals("hdfs")) {
+        System.out.println("Only supports HDFS, " +
+            "added Mount Point failed , " + mountKey);
+      }
+      if (!mount.startsWith(rootPath) ||
+          !destUri.getPath().startsWith(rootPath)) {
+        System.out.println("Added Mount Point failed " + mountKey);
+        continue;
+      }
+      String[] nss = new String[]{destUri.getAuthority()};
+      boolean added = addMount(
+          mount, nss, destUri.getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path, getConf()));

Review comment:
       @Hexiaoqiao  I didn't find any problems here, can you tell me the details, thank you very much.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-840340141


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 50s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 39s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 38s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 23s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 38s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 14s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 42s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 35s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 35s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 29s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 29s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 16s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/8/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 49s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 18s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  17m 20s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  24m 20s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 28s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 104m 43s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/8/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux 6906f7df6f8f 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 33701470f5825366a00169e05cb02c0cc54a6bab |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/8/testReport/ |
   | Max. process+thread count | 2350 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/8/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-839929328


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 55s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 41s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 23s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 39s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 15s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 59s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 34s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 34s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 27s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 27s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 15s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/7/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 30s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 46s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 17s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  17m 18s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  23m 19s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 30s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 104m  2s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/7/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux 6dfd2910487e 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 698a3a94ada4348e981cc7ad6e76d213fad4e04f |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/7/testReport/ |
   | Max. process+thread count | 2342 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/7/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
Hexiaoqiao commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r631534832



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -171,7 +179,9 @@ private String getUsage(String cmd) {
       return "\t[-clrQuota <path>]";
     } else if (cmd.equals("-clrStorageTypeQuota")) {
       return "\t[-clrStorageTypeQuota <path>]";
-    } else if (cmd.equals("-safemode")) {
+    }  else if (cmd.equals("-initViewFsToMountTable")) {
+      return "\t[-initViewFsToMountTable <clusterName>] | allClusters";

Review comment:
       it should be `\t[-initViewFsToMountTable <clusterName> | allClusters]`?

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -384,6 +398,13 @@ public int run(String[] argv) throws Exception {
         getDisabledNameservices();
       } else if ("-refresh".equals(cmd)) {
         refresh(address);
+      } else if ("-initViewFsToMountTable".equals(cmd)) {
+        if (initViewFsToMountTable(argv[i])) {
+          System.out.println("Successfully init ViewFs mapping to router " +
+              argv[i]);
+        } else {

Review comment:
       if this command run failed, we should print err information such as `System.err.println("Failed when execute command 'initViewFsToMountTable'")` or something else;

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1057,83 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * Initialize the ViewFS mount point to the Router,
+   * either to specify a cluster or to initialize it all.
+   * @param clusterName The specified cluster to initialize,
+   * AllCluster was then all clusters.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix;
+    if (clusterName.equals(ALL_CLUSTERS)) {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + ".*" +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    } else {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    }
+    final String rootPath = "/";
+    Map<String, String> viewFsMap = getConf().getValByRegex(
+        mountTablePrefix  + rootPath);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      URI destUri = path.toUri();
+      String mountKey = entry.getKey();
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount = mountKey.replaceAll(mountTablePrefix, "");
+      if (!destUri.getScheme().equals("hdfs")) {

Review comment:
       It is better to use HdfsConstants#HDFS_URI_SCHEME?

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1057,83 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * Initialize the ViewFS mount point to the Router,
+   * either to specify a cluster or to initialize it all.
+   * @param clusterName The specified cluster to initialize,
+   * AllCluster was then all clusters.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix;
+    if (clusterName.equals(ALL_CLUSTERS)) {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + ".*" +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    } else {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    }
+    final String rootPath = "/";
+    Map<String, String> viewFsMap = getConf().getValByRegex(
+        mountTablePrefix  + rootPath);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      URI destUri = path.toUri();
+      String mountKey = entry.getKey();
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount = mountKey.replaceAll(mountTablePrefix, "");
+      if (!destUri.getScheme().equals("hdfs")) {
+        System.out.println("Only supports HDFS, " +
+            "added Mount Point failed , " + mountKey);
+      }
+      if (!mount.startsWith(rootPath) ||
+          !destUri.getPath().startsWith(rootPath)) {
+        System.out.println("Added Mount Point failed " + mountKey);
+        continue;
+      }
+      String[] nss = new String[]{destUri.getAuthority()};
+      boolean added = addMount(
+          mount, nss, destUri.getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path, getConf()));
+      if (added) {
+        System.out.println("Added mount point " + mount);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  static public ACLEntity getACLEntityFormHdfsPath(
+      Path path, Configuration conf) {
+    String owner = null;
+    String group = null;
+    FsPermission mode = null;
+    try {
+      FileSystem fs = path.getFileSystem(conf);
+      if (fs.exists(path)) {
+        FileStatus fileStatus = fs.getFileStatus(path);
+        owner = fileStatus.getOwner();
+        group = fileStatus.getGroup();
+        mode = fileStatus.getPermission();
+      }
+    } catch (IOException e) {
+      System.err.println("Exception encountered " + e);

Review comment:
       just suggest to throw exception rather than just print the error information.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
##########
@@ -241,6 +241,26 @@ Mount table permission can be set by following command:
 
 The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
 
+#### Init ViewFs To Router
+Router supports initializing the [ViewFs](../hadoop-hdfs/ViewFs.html) mount point to the Router. The mapping directory protocol of ViewFS must be HDFS, and the initializer only supports one-to-one mapping.
+
+For example, use the following [ViewFs](../hadoop-hdfs/ViewFs.html) to configure the initial mount table to the router.
+
+    <configuration>
+      <property>
+        <name>fs.viewfs.mounttable.ClusterX.link./data</name>
+        <value>hdfs://nn1-clusterx.example.com:8020/data</value>
+      </property>
+      <property>
+        <name>fs.viewfs.mounttable.ClusterY.link./project</name>
+        <value>hdfs://nn1-clustery.example.com:8020/project</value>
+      </property>
+    </configuration>
+
+The [ViewFs](../hadoop-hdfs/ViewFs.html) mount table can be initialized to the Router by using the following command:
+
+    [hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable <clusterName>] | allClusters

Review comment:
       `$HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable <clusterName>] | allClusters`
   to 
   `$HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable [<clusterName> | allClusters]`

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1036,6 +1057,83 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
     return updateResponse.getStatus();
   }
 
+  /**
+   * Initialize the ViewFS mount point to the Router,
+   * either to specify a cluster or to initialize it all.
+   * @param clusterName The specified cluster to initialize,
+   * AllCluster was then all clusters.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix;
+    if (clusterName.equals(ALL_CLUSTERS)) {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + ".*" +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    } else {
+      mountTablePrefix =
+          Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+              Constants.CONFIG_VIEWFS_LINK + ".";
+    }
+    final String rootPath = "/";
+    Map<String, String> viewFsMap = getConf().getValByRegex(
+        mountTablePrefix  + rootPath);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      URI destUri = path.toUri();
+      String mountKey = entry.getKey();
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount = mountKey.replaceAll(mountTablePrefix, "");
+      if (!destUri.getScheme().equals("hdfs")) {
+        System.out.println("Only supports HDFS, " +
+            "added Mount Point failed , " + mountKey);
+      }
+      if (!mount.startsWith(rootPath) ||
+          !destUri.getPath().startsWith(rootPath)) {
+        System.out.println("Added Mount Point failed " + mountKey);
+        continue;
+      }
+      String[] nss = new String[]{destUri.getAuthority()};
+      boolean added = addMount(
+          mount, nss, destUri.getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path, getConf()));

Review comment:
       if we specify `clustername`, it could be not proper to invoke directly, right?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-837999827


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 39s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  36m 42s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 43s |  |  trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |   0m 37s |  |  trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   0m 31s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 45s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 40s |  |  trunk passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 54s |  |  trunk passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   1m 18s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  14m 34s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 34s |  |  the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |   0m 34s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 29s |  |  the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |   0m 29s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 16s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/6/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 31s |  |  the patch passed with JDK Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 46s |  |  the patch passed with JDK Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   1m 19s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  15m 26s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  18m 40s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 37s |  |  The patch does not generate ASF License warnings.  |
   |  |   |  98m 39s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/6/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux c018e51957af 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / c33ba431ae357c5ac891bbe8e90140cea4243c9f |
   | Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/6/testReport/ |
   | Max. process+thread count | 2337 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/6/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] goiri commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627643299



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());
+        continue;
+      }
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      boolean added = addMount(
+          mount[1], nss, path.toUri().getPath(), false,

Review comment:
       Extract mount[1]

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
##########
@@ -700,6 +707,46 @@ public void testAddMountTableIfParentExist() throws Exception {
     }
   }
 
+  @Test
+  public void testInitViewFsToMountTable() throws Exception {
+    // re-set system out for testing
+    System.setOut(new PrintStream(out));
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    String nnAddress = cluster.getRandomNamenode().getNamenode().getHostAndPort();
+
+    String src = "/data";
+    Path destPath = new Path("hdfs://" + nnAddress + "/data");
+    String user = "user1";
+    String group = "group1";
+    String clusterName = "ClusterX";
+
+    // 0.mkdir destPath
+    hdfs.mkdirs(destPath);
+    // 1.set owner
+    hdfs.setOwner(destPath, user, group);
+    // 2.set viewFs mapping
+    admin.getConf().set("fs.viewfs.mounttable.ClusterX.link." + src, destPath.toString());
+    // 3.run initialization
+    String[] argv = new String[]{"-initViewFsToMountTable", clusterName};
+    assertEquals(0, ToolRunner.run(admin, argv));
+    // 4.gets the mount point entries
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    GetMountTableEntriesRequest getRequest = GetMountTableEntriesRequest
+        .newInstance("/");
+    GetMountTableEntriesResponse getResponse = client.getMountTableManager()
+        .getMountTableEntries(getRequest);
+    List<MountTable> mountTables = getResponse.getEntries();
+    // 5.check
+    assertEquals(1, mountTables.size());
+    assertEquals(user, mountTables.get(0).getOwnerName());

Review comment:
       Extract mountTables.get(0) after the assert for size.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
##########
@@ -78,6 +81,8 @@
   private static RouterClient client;
   private static Router router;
 
+  private static DistributedFileSystem hdfs;

Review comment:
       I don't think we need any HDFS feature do we? Let's do: fs = cluster.getCluster().getFileSystem();

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());

Review comment:
       Extract entry.getKey()

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(

Review comment:
       Would it be cleaner to use do a regex or something better than splitting blindly?

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterAdminCLI.java
##########
@@ -700,6 +707,46 @@ public void testAddMountTableIfParentExist() throws Exception {
     }
   }
 
+  @Test
+  public void testInitViewFsToMountTable() throws Exception {
+    // re-set system out for testing
+    System.setOut(new PrintStream(out));
+    stateStore.loadCache(MountTableStoreImpl.class, true);
+    String nnAddress = cluster.getRandomNamenode().getNamenode().getHostAndPort();
+
+    String src = "/data";
+    Path destPath = new Path("hdfs://" + nnAddress + "/data");
+    String user = "user1";
+    String group = "group1";
+    String clusterName = "ClusterX";
+
+    // 0.mkdir destPath
+    hdfs.mkdirs(destPath);
+    // 1.set owner
+    hdfs.setOwner(destPath, user, group);
+    // 2.set viewFs mapping
+    admin.getConf().set("fs.viewfs.mounttable.ClusterX.link." + src, destPath.toString());

Review comment:
       Can we add some documentation to HDFSRouterFederation.md?
   With pointers to viewfs, etc.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());
+        continue;
+      }
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      boolean added = addMount(
+          mount[1], nss, path.toUri().getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path));
+      if (added) {
+        System.out.println("added mount point " + mount[1]);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  public ACLEntity getACLEntityFormHdfsPath(Path path){

Review comment:
       static

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());
+        continue;
+      }
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      boolean added = addMount(
+          mount[1], nss, path.toUri().getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path));
+      if (added) {
+        System.out.println("added mount point " + mount[1]);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  public ACLEntity getACLEntityFormHdfsPath(Path path){
+    String owner = null;
+    String group = null;
+    FsPermission mode = null;
+    try {
+      FileSystem fs = path.getFileSystem(getConf());
+      if (fs.exists(path)) {
+        FileStatus fileStatus = fs.getFileStatus(path);
+        owner = fileStatus.getOwner();
+        group = fileStatus.getGroup();
+        mode = fileStatus.getPermission();
+      }
+    } catch (IOException e) {
+      System.out.println("Exception encountered " + e);

Review comment:
       .err.?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r628667273



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
##########
@@ -241,6 +241,22 @@ Mount table permission can be set by following command:
 
 The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
 
+#### Init ViewFs To Router
+Router supports initializing the ViewFS mount point to the Router. The mapping directory protocol of ViewFS must be HDFS, and the initializer only supports one-to-one mapping.
+
+For example, use the following viewfs to configure the initial mount table to the router.
+
+    <configuration>
+      <property>
+        <name>fs.viewfs.mounttable.ClusterX.link./data</name>
+        <value>hdfs://nn1-clusterx.example.com:8020/data</value>
+      </property>
+    </configuration>
+
+The ViewFS mount table can be initialized to the Router by using the following command:
+
+    [hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -initViewFsToMountTable ClusterX

Review comment:
       Do you mean that there are mount points for multiple clusters, can the user import all of them without specifying the cluster name? For example, the following configuration.
   
   <configuration>
     <property>
       <name>fs.viewfs.mounttable.ClusterX.link./data</name>
       <value>hdfs://nn1-clusterx.example.com:8020/data</value>
     </property>
     <property>
       <name>fs.viewfs.mounttable.ClusterY.link./project</name>
       <value>hdfs://nn2-clusterx.example.com:8020/project</value>
     </property>
   </configuration>
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-833463912


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 46s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  64m 40s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 44s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 40s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 28s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 41s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m  0s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 20s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m 39s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 34s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 34s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 28s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 28s |  |  the patch passed  |
   | -1 :x: |  blanks  |   0m  0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/2/artifact/out/blanks-eol.txt) |  The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   0m 16s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 47s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 17s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 53s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | -1 :x: |  unit  |  22m 14s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 30s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 134m  8s |  |  |
   
   
   | Reason | Tests |
   |-------:|:------|
   | Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/2/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 48f24d232040 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 9a7c9f1cee89bd5ee8def106078b925b5d0f65b3 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/2/testReport/ |
   | Max. process+thread count | 2329 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/2/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-840409129


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 49s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  1s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 25s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 46s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 33s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 44s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 51s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 22s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  9s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 28s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 28s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 15s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/11/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 46s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 19s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 39s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  22m  8s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 29s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 102m 23s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/11/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux d078a88c4da8 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / bc5089e4eaf5828f5ddb48cbc6e69219ad18702c |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/11/testReport/ |
   | Max. process+thread count | 2296 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/11/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] zhuxiangyi commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
zhuxiangyi commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627895751



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,6 +1054,65 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
+  
+  /**
+   * initViewFsToMountTable.
+   * @param clusterName The specified cluster to initialize.
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String clusterName)
+      throws IOException {
+    // fs.viewfs.mounttable.ClusterX.link./data
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.isEmpty()) {
+      System.out.println("There is no ViewFs mapping to initialize.");
+      return true;
+    }
+    for (Entry<String, String> entry : viewFsMap.entrySet()) {
+      Path path = new Path(entry.getValue());
+      DestinationOrder order = DestinationOrder.HASH;
+      String[] mount = entry.getKey().split(
+              clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".");
+      if (mount.length < 2) {
+        System.out.println("Added Mount Point failed " + entry.getKey());
+        continue;
+      }
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      boolean added = addMount(
+          mount[1], nss, path.toUri().getPath(), false,
+          false, order, getACLEntityFormHdfsPath(path));
+      if (added) {
+        System.out.println("added mount point " + mount[1]);
+      }
+    }
+    return true;
+  }
+
+  /**
+   * Returns ACLEntity according to a HDFS pat.
+   * @param path A path of HDFS.
+   */
+  public ACLEntity getACLEntityFormHdfsPath(Path path){
+    String owner = null;
+    String group = null;
+    FsPermission mode = null;
+    try {
+      FileSystem fs = path.getFileSystem(getConf());
+      if (fs.exists(path)) {
+        FileStatus fileStatus = fs.getFileStatus(path);
+        owner = fileStatus.getOwner();
+        group = fileStatus.getGroup();
+        mode = fileStatus.getPermission();
+      }
+    } catch (IOException e) {
+      System.out.println("Exception encountered " + e);

Review comment:
       Yes,thank you for reminding me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-834228785


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 50s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 50s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 38s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 23s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 38s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 51s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 14s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  1s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 27s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 27s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 16s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/4/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 45s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 17s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 41s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  21m 57s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 30s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 101m 58s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/4/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux 411e1f12e5ff 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 7fa083ac87f720663afb7c52953c830e45cefd4a |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/4/testReport/ |
   | Max. process+thread count | 2336 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/4/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] goiri commented on a change in pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#discussion_r627061466



##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {

Review comment:
       Pass the cluster directly and do the check outside.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =

Review comment:
       Put a comment with an example of the path.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.size() == 0) {

Review comment:
       .empty()

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.size() == 0) {
+      System.out.println("Please check the cluster name and veiwfs " +

Review comment:
       Typo

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.size() == 0) {
+      System.out.println("Please check the cluster name and veiwfs " +
+          "configuration.");
+    }
+    for (String key : viewFsMap.keySet()) {
+      Path path = new Path(viewFsMap.get(key));
+      String owner = null;

Review comment:
       Extract this into a method that generates the ACLEntity right away.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.size() == 0) {
+      System.out.println("Please check the cluster name and veiwfs " +
+          "configuration.");
+    }
+    for (String key : viewFsMap.keySet()) {
+      Path path = new Path(viewFsMap.get(key));
+      String owner = null;
+      String group = null;
+      FsPermission mode = null;
+      try {
+        FileSystem fs = path.getFileSystem(getConf());
+        if (fs.exists(path)) {
+          FileStatus fileStatus = fs.getFileStatus(path);
+          owner = fileStatus.getOwner();
+          group = fileStatus.getGroup();
+          mode = fileStatus.getPermission();
+        }
+      } catch (Exception e) {
+        LOG.warn("Exception encountered", e);
+      }
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount =
+          key.split(clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".")[1];
+      String dest = path.toUri().getPath();
+      String[] nss = new String[]{path.toUri().getAuthority()};
+      addMount(mount, nss, dest, false, false, order,

Review comment:
       Log the mount added.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.size() == 0) {
+      System.out.println("Please check the cluster name and veiwfs " +
+          "configuration.");
+    }
+    for (String key : viewFsMap.keySet()) {
+      Path path = new Path(viewFsMap.get(key));
+      String owner = null;
+      String group = null;
+      FsPermission mode = null;
+      try {
+        FileSystem fs = path.getFileSystem(getConf());
+        if (fs.exists(path)) {
+          FileStatus fileStatus = fs.getFileStatus(path);
+          owner = fileStatus.getOwner();
+          group = fileStatus.getGroup();
+          mode = fileStatus.getPermission();
+        }
+      } catch (Exception e) {
+        LOG.warn("Exception encountered", e);

Review comment:
       Inconsistent to switch between System.out and LOG, choose one.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -132,7 +136,7 @@ private String getUsage(String cmd) {
           {"-add", "-update", "-rm", "-ls", "-getDestination", "-setQuota",
               "-setStorageTypeQuota", "-clrQuota", "-clrStorageTypeQuota",
               "-safemode", "-nameservice", "-getDisabledNameservices",
-              "-refresh", "-refreshRouterArgs",
+              "-refresh", "-initViewFsToMountTable", "-refreshRouterArgs",

Review comment:
       Let's add a unit test.

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.size() == 0) {
+      System.out.println("Please check the cluster name and veiwfs " +
+          "configuration.");
+    }
+    for (String key : viewFsMap.keySet()) {

Review comment:
       entrySet() given that you are using both key and value

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -1035,7 +1049,56 @@ private boolean updateQuota(String mount, long nsQuota, long ssQuota)
         .updateMountTableEntry(updateRequest);
     return updateResponse.getStatus();
   }
-
+  
+  /**
+   * initViewFsToMountTable.
+   *
+   * @param parameters The specified cluster to initialize.
+   * @param i Index in the parameters
+   * @return If the quota was updated.
+   * @throws IOException Error adding the mount point.
+   */
+  public boolean initViewFsToMountTable(String[] parameters, int i) throws IOException {
+    String clusterName = parameters[i++];
+    if (clusterName == null) {
+      System.out.println("Please enter the cluster name.");
+      return false;
+    }
+    final String mountTablePrefix =
+        Constants.CONFIG_VIEWFS_PREFIX + "." + clusterName + "." +
+            Constants.CONFIG_VIEWFS_LINK + "./";
+    Map<String, String> viewFsMap = getConf().getValByRegex(mountTablePrefix);
+    if (viewFsMap.size() == 0) {
+      System.out.println("Please check the cluster name and veiwfs " +
+          "configuration.");
+    }
+    for (String key : viewFsMap.keySet()) {
+      Path path = new Path(viewFsMap.get(key));
+      String owner = null;
+      String group = null;
+      FsPermission mode = null;
+      try {
+        FileSystem fs = path.getFileSystem(getConf());
+        if (fs.exists(path)) {
+          FileStatus fileStatus = fs.getFileStatus(path);
+          owner = fileStatus.getOwner();
+          group = fileStatus.getGroup();
+          mode = fileStatus.getPermission();
+        }
+      } catch (Exception e) {
+        LOG.warn("Exception encountered", e);
+      }
+      DestinationOrder order = DestinationOrder.HASH;
+      String mount =
+          key.split(clusterName + "." + Constants.CONFIG_VIEWFS_LINK + ".")[1];
+      String dest = path.toUri().getPath();

Review comment:
       Extract path.toUri()

##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/tools/federation/RouterAdmin.java
##########
@@ -384,7 +391,14 @@ public int run(String[] argv) throws Exception {
         getDisabledNameservices();
       } else if ("-refresh".equals(cmd)) {
         refresh(address);
-      } else if ("-refreshRouterArgs".equals(cmd)) {
+      } else if ("-initViewFsToMountTable".equals(cmd)) {
+        if (initViewFsToMountTable(argv, i)) {

Review comment:
       Can we extract argv[i] (and whatever) here instead of passing a generic array and an index?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-840394715


   :confetti_ball: **+1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 55s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  1s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  36m 20s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 40s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 41s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 25s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 50s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 44s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   1m  4s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 38s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  19m 25s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 36s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 41s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 41s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 35s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 35s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks issues.  |
   | -0 :warning: |  checkstyle  |   0m 18s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/9/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 38s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 38s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 53s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 45s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  20m  1s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  25m 39s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 35s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 116m  3s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/9/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux 78eb2344543e 4.15.0-142-generic #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 93dbc90ef7d3c3ee2fee43a84706248a2f3db9b3 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/9/testReport/ |
   | Max. process+thread count | 2517 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/9/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


[GitHub] [hadoop] hadoop-yetus commented on pull request #2981: HDFS-16008. RBF: Tool to initialize ViewFS Mapping to Router

Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2981:
URL: https://github.com/apache/hadoop/pull/2981#issuecomment-834155907


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |:----:|----------:|--------:|:--------:|:-------:|
   | +0 :ok: |  reexec  |   0m 48s |  |  Docker mode activated.  |
   |||| _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +0 :ok: |  markdownlint  |   0m  0s |  |  markdownlint was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to include 1 new or modified test files.  |
   |||| _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  33m 38s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 41s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  compile  |   0m 33s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  checkstyle  |   0m 22s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 39s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 37s |  |  trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 51s |  |  trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 15s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  17m  2s |  |  branch has no errors when building and testing our client artifacts.  |
   |||| _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 32s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 34s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javac  |   0m 34s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 28s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  javac  |   0m 28s |  |  the patch passed  |
   | -1 :x: |  blanks  |   0m  0s | [/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/3/artifact/out/blanks-eol.txt) |  The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply  |
   | -0 :warning: |  checkstyle  |   0m 16s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt) |  hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)  |
   | +1 :green_heart: |  mvnsite  |   0m 31s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 31s |  |  the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  |  the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10  |
   | +1 :green_heart: |  spotbugs  |   1m 19s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 46s |  |  patch has no errors when building and testing our client artifacts.  |
   |||| _ Other Tests _ |
   | +1 :green_heart: |  unit  |  22m  5s |  |  hadoop-hdfs-rbf in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 29s |  |  The patch does not generate ASF License warnings.  |
   |  |   | 102m  1s |  |  |
   
   
   | Subsystem | Report/Notes |
   |----------:|:-------------|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/3/artifact/out/Dockerfile |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2981 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell markdownlint |
   | uname | Linux 19992cf6eb20 4.15.0-128-generic #131-Ubuntu SMP Wed Dec 9 06:57:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / e0b806f234d7217010f9ac1d804f08b77025bc70 |
   | Default Java | Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 |
   |  Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/3/testReport/ |
   | Max. process+thread count | 2327 (vs. ulimit of 5500) |
   | modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project/hadoop-hdfs-rbf |
   | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2981/3/console |
   | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
   | Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
   
   
   This message was automatically generated.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org