You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/02/06 09:16:45 UTC
[GitHub] [hadoop] aajisaka opened a new pull request #1832: HDFS-13989. RBF:
Add FSCK to the Router
aajisaka opened a new pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832
JIRA: https://issues.apache.org/jira/browse/HDFS-13989
* Refactor the code
* (hopefully) support https
* Add end-to-end unit test
Note that RouterFsck does not consider the mount table for now. The request is always forwarded to all the active downstream namenodes. The path of the fsck arguments is not changed when forwarding the request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1832: HDFS-13989. RBF:
Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#issuecomment-582886312
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 7s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 7s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 18m 24s | trunk passed |
| +1 :green_heart: | compile | 3m 23s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 55s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 49s | trunk passed |
| +1 :green_heart: | shadedclient | 15m 23s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 18s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 51s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 34s | the patch passed |
| +1 :green_heart: | compile | 3m 18s | the patch passed |
| +1 :green_heart: | javac | 3m 18s | the patch passed |
| -0 :warning: | checkstyle | 0m 48s | hadoop-hdfs-project: The patch generated 7 new + 1 unchanged - 0 fixed = 8 total (was 1) |
| +1 :green_heart: | mvnsite | 1m 36s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 13m 3s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 15s | the patch passed |
| -1 :x: | findbugs | 1m 12s | hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
||| _ Other Tests _ |
| -1 :x: | unit | 113m 34s | hadoop-hdfs in the patch failed. |
| -1 :x: | unit | 8m 16s | hadoop-hdfs-rbf in the patch failed. |
| +1 :green_heart: | asflicense | 0m 44s | The patch does not generate ASF License warnings. |
| | | 198m 19s | |
| Reason | Tests |
|-------:|:------|
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| | Found reliance on default encoding in org.apache.hadoop.hdfs.server.federation.router.RouterFsck.remoteFsck(MembershipState):in org.apache.hadoop.hdfs.server.federation.router.RouterFsck.remoteFsck(MembershipState): new java.io.InputStreamReader(InputStream) At RouterFsck.java:[line 128] |
| Failed junit tests | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.hdfs.TestDFSInputStream |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| | hadoop.hdfs.server.federation.router.TestRouterClientRejectOverload |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1832 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 99e28cac3da8 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 314e2f9 |
| Default Java | 1.8.0_242 |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/1/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt |
| findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/1/artifact/out/new-findbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/1/testReport/ |
| Max. process+thread count | 4046 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/1/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on issue #1832: HDFS-13989. RBF: Add FSCK
to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#issuecomment-583562603
Yetus does not look very happy but it seems unrelated.
Please double check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1832: HDFS-13989. RBF:
Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#issuecomment-584002175
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 46s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 14s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 0s | trunk passed |
| +1 :green_heart: | compile | 3m 32s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 56s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 25s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 12s | trunk passed |
| +0 :ok: | spotbugs | 1m 7s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 2s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 37s | the patch passed |
| +1 :green_heart: | compile | 3m 31s | the patch passed |
| +1 :green_heart: | javac | 3m 31s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 51s | the patch passed |
| +1 :green_heart: | mvnsite | 1m 40s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 15m 24s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 7s | the patch passed |
| +1 :green_heart: | findbugs | 4m 17s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 114m 20s | hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 9m 54s | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 238m 20s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.federation.router.TestRouterFaultTolerant |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1832 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 074a86392577 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d5467d2 |
| Default Java | 1.8.0_242 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/3/testReport/ |
| Max. process+thread count | 2924 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/3/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r377217468
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterFsckServlet.java
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.InetAddress;
+import java.security.PrivilegedExceptionAction;
+import java.util.Map;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.common.JspHelper;
+import org.apache.hadoop.security.UserGroupInformation;
+
+/**
+ * This class is used in Namesystem's web server to do fsck on namenode.
+ */
+@InterfaceAudience.Private
+public class RouterFsckServlet extends HttpServlet {
+ /** for java.io.Serializable. */
+ private static final long serialVersionUID = 1L;
+
+ public static final String SERVLET_NAME = "fsck";
+ public static final String PATH_SPEC = "/fsck";
+
+ /** Handle fsck request. */
+ @Override
+ public void doGet(HttpServletRequest request, HttpServletResponse response)
+ throws IOException {
+ final Map<String, String[]> pmap = request.getParameterMap();
+ final PrintWriter out = response.getWriter();
+ final InetAddress remoteAddress =
+ InetAddress.getByName(request.getRemoteAddr());
+ final ServletContext context = getServletContext();
+ final Configuration conf = RouterHttpServer.getConfFromContext(context);
+ final UserGroupInformation ugi = getUGI(request, conf);
+ try {
+ ugi.doAs((PrivilegedExceptionAction<Object>) () -> {
+ Router router = RouterHttpServer.getRouterFromContext(context);
+ new RouterFsck(router, pmap, out, remoteAddress).fsck();
+ return null;
+ });
+ } catch (InterruptedException e) {
+ response.sendError(400, e.getMessage());
Review comment:
Or HttpStatus.SC_BAD_REQUEST if you want to follow the same as the unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376093205
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterHttpServer.java
##########
@@ -118,10 +118,14 @@ protected void serviceStop() throws Exception {
private static void setupServlets(
HttpServer2 httpServer, Configuration conf) {
- // TODO Add servlets for FSCK, etc
+ // TODO: Add more required servlets
Review comment:
I think we can remove this now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on issue #1832: HDFS-13989. RBF: Add
FSCK to the Router
Posted by GitBox <gi...@apache.org>.
aajisaka commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#issuecomment-585497638
Merged. Thanks @goiri for your reviews.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1832: HDFS-13989. RBF:
Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#issuecomment-583404297
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 27m 37s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 44s | trunk passed |
| +1 :green_heart: | compile | 3m 39s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 57s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 52s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 21s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 16s | trunk passed |
| +0 :ok: | spotbugs | 1m 16s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 21s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | the patch passed |
| -1 :x: | compile | 4m 23s | hadoop-hdfs-project in the patch failed. |
| -1 :x: | javac | 4m 23s | hadoop-hdfs-project in the patch failed. |
| -0 :warning: | checkstyle | 1m 15s | hadoop-hdfs-project: The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1) |
| +1 :green_heart: | mvnsite | 2m 29s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 21m 8s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 45s | the patch passed |
| +1 :green_heart: | findbugs | 6m 5s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 98m 49s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 8m 36s | hadoop-hdfs-rbf in the patch passed. |
| -1 :x: | asflicense | 0m 36s | The patch generated 43 ASF License warnings. |
| | | 227m 15s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.fs.viewfs.TestViewFsHdfs |
| | hadoop.hdfs.server.diskbalancer.command.TestDiskBalancerCommand |
| | hadoop.hdfs.TestMiniDFSCluster |
| | hadoop.fs.viewfs.TestViewFileSystemAtHdfsRoot |
| | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| | hadoop.fs.viewfs.TestViewFileSystemWithAcls |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.fs.viewfs.TestViewFileSystemHdfs |
| | hadoop.fs.TestEnhancedByteBufferAccess |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.server.diskbalancer.TestDiskBalancerWithMockMover |
| | hadoop.hdfs.server.diskbalancer.TestConnectors |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1832 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 967dc8cb56b9 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fafe78f |
| Default Java | 1.8.0_242 |
| compile | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/artifact/out/patch-compile-hadoop-hdfs-project.txt |
| javac | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/artifact/out/patch-compile-hadoop-hdfs-project.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/testReport/ |
| asflicense | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 4229 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376093596
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterFsck.java
##########
@@ -0,0 +1,172 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.RouterConfigBuilder;
+import org.apache.hadoop.hdfs.server.federation.StateStoreDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.resolver.MountTableManager;
+import org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.AddMountTableEntryRequest;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.AddMountTableEntryResponse;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetMountTableEntriesRequest;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetMountTableEntriesResponse;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.RemoveMountTableEntryRequest;
+import org.apache.hadoop.hdfs.server.federation.store.records.MountTable;
+import org.apache.http.HttpStatus;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+
+/**
+ * End-to-end tests for fsck via DFSRouter
+ */
+public class TestRouterFsck {
+
+ private static StateStoreDFSCluster cluster;
+ private static MiniRouterDFSCluster.RouterContext routerContext;
+ private static MountTableResolver mountTable;
+ private static FileSystem routerFs;
+ private static InetSocketAddress webAddress;
+
+ @BeforeClass
+ public static void globalSetUp() throws Exception {
+ // Build and start a federated cluster
+ cluster = new StateStoreDFSCluster(false, 2);
+ Configuration conf = new RouterConfigBuilder()
+ .stateStore()
+ .admin()
+ .rpc()
+ .http()
+ .build();
+ cluster.addRouterOverrides(conf);
+ cluster.startCluster();
+ cluster.startRouters();
+ cluster.waitClusterUp();
+
+ // Get the end points
+ routerContext = cluster.getRandomRouter();
+ routerFs = routerContext.getFileSystem();
+ Router router = routerContext.getRouter();
+ mountTable = (MountTableResolver) router.getSubclusterResolver();
+ webAddress = router.getHttpServerAddress();
+ Assert.assertNotNull(webAddress);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+ if (cluster != null) {
+ cluster.stopRouter(routerContext);
+ cluster.shutdown();
+ cluster = null;
+ }
+ }
+
+ @After
+ public void clearMountTable() throws IOException {
+ RouterClient client = routerContext.getAdminClient();
+ MountTableManager mountTableManager = client.getMountTableManager();
+ GetMountTableEntriesRequest req1 =
+ GetMountTableEntriesRequest.newInstance("/");
+ GetMountTableEntriesResponse response =
+ mountTableManager.getMountTableEntries(req1);
+ for (MountTable entry : response.getEntries()) {
+ RemoveMountTableEntryRequest req2 =
+ RemoveMountTableEntryRequest.newInstance(entry.getSourcePath());
+ mountTableManager.removeMountTableEntry(req2);
+ }
+ }
+
+ private boolean addMountTable(final MountTable entry) throws IOException {
+ RouterClient client = routerContext.getAdminClient();
+ MountTableManager mountTableManager = client.getMountTableManager();
+ AddMountTableEntryRequest addRequest =
+ AddMountTableEntryRequest.newInstance(entry);
+ AddMountTableEntryResponse addResponse =
+ mountTableManager.addMountTableEntry(addRequest);
+ // Reload the Router cache
+ mountTable.loadCache(true);
+ return addResponse.getStatus();
+ }
+
+ @Test
+ public void testFsck() throws Exception {
+ MountTable addEntry = MountTable.newInstance("/testdir",
+ Collections.singletonMap("ns0", "/testdir"));
+ Assert.assertTrue(addMountTable(addEntry));
+ addEntry = MountTable.newInstance("/testdir2",
+ Collections.singletonMap("ns1", "/testdir2"));
+ Assert.assertTrue(addMountTable(addEntry));
+ // create 1 file on ns0
+ routerFs.createNewFile(new Path("/testdir/testfile"));
+ // create 3 files on ns1
+ routerFs.createNewFile(new Path("/testdir2/testfile2"));
+ routerFs.createNewFile(new Path("/testdir2/testfile3"));
+ routerFs.createNewFile(new Path("/testdir2/testfile4"));
+
+ try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
+ // TODO: support https
+ HttpGet httpGet = new HttpGet("http://" + webAddress.getHostName() +
+ ":" + webAddress.getPort() + "/fsck");
+ try (CloseableHttpResponse httpResponse = httpClient.execute(httpGet)) {
+ Assert.assertEquals(HttpStatus.SC_OK,
+ httpResponse.getStatusLine().getStatusCode());
+ String out = EntityUtils.toString(
+ httpResponse.getEntity(), StandardCharsets.UTF_8);
+ System.out.println(out);
Review comment:
LOG?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376094092
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterFsck.java
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.PrintWriter;
+import java.net.InetAddress;
+import java.net.URL;
+import java.net.URLConnection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.hdfs.server.federation.resolver.FederationNamenodeServiceState;
+import org.apache.hadoop.hdfs.server.federation.store.MembershipStore;
+import org.apache.hadoop.hdfs.server.federation.store.StateStoreService;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetNamenodeRegistrationsRequest;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetNamenodeRegistrationsResponse;
+import org.apache.hadoop.hdfs.server.federation.store.records.MembershipState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Wrapper for the Router to offer the Namenode FSCK.
+ */
+@InterfaceAudience.Private
+public class RouterFsck {
+
+ public static final Logger LOG =
+ LoggerFactory.getLogger(RouterFsck.class.getName());
+
+ private final Router router;
+ private final InetAddress remoteAddress;
+ private final PrintWriter out;
+ private final Map<String, String[]> pmap;
+
+ public RouterFsck(Router router, Map<String, String[]> pmap,
+ PrintWriter out, InetAddress remoteAddress) {
+ this.router = router;
+ this.remoteAddress = remoteAddress;
+ this.out = out;
+ this.pmap = pmap;
+ }
+
+ public void fsck() {
+ final long startTime = Time.monotonicNow();
+ try {
+ String msg = "Federated FSCK started by " +
+ UserGroupInformation.getCurrentUser() + " from " + remoteAddress +
+ " at " + new Date();
+ LOG.info(msg);
+ out.println(msg);
+
+ // Check each Namenode in the federation
+ StateStoreService stateStore = router.getStateStore();
+ MembershipStore membership =
+ stateStore.getRegisteredRecordStore(MembershipStore.class);
+ GetNamenodeRegistrationsRequest request =
+ GetNamenodeRegistrationsRequest.newInstance();
+ GetNamenodeRegistrationsResponse response =
+ membership.getNamenodeRegistrations(request);
+ List<MembershipState> memberships = response.getNamenodeMemberships();
+ Collections.sort(memberships);
+ for (MembershipState nn : memberships) {
+ if (nn.getState() == FederationNamenodeServiceState.ACTIVE) {
+ try {
+ String webAddress = nn.getWebAddress();
+ out.write("Checking " + nn + " at " + webAddress + "\n");
Review comment:
Can we check for this with the webAddress in the unit test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376093554
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterFsck.java
##########
@@ -0,0 +1,172 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import java.net.InetSocketAddress;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.server.federation.MiniRouterDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.RouterConfigBuilder;
+import org.apache.hadoop.hdfs.server.federation.StateStoreDFSCluster;
+import org.apache.hadoop.hdfs.server.federation.resolver.MountTableManager;
+import org.apache.hadoop.hdfs.server.federation.resolver.MountTableResolver;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.AddMountTableEntryRequest;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.AddMountTableEntryResponse;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetMountTableEntriesRequest;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetMountTableEntriesResponse;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.RemoveMountTableEntryRequest;
+import org.apache.hadoop.hdfs.server.federation.store.records.MountTable;
+import org.apache.http.HttpStatus;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.util.EntityUtils;
+
+/**
+ * End-to-end tests for fsck via DFSRouter
+ */
+public class TestRouterFsck {
+
+ private static StateStoreDFSCluster cluster;
+ private static MiniRouterDFSCluster.RouterContext routerContext;
+ private static MountTableResolver mountTable;
+ private static FileSystem routerFs;
+ private static InetSocketAddress webAddress;
+
+ @BeforeClass
+ public static void globalSetUp() throws Exception {
+ // Build and start a federated cluster
+ cluster = new StateStoreDFSCluster(false, 2);
+ Configuration conf = new RouterConfigBuilder()
+ .stateStore()
+ .admin()
+ .rpc()
+ .http()
+ .build();
+ cluster.addRouterOverrides(conf);
+ cluster.startCluster();
+ cluster.startRouters();
+ cluster.waitClusterUp();
+
+ // Get the end points
+ routerContext = cluster.getRandomRouter();
+ routerFs = routerContext.getFileSystem();
+ Router router = routerContext.getRouter();
+ mountTable = (MountTableResolver) router.getSubclusterResolver();
+ webAddress = router.getHttpServerAddress();
+ Assert.assertNotNull(webAddress);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+ if (cluster != null) {
+ cluster.stopRouter(routerContext);
+ cluster.shutdown();
+ cluster = null;
+ }
+ }
+
+ @After
+ public void clearMountTable() throws IOException {
+ RouterClient client = routerContext.getAdminClient();
+ MountTableManager mountTableManager = client.getMountTableManager();
+ GetMountTableEntriesRequest req1 =
+ GetMountTableEntriesRequest.newInstance("/");
+ GetMountTableEntriesResponse response =
+ mountTableManager.getMountTableEntries(req1);
+ for (MountTable entry : response.getEntries()) {
+ RemoveMountTableEntryRequest req2 =
+ RemoveMountTableEntryRequest.newInstance(entry.getSourcePath());
+ mountTableManager.removeMountTableEntry(req2);
+ }
+ }
+
+ private boolean addMountTable(final MountTable entry) throws IOException {
+ RouterClient client = routerContext.getAdminClient();
+ MountTableManager mountTableManager = client.getMountTableManager();
+ AddMountTableEntryRequest addRequest =
+ AddMountTableEntryRequest.newInstance(entry);
+ AddMountTableEntryResponse addResponse =
+ mountTableManager.addMountTableEntry(addRequest);
+ // Reload the Router cache
+ mountTable.loadCache(true);
+ return addResponse.getStatus();
+ }
+
+ @Test
+ public void testFsck() throws Exception {
+ MountTable addEntry = MountTable.newInstance("/testdir",
+ Collections.singletonMap("ns0", "/testdir"));
+ Assert.assertTrue(addMountTable(addEntry));
+ addEntry = MountTable.newInstance("/testdir2",
+ Collections.singletonMap("ns1", "/testdir2"));
+ Assert.assertTrue(addMountTable(addEntry));
+ // create 1 file on ns0
+ routerFs.createNewFile(new Path("/testdir/testfile"));
+ // create 3 files on ns1
+ routerFs.createNewFile(new Path("/testdir2/testfile2"));
+ routerFs.createNewFile(new Path("/testdir2/testfile3"));
+ routerFs.createNewFile(new Path("/testdir2/testfile4"));
+
+ try (CloseableHttpClient httpClient = HttpClients.createDefault()) {
+ // TODO: support https
+ HttpGet httpGet = new HttpGet("http://" + webAddress.getHostName() +
+ ":" + webAddress.getPort() + "/fsck");
+ try (CloseableHttpResponse httpResponse = httpClient.execute(httpGet)) {
+ Assert.assertEquals(HttpStatus.SC_OK,
+ httpResponse.getStatusLine().getStatusCode());
+ String out = EntityUtils.toString(
+ httpResponse.getEntity(), StandardCharsets.UTF_8);
+ System.out.println(out);
+ Assert.assertTrue(out.contains("Federated FSCK started"));
Review comment:
Let's make an static import for assertTrue and add a message with the out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on issue #1832: HDFS-13989. RBF: Add
FSCK to the Router
Posted by GitBox <gi...@apache.org>.
aajisaka commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#issuecomment-583946905
@goiri Thank you for your review. Fixed checkstyle issues.
> Yetus does not look very happy but it seems unrelated.
Agreed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
aajisaka commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376213756
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterFsck.java
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.PrintWriter;
+import java.net.InetAddress;
+import java.net.URL;
+import java.net.URLConnection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.hdfs.server.federation.resolver.FederationNamenodeServiceState;
+import org.apache.hadoop.hdfs.server.federation.store.MembershipStore;
+import org.apache.hadoop.hdfs.server.federation.store.StateStoreService;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetNamenodeRegistrationsRequest;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetNamenodeRegistrationsResponse;
+import org.apache.hadoop.hdfs.server.federation.store.records.MembershipState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Wrapper for the Router to offer the Namenode FSCK.
+ */
+@InterfaceAudience.Private
+public class RouterFsck {
+
+ public static final Logger LOG =
+ LoggerFactory.getLogger(RouterFsck.class.getName());
+
+ private final Router router;
+ private final InetAddress remoteAddress;
+ private final PrintWriter out;
+ private final Map<String, String[]> pmap;
+
+ public RouterFsck(Router router, Map<String, String[]> pmap,
+ PrintWriter out, InetAddress remoteAddress) {
+ this.router = router;
+ this.remoteAddress = remoteAddress;
+ this.out = out;
+ this.pmap = pmap;
+ }
+
+ public void fsck() {
+ final long startTime = Time.monotonicNow();
+ try {
+ String msg = "Federated FSCK started by " +
+ UserGroupInformation.getCurrentUser() + " from " + remoteAddress +
+ " at " + new Date();
+ LOG.info(msg);
+ out.println(msg);
+
+ // Check each Namenode in the federation
+ StateStoreService stateStore = router.getStateStore();
+ MembershipStore membership =
+ stateStore.getRegisteredRecordStore(MembershipStore.class);
+ GetNamenodeRegistrationsRequest request =
+ GetNamenodeRegistrationsRequest.newInstance();
+ GetNamenodeRegistrationsResponse response =
+ membership.getNamenodeRegistrations(request);
+ List<MembershipState> memberships = response.getNamenodeMemberships();
+ Collections.sort(memberships);
+ for (MembershipState nn : memberships) {
+ if (nn.getState() == FederationNamenodeServiceState.ACTIVE) {
+ try {
+ String webAddress = nn.getWebAddress();
+ out.write("Checking " + nn + " at " + webAddress + "\n");
Review comment:
Added the check in the unit test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r377216849
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterFsckServlet.java
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.InetAddress;
+import java.security.PrivilegedExceptionAction;
+import java.util.Map;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.common.JspHelper;
+import org.apache.hadoop.security.UserGroupInformation;
+
+/**
+ * This class is used in Namesystem's web server to do fsck on namenode.
+ */
+@InterfaceAudience.Private
+public class RouterFsckServlet extends HttpServlet {
+ /** for java.io.Serializable. */
+ private static final long serialVersionUID = 1L;
+
+ public static final String SERVLET_NAME = "fsck";
+ public static final String PATH_SPEC = "/fsck";
+
+ /** Handle fsck request. */
+ @Override
+ public void doGet(HttpServletRequest request, HttpServletResponse response)
+ throws IOException {
+ final Map<String, String[]> pmap = request.getParameterMap();
+ final PrintWriter out = response.getWriter();
+ final InetAddress remoteAddress =
+ InetAddress.getByName(request.getRemoteAddr());
+ final ServletContext context = getServletContext();
+ final Configuration conf = RouterHttpServer.getConfFromContext(context);
+ final UserGroupInformation ugi = getUGI(request, conf);
+ try {
+ ugi.doAs((PrivilegedExceptionAction<Object>) () -> {
+ Router router = RouterHttpServer.getRouterFromContext(context);
+ new RouterFsck(router, pmap, out, remoteAddress).fsck();
+ return null;
+ });
+ } catch (InterruptedException e) {
+ response.sendError(400, e.getMessage());
Review comment:
HttpURLConnection.HTTP_BAD_REQUEST in java.net.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376562517
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterFsckServlet.java
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.InetAddress;
+import java.security.PrivilegedExceptionAction;
+import java.util.Map;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.common.JspHelper;
+import org.apache.hadoop.security.UserGroupInformation;
+
+/**
+ * This class is used in Namesystem's web server to do fsck on namenode.
+ */
+@InterfaceAudience.Private
+public class RouterFsckServlet extends HttpServlet {
+ /** for java.io.Serializable */
Review comment:
First sentence should end with a period. [JavadocStyle]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] aajisaka merged pull request #1832: HDFS-13989. RBF: Add
FSCK to the Router
Posted by GitBox <gi...@apache.org>.
aajisaka merged pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376562407
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterFsck.java
##########
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.PrintWriter;
+import java.net.InetAddress;
+import java.net.URL;
+import java.net.URLConnection;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.hdfs.server.federation.resolver.FederationNamenodeServiceState;
+import org.apache.hadoop.hdfs.server.federation.store.MembershipStore;
+import org.apache.hadoop.hdfs.server.federation.store.StateStoreService;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetNamenodeRegistrationsRequest;
+import org.apache.hadoop.hdfs.server.federation.store.protocol.GetNamenodeRegistrationsResponse;
+import org.apache.hadoop.hdfs.server.federation.store.records.MembershipState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.util.Time;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Wrapper for the Router to offer the Namenode FSCK.
+ */
+@InterfaceAudience.Private
+public class RouterFsck {
+
+ public static final Logger LOG =
+ LoggerFactory.getLogger(RouterFsck.class.getName());
+
+ private final Router router;
+ private final InetAddress remoteAddress;
+ private final PrintWriter out;
+ private final Map<String, String[]> pmap;
+
+ public RouterFsck(Router router, Map<String, String[]> pmap,
+ PrintWriter out, InetAddress remoteAddress) {
+ this.router = router;
+ this.remoteAddress = remoteAddress;
+ this.out = out;
+ this.pmap = pmap;
+ }
+
+ public void fsck() {
+ final long startTime = Time.monotonicNow();
+ try {
+ String msg = "Federated FSCK started by " +
+ UserGroupInformation.getCurrentUser() + " from " + remoteAddress +
+ " at " + new Date();
+ LOG.info(msg);
+ out.println(msg);
+
+ // Check each Namenode in the federation
+ StateStoreService stateStore = router.getStateStore();
+ MembershipStore membership =
+ stateStore.getRegisteredRecordStore(MembershipStore.class);
+ GetNamenodeRegistrationsRequest request =
+ GetNamenodeRegistrationsRequest.newInstance();
+ GetNamenodeRegistrationsResponse response =
+ membership.getNamenodeRegistrations(request);
+ List<MembershipState> memberships = response.getNamenodeMemberships();
+ Collections.sort(memberships);
+ for (MembershipState nn : memberships) {
+ if (nn.getState() == FederationNamenodeServiceState.ACTIVE) {
+ try {
+ String webAddress = nn.getWebAddress();
+ out.write("Checking " + nn + " at " + webAddress + "\n");
+ remoteFsck(nn);
+ } catch (IOException ioe) {
+ out.println("Cannot query " + nn + ": " + ioe.getMessage() + "\n");
+ }
+ }
+ }
+
+ out.println("Federated FSCK ended at " + new Date() + " in "
+ + (Time.monotonicNow() - startTime + " milliseconds"));
+ } catch (Exception e) {
+ String errMsg = "Fsck " + e.getMessage();
+ LOG.warn(errMsg, e);
+ out.println("Federated FSCK ended at " + new Date() + " in "
+ + (Time.monotonicNow() - startTime + " milliseconds"));
+ out.println(e.getMessage());
+ out.print("\n\n" + errMsg);
+ } finally {
+ out.close();
+ }
+ }
+
+ /**
+ * Perform FSCK in a remote Namenode.
+ *
+ * @param nn The state of the remote NameNode
+ * @throws IOException Failed to fsck in a remote NameNode
+ */
+ private void remoteFsck(MembershipState nn) throws IOException {
+ final String scheme = nn.getWebScheme();
+ final String webAddress = nn.getWebAddress();
+ final String args = getURLArguments(pmap);
+ final URL url = new URL(scheme + "://" + webAddress + "/fsck?" + args);
+
+ // Connect to the Namenode and output
+ final URLConnection conn = url.openConnection();
+ try (InputStream is = conn.getInputStream();
+ InputStreamReader isr = new InputStreamReader(is, StandardCharsets.UTF_8);
Review comment:
Too long.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a change in pull request #1832:
HDFS-13989. RBF: Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
goiri commented on a change in pull request #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#discussion_r376562727
##########
File path: hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterFsckServlet.java
##########
@@ -0,0 +1,80 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import java.io.IOException;
+import java.io.PrintWriter;
+import java.net.InetAddress;
+import java.security.PrivilegedExceptionAction;
+import java.util.Map;
+
+import javax.servlet.ServletContext;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hdfs.server.common.JspHelper;
+import org.apache.hadoop.security.UserGroupInformation;
+
+/**
+ * This class is used in Namesystem's web server to do fsck on namenode.
+ */
+@InterfaceAudience.Private
+public class RouterFsckServlet extends HttpServlet {
+ /** for java.io.Serializable */
+ private static final long serialVersionUID = 1L;
+
+ public static final String SERVLET_NAME = "fsck";
+ public static final String PATH_SPEC = "/fsck";
+
+ /** Handle fsck request */
+ @Override
+ public void doGet(HttpServletRequest request, HttpServletResponse response
+ ) throws IOException {
Review comment:
Bad indent.
Check other checkstyles:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/2/artifact/out/diff-checkstyle-hadoop-hdfs-project.txt
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on issue #1832: HDFS-13989. RBF:
Add FSCK to the Router
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on issue #1832: HDFS-13989. RBF: Add FSCK to the Router
URL: https://github.com/apache/hadoop/pull/1832#issuecomment-585029529
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 10s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 55s | trunk passed |
| +1 :green_heart: | compile | 3m 37s | trunk passed |
| +1 :green_heart: | checkstyle | 0m 56s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 50s | trunk passed |
| +1 :green_heart: | shadedclient | 18m 20s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 15s | trunk passed |
| +0 :ok: | spotbugs | 1m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 4m 8s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 42s | the patch passed |
| +1 :green_heart: | compile | 3m 33s | the patch passed |
| +1 :green_heart: | javac | 3m 33s | the patch passed |
| +1 :green_heart: | checkstyle | 0m 52s | the patch passed |
| +1 :green_heart: | mvnsite | 1m 45s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | shadedclient | 16m 30s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 12s | the patch passed |
| +1 :green_heart: | findbugs | 4m 24s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 112m 42s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | unit | 10m 25s | hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 37s | The patch does not generate ASF License warnings. |
| | | 209m 18s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDelegationTokensWithHA |
| | hadoop.hdfs.TestFileCreation |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=19.03.5 Server=19.03.5 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/1832 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5f5a9163a6a8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 9b8a78d |
| Default Java | 1.8.0_242 |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/4/testReport/ |
| Max. process+thread count | 2730 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-rbf U: hadoop-hdfs-project |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1832/4/console |
| versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 |
| Powered by | Apache Yetus 0.11.1 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org