You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2022/10/23 05:29:33 UTC
[GitHub] [hadoop] haiyang1987 opened a new pull request, #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
haiyang1987 opened a new pull request, #5068:
URL: https://github.com/apache/hadoop/pull/5068
### Description of PR
HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
When the Backoff monitor is enabled, the parameter dfs.namenode.decommission.backoff.monitor.pending.limit can be dynamically adjusted to determines the maximum number of blocks related to decommission and maintenance operations that can be loaded into the replication queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008773687
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##########
@@ -2601,6 +2611,36 @@ private String reconfigureBlockInvalidateLimit(final DatanodeManager datanodeMan
}
}
+ private String reconfigureDecommissionBackoffMonitorParameters(
+ final DatanodeManager datanodeManager, final String property, final String newVal)
+ throws ReconfigurationException {
+ String newSetting;
+ try {
+ if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT)) {
+ int pendingRepLimit = (newVal == null ?
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT_DEFAULT :
+ Integer.parseInt(newVal));
+ datanodeManager.getDatanodeAdminManager().refreshPendingRepLimit(pendingRepLimit,
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT);
+ newSetting = String.valueOf(datanodeManager.getDatanodeAdminManager().getPendingRepLimit());
+ } else if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK)) {
+ int blocksPerLock = (newVal == null ?
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK_DEFAULT :
+ Integer.parseInt(newVal));
+ datanodeManager.getDatanodeAdminManager().refreshBlocksPerLock(blocksPerLock,
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK);
+ newSetting = String.valueOf(datanodeManager.getDatanodeAdminManager().getBlocksPerLock());
+ } else {
+ throw new IllegalArgumentException("Unexpected property " +
Review Comment:
Thanks @haiyang1987 for updating. This can be removed because the value of the key is already judged in the outer layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #5068: HDFS-16811. Support DecommissionBackoffMonitor parameters reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1303028191
Thanks @haiyang1987 for your contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
haiyang1987 commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1299492638
@tomscut thanks help to review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1296347385
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 13s | | trunk passed |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 5s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 0m 57s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/3/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 75 unchanged - 0 fixed = 76 total (was 75) |
| +1 :green_heart: | mvnsite | 1m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 49s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 240m 57s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not generate ASF License warnings. |
| | | 350m 20s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5068 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e0988b2a3c20 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9219530a094dcae9e01ddfbee46a559542c468e3 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/3/testReport/ |
| Max. process+thread count | 2995 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #5068: HDFS-16811. Support DecommissionBackoffMonitor parameters reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1303032871
Hi @haiyang1987 please cherry-pick this to branch-3.3. There are some conflicts. And please update the typo `reconfigurablereconfigurable` in commit info. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1288090517
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 1s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 28s | | trunk passed |
| +1 :green_heart: | compile | 1m 36s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 15s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 55s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 244m 37s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not generate ASF License warnings. |
| | | 354m 56s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5068 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a43f7ada2d75 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fab3cd36bf2f947f6bfc4c23ac46ff7400063552 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/1/testReport/ |
| Max. process+thread count | 3192 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008775968
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminBackoffMonitor.java:
##########
@@ -801,6 +801,23 @@ private boolean isBlockReplicatedOk(DatanodeDescriptor datanode,
return false;
}
+
Review Comment:
Add `@VisibleForTesting`.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminBackoffMonitor.java:
##########
@@ -801,6 +801,23 @@ private boolean isBlockReplicatedOk(DatanodeDescriptor datanode,
return false;
}
+
+ public int getPendingRepLimit() {
+ return pendingRepLimit;
+ }
+
+ public void setPendingRepLimit(int pendingRepLimit) {
+ this.pendingRepLimit = pendingRepLimit;
+ }
+
+ public int getBlocksPerLock() {
Review Comment:
Add `@VisibleForTesting`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1003894972
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##########
@@ -2321,6 +2324,9 @@ protected String reconfigurePropertyImpl(String property, String newVal)
return reconfigureSlowNodesParameters(datanodeManager, property, newVal);
} else if (property.equals(DFS_BLOCK_INVALIDATE_LIMIT_KEY)) {
return reconfigureBlockInvalidateLimit(datanodeManager, property, newVal);
+ } else if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT)) {
Review Comment:
Hi @haiyang1987 , can we also make `DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK_DEFAULT` reconfigurable? And create a method for `DecommissionBackoffMonitor`. Just like `reconfigureSlowNodesParameters()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1296410834
> Updata PR. @tomscut please help review again , Thanks
Please fix the checkstyles warning. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008775990
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##########
@@ -137,6 +137,26 @@ public int getNumNodesChecked() {
return numNodesChecked;
}
+ @Override
Review Comment:
Add `@VisibleForTesting`.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java:
##########
@@ -137,6 +137,26 @@ public int getNumNodesChecked() {
return numNodesChecked;
}
+ @Override
+ public int getPendingRepLimit() {
+ return 0;
+ }
+
+ @Override
+ public void setPendingRepLimit(int pendingRepLimit) {
+ // nothing.
+ }
+
+ @Override
Review Comment:
Add `@VisibleForTesting`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
haiyang1987 commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1003973853
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##########
@@ -2321,6 +2324,9 @@ protected String reconfigurePropertyImpl(String property, String newVal)
return reconfigureSlowNodesParameters(datanodeManager, property, newVal);
} else if (property.equals(DFS_BLOCK_INVALIDATE_LIMIT_KEY)) {
return reconfigureBlockInvalidateLimit(datanodeManager, property, newVal);
+ } else if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT)) {
Review Comment:
Hi @tomscut, thanks for your comment.
yeah, can create new method reconfigureDecommissionBackoffMonitorParameters() to update DecommissionBackoffMonitor related parameters.
HDFS-16810 to support ‘DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK_DEFAULT’ reconfigurable
Do we need to merge 2 issues into one to support?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008775655
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java:
##########
@@ -419,4 +419,28 @@ void runMonitorForTest() throws ExecutionException, InterruptedException {
executor.submit(monitor).get();
}
+ public void refreshPendingRepLimit(int pendingRepLimit, String key) {
+ ensurePositiveInt(pendingRepLimit, key);
+ this.monitor.setPendingRepLimit(pendingRepLimit);
+ }
+
+ public int getPendingRepLimit() {
Review Comment:
Please add `@VisibleForTesting`.
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminManager.java:
##########
@@ -419,4 +419,28 @@ void runMonitorForTest() throws ExecutionException, InterruptedException {
executor.submit(monitor).get();
}
+ public void refreshPendingRepLimit(int pendingRepLimit, String key) {
+ ensurePositiveInt(pendingRepLimit, key);
+ this.monitor.setPendingRepLimit(pendingRepLimit);
+ }
+
+ public int getPendingRepLimit() {
+ return this.monitor.getPendingRepLimit();
+ }
+
+ public void refreshBlocksPerLock(int blocksPerLock, String key) {
+ ensurePositiveInt(blocksPerLock, key);
+ this.monitor.setBlocksPerLock(blocksPerLock);
+ }
+
+ public int getBlocksPerLock() {
Review Comment:
Add `@VisibleForTesting`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008775125
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminBackoffMonitor.java:
##########
@@ -73,7 +73,7 @@ public class DatanodeAdminBackoffMonitor extends DatanodeAdminMonitorBase
* The numbe of blocks to process when moving blocks to pendingReplication
Review Comment:
Please fix this typo `numbe` by the way. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5068: HDFS-16811. Support DecommissionBackoffMonitor parameters reconfigurable
Posted by GitBox <gi...@apache.org>.
haiyang1987 commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1305140936
Hi @tomscut sorry for the late reply, I will do it later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
haiyang1987 commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1006849654
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##########
@@ -2321,6 +2324,9 @@ protected String reconfigurePropertyImpl(String property, String newVal)
return reconfigureSlowNodesParameters(datanodeManager, property, newVal);
} else if (property.equals(DFS_BLOCK_INVALIDATE_LIMIT_KEY)) {
return reconfigureBlockInvalidateLimit(datanodeManager, property, newVal);
+ } else if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT)) {
Review Comment:
@tomscut Thanks for comment,
I will merge this two issues support into here, update PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008774407
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java:
##########
@@ -567,6 +571,92 @@ private List<Boolean> validatePeerReport(String jsonReport) {
return containReport;
}
+ @Test
+ public void testReconfigureDecommissionBackoffMonitorParameters()
+ throws ReconfigurationException, IOException {
+ Configuration conf = new HdfsConfiguration();
+ conf.setClass(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MONITOR_CLASS,
+ DatanodeAdminBackoffMonitor.class, DatanodeAdminMonitorInterface.class);
+ int defaultPendingRepLimit = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT, defaultPendingRepLimit);
+ int defaultBlocksPerLock = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ defaultBlocksPerLock);
+ MiniDFSCluster newCluster = new MiniDFSCluster.Builder(conf).build();
+ newCluster.waitActive();
+
+ try {
+ final NameNode nameNode = newCluster.getNameNode();
+ final DatanodeManager datanodeManager = nameNode.namesystem
+ .getBlockManager().getDatanodeManager();
+
+ // verify defaultPendingRepLimit.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(),
+ defaultPendingRepLimit);
+
+ // try invalid pendingRepLimit.
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to '-1'", e.getMessage());
+ }
+
+ // try correct pendingRepLimit.
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "20000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(), 20000);
+
+ // verify defaultBlocksPerLock.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(),
+ defaultBlocksPerLock);
+
+ // try invalid blocksPerLock.
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to '-1'", e.getMessage());
+ }
+
+ // try correct blocksPerLock.
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "10000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(), 10000);
+ } finally {
Review Comment:
```suggestion
```
Please also update the finally block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1297454194
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 27s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 38s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 40s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 0s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 24s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 56s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 243m 50s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not generate ASF License warnings. |
| | | 353m 42s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5068 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d7d67e1785b0 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6feb398a6aef36b4ecfade645d877a3070835b6f |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/4/testReport/ |
| Max. process+thread count | 3069 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1005110587
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##########
@@ -2321,6 +2324,9 @@ protected String reconfigurePropertyImpl(String property, String newVal)
return reconfigureSlowNodesParameters(datanodeManager, property, newVal);
} else if (property.equals(DFS_BLOCK_INVALIDATE_LIMIT_KEY)) {
return reconfigureBlockInvalidateLimit(datanodeManager, property, newVal);
+ } else if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT)) {
Review Comment:
> Hi @tomscut, thanks for your comment. yeah, can create new method reconfigureDecommissionBackoffMonitorParameters() to update DecommissionBackoffMonitor related parameters.
>
> [HDFS-16810](https://issues.apache.org/jira/browse/HDFS-16810) to support ‘DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK_DEFAULT’ reconfigurable
>
> Do we need to merge 2 issues into one to support?
I think it could be merged together to make it clearer. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008774411
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java:
##########
@@ -567,6 +571,92 @@ private List<Boolean> validatePeerReport(String jsonReport) {
return containReport;
}
+ @Test
+ public void testReconfigureDecommissionBackoffMonitorParameters()
+ throws ReconfigurationException, IOException {
+ Configuration conf = new HdfsConfiguration();
+ conf.setClass(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MONITOR_CLASS,
+ DatanodeAdminBackoffMonitor.class, DatanodeAdminMonitorInterface.class);
+ int defaultPendingRepLimit = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT, defaultPendingRepLimit);
+ int defaultBlocksPerLock = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ defaultBlocksPerLock);
+ MiniDFSCluster newCluster = new MiniDFSCluster.Builder(conf).build();
+ newCluster.waitActive();
+
+ try {
+ final NameNode nameNode = newCluster.getNameNode();
+ final DatanodeManager datanodeManager = nameNode.namesystem
+ .getBlockManager().getDatanodeManager();
+
+ // verify defaultPendingRepLimit.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(),
+ defaultPendingRepLimit);
+
+ // try invalid pendingRepLimit.
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to '-1'", e.getMessage());
+ }
+
+ // try correct pendingRepLimit.
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "20000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(), 20000);
+
+ // verify defaultBlocksPerLock.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(),
+ defaultBlocksPerLock);
+
+ // try invalid blocksPerLock.
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to '-1'", e.getMessage());
+ }
+
+ // try correct blocksPerLock.
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "10000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(), 10000);
+ } finally {
+ if (newCluster != null) {
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008774315
##########
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##########
@@ -2601,6 +2611,36 @@ private String reconfigureBlockInvalidateLimit(final DatanodeManager datanodeMan
}
}
+ private String reconfigureDecommissionBackoffMonitorParameters(
+ final DatanodeManager datanodeManager, final String property, final String newVal)
+ throws ReconfigurationException {
+ String newSetting;
+ try {
+ if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT)) {
+ int pendingRepLimit = (newVal == null ?
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT_DEFAULT :
+ Integer.parseInt(newVal));
+ datanodeManager.getDatanodeAdminManager().refreshPendingRepLimit(pendingRepLimit,
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT);
+ newSetting = String.valueOf(datanodeManager.getDatanodeAdminManager().getPendingRepLimit());
+ } else if (property.equals(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK)) {
Review Comment:
Please fix the checkstyles warn here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1295952262
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 2 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 40s | | trunk passed |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 11s | | branch has no errors when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 19s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | checkstyle | 1m 0s | [/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 75 unchanged - 0 fixed = 76 total (was 75) |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 44s | | patch has no errors when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 237m 52s | | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not generate ASF License warnings. |
| | | 348m 9s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/5068 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux af62785a4c81 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5eb3fcea335142d2bf0e4c892d78c2c22f2c7126 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/2/testReport/ |
| Max. process+thread count | 3168 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5068/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1298265787
The failed unit test seems unrelated to the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut merged pull request #5068: HDFS-16811. Support DecommissionBackoffMonitor parameters reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut merged PR #5068:
URL: https://github.com/apache/hadoop/pull/5068
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
haiyang1987 commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1298205771
Trigger notification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008774407
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java:
##########
@@ -567,6 +571,92 @@ private List<Boolean> validatePeerReport(String jsonReport) {
return containReport;
}
+ @Test
+ public void testReconfigureDecommissionBackoffMonitorParameters()
+ throws ReconfigurationException, IOException {
+ Configuration conf = new HdfsConfiguration();
+ conf.setClass(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MONITOR_CLASS,
+ DatanodeAdminBackoffMonitor.class, DatanodeAdminMonitorInterface.class);
+ int defaultPendingRepLimit = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT, defaultPendingRepLimit);
+ int defaultBlocksPerLock = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ defaultBlocksPerLock);
+ MiniDFSCluster newCluster = new MiniDFSCluster.Builder(conf).build();
+ newCluster.waitActive();
+
+ try {
+ final NameNode nameNode = newCluster.getNameNode();
+ final DatanodeManager datanodeManager = nameNode.namesystem
+ .getBlockManager().getDatanodeManager();
+
+ // verify defaultPendingRepLimit.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(),
+ defaultPendingRepLimit);
+
+ // try invalid pendingRepLimit.
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to '-1'", e.getMessage());
+ }
+
+ // try correct pendingRepLimit.
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "20000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(), 20000);
+
+ // verify defaultBlocksPerLock.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(),
+ defaultBlocksPerLock);
+
+ // try invalid blocksPerLock.
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to '-1'", e.getMessage());
+ }
+
+ // try correct blocksPerLock.
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "10000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(), 10000);
+ } finally {
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008774407
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java:
##########
@@ -567,6 +571,92 @@ private List<Boolean> validatePeerReport(String jsonReport) {
return containReport;
}
+ @Test
+ public void testReconfigureDecommissionBackoffMonitorParameters()
+ throws ReconfigurationException, IOException {
+ Configuration conf = new HdfsConfiguration();
+ conf.setClass(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MONITOR_CLASS,
+ DatanodeAdminBackoffMonitor.class, DatanodeAdminMonitorInterface.class);
+ int defaultPendingRepLimit = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT, defaultPendingRepLimit);
+ int defaultBlocksPerLock = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ defaultBlocksPerLock);
+ MiniDFSCluster newCluster = new MiniDFSCluster.Builder(conf).build();
+ newCluster.waitActive();
+
+ try {
+ final NameNode nameNode = newCluster.getNameNode();
+ final DatanodeManager datanodeManager = nameNode.namesystem
+ .getBlockManager().getDatanodeManager();
+
+ // verify defaultPendingRepLimit.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(),
+ defaultPendingRepLimit);
+
+ // try invalid pendingRepLimit.
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.limit from '" +
+ defaultPendingRepLimit + "' to '-1'", e.getMessage());
+ }
+
+ // try correct pendingRepLimit.
+ nameNode.reconfigureProperty(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT,
+ "20000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getPendingRepLimit(), 20000);
+
+ // verify defaultBlocksPerLock.
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(),
+ defaultBlocksPerLock);
+
+ // try invalid blocksPerLock.
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ "non-numeric");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to 'non-numeric'", e.getMessage());
+ }
+
+ try {
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "-1");
+ fail("Should not reach here");
+ } catch (ReconfigurationException e) {
+ assertEquals("Could not change property " +
+ "dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock from '" +
+ defaultBlocksPerLock + "' to '-1'", e.getMessage());
+ }
+
+ // try correct blocksPerLock.
+ nameNode.reconfigureProperty(
+ DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK, "10000");
+ assertEquals(datanodeManager.getDatanodeAdminManager().getBlocksPerLock(), 10000);
+ } finally {
Review Comment:
Please also update the finally block.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] tomscut commented on a diff in pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
tomscut commented on code in PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#discussion_r1008773817
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeReconfigure.java:
##########
@@ -567,6 +571,92 @@ private List<Boolean> validatePeerReport(String jsonReport) {
return containReport;
}
+ @Test
+ public void testReconfigureDecommissionBackoffMonitorParameters()
+ throws ReconfigurationException, IOException {
+ Configuration conf = new HdfsConfiguration();
+ conf.setClass(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_MONITOR_CLASS,
+ DatanodeAdminBackoffMonitor.class, DatanodeAdminMonitorInterface.class);
+ int defaultPendingRepLimit = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_LIMIT, defaultPendingRepLimit);
+ int defaultBlocksPerLock = 1000;
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_BACKOFF_MONITOR_PENDING_BLOCKS_PER_LOCK,
+ defaultBlocksPerLock);
+ MiniDFSCluster newCluster = new MiniDFSCluster.Builder(conf).build();
+ newCluster.waitActive();
+
+ try {
Review Comment:
```suggestion
try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).build()) {
newCluster.waitActive();
```
Please also update this by the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] haiyang1987 commented on pull request #5068: HDFS-16811. Support to make dfs.namenode.decommission.backoff.monitor.pending.limit reconfigurable
Posted by GitBox <gi...@apache.org>.
haiyang1987 commented on PR #5068:
URL: https://github.com/apache/hadoop/pull/5068#issuecomment-1296278479
Updata PR.
@tomscut please help review again , Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org