You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/07/29 17:41:09 UTC
[GitHub] [hadoop] bshashikant opened a new pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
bshashikant opened a new pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175
please see https://issues.apache.org/jira/browse/HDFS-15497.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on a change in pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#discussion_r463820223
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -498,9 +498,14 @@
public static final int
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT = 1000;
- public static final String DFS_NAMENODE_SNAPSHOT_MAX_LIMIT =
- "dfs.namenode.snapshot.max.limit";
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT = "dfs.namenode.snapshot.max.limit";
Review comment:
It was changed back to the original name as per Nicholas suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on a change in pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#discussion_r464587720
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##########
@@ -99,13 +99,15 @@
private boolean allowNestedSnapshots = false;
private int snapshotCounter = 0;
+ private final int maxSnapshotLimitPerDirectory;
Review comment:
sure
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -498,9 +498,14 @@
public static final int
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT = 1000;
- public static final String DFS_NAMENODE_SNAPSHOT_MAX_LIMIT =
- "dfs.namenode.snapshot.max.limit";
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT = "dfs.namenode.snapshot.max.limit";
public static final int DFS_NAMENODE_SNAPSHOT_MAX_LIMIT_DEFAULT = 65536;
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_GLOBAL_LIMIT = "dfs.namenode.snapshot.global.limit";
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#issuecomment-666254235
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 27m 9s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 17s | trunk passed |
| +1 :green_heart: | compile | 1m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 1m 9s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 58s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 16s | trunk passed |
| +1 :green_heart: | shadedclient | 16m 35s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 38s | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| +1 :green_heart: | javadoc | 0m 48s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 8s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 6s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | the patch passed |
| +1 :green_heart: | compile | 1m 12s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 1m 12s | the patch passed |
| +1 :green_heart: | compile | 1m 3s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 1m 3s | the patch passed |
| -0 :warning: | checkstyle | 0m 49s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 1 fixed = 470 total (was 468) |
| +1 :green_heart: | mvnsite | 1m 9s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 52s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 31s | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 2m 55s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 98m 5s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | The patch does not generate ASF License warnings. |
| | | 196m 6s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2175 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 940f4a7ecaf5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / cf4eb756085 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/2/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/2/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/2/testReport/ |
| Max. process+thread count | 4227 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#issuecomment-668390729
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 5s | https://github.com/apache/hadoop/pull/2175 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/2175 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2175/6/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#issuecomment-665645278
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 19m 19s | trunk passed |
| +1 :green_heart: | compile | 1m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 1m 12s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 0m 58s | trunk passed |
| +1 :green_heart: | mvnsite | 1m 15s | trunk passed |
| +1 :green_heart: | shadedclient | 17m 21s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 36s | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| +1 :green_heart: | javadoc | 0m 45s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 29s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 3m 28s | trunk passed |
||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 17s | the patch passed |
| +1 :green_heart: | compile | 1m 7s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javac | 1m 7s | the patch passed |
| +1 :green_heart: | compile | 1m 2s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | javac | 1m 2s | the patch passed |
| -0 :warning: | checkstyle | 0m 49s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 467 unchanged - 1 fixed = 470 total (was 468) |
| +1 :green_heart: | mvnsite | 1m 8s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 13m 38s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 32s | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| +1 :green_heart: | javadoc | 0m 40s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 2m 55s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 100m 43s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | The patch does not generate ASF License warnings. |
| | | 173m 9s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2175 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 25cdfca6e6b6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 3eaf62726ff |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/1/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/1/testReport/ |
| Max. process+thread count | 4003 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-2175/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on a change in pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#discussion_r463819929
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -498,9 +498,14 @@
public static final int
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT = 1000;
- public static final String DFS_NAMENODE_SNAPSHOT_MAX_LIMIT =
- "dfs.namenode.snapshot.max.limit";
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT = "dfs.namenode.snapshot.max.limit";
Review comment:
It was suggested that the config name should not be changed as it will be an incomatible change. The constant name followes default convention of defining the configs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on a change in pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#discussion_r464420469
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##########
@@ -117,23 +119,37 @@ public SnapshotManager(final Configuration conf, final FSDirectory fsdir) {
DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_DIFF_ALLOW_SNAP_ROOT_DESCENDANT,
DFSConfigKeys.
DFS_NAMENODE_SNAPSHOT_DIFF_ALLOW_SNAP_ROOT_DESCENDANT_DEFAULT);
+ this.maxSnapshotLimitPerDirectory = conf.getInt(
+ DFSConfigKeys.
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT,
+ DFSConfigKeys.
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT_DEFAULT);
this.maxSnapshotLimit = conf.getInt(
- DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_MAX_LIMIT,
- DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_MAX_LIMIT_DEFAULT);
+ DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_GLOBAL_LIMIT,
+ DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_GLOBAL_LIMIT_DEFAULT);
LOG.info("Loaded config captureOpenFiles: " + captureOpenFiles
+ ", skipCaptureAccessTimeOnlyChange: "
+ skipCaptureAccessTimeOnlyChange
+ ", snapshotDiffAllowSnapRootDescendant: "
+ snapshotDiffAllowSnapRootDescendant
+ ", maxSnapshotLimit: "
- + maxSnapshotLimit);
+ + maxSnapshotLimit
Review comment:
maxSnapshotLimitPerDirectory is already printed in the line below. I hope this works
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] szetszwo commented on a change in pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
szetszwo commented on a change in pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#discussion_r464571396
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##########
@@ -99,13 +99,15 @@
private boolean allowNestedSnapshots = false;
private int snapshotCounter = 0;
+ private final int maxSnapshotLimitPerDirectory;
Review comment:
Let's keep using maxSnapshotLimit for per directory limit and add a new variable for the new filesystem limit so that it is consistent with the conf.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -498,9 +498,14 @@
public static final int
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT = 1000;
- public static final String DFS_NAMENODE_SNAPSHOT_MAX_LIMIT =
- "dfs.namenode.snapshot.max.limit";
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT = "dfs.namenode.snapshot.max.limit";
Review comment:
Yes, we cannot change config names.
BTW, please revert the white space change so that it will be easier to back port.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -498,9 +498,14 @@
public static final int
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT = 1000;
- public static final String DFS_NAMENODE_SNAPSHOT_MAX_LIMIT =
- "dfs.namenode.snapshot.max.limit";
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT = "dfs.namenode.snapshot.max.limit";
public static final int DFS_NAMENODE_SNAPSHOT_MAX_LIMIT_DEFAULT = 65536;
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_GLOBAL_LIMIT = "dfs.namenode.snapshot.global.limit";
Review comment:
Let's call this filesystem.limit? Since there are multiple datacenter settings, "global" may be confusing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] mukul1987 commented on a change in pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#discussion_r463786951
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -498,9 +498,14 @@
public static final int
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT = 1000;
- public static final String DFS_NAMENODE_SNAPSHOT_MAX_LIMIT =
- "dfs.namenode.snapshot.max.limit";
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT = "dfs.namenode.snapshot.max.limit";
Review comment:
Lets change DFS_NAMENODE_SNAPSHOT_MAX_LIMIT to DFS_NAMENODE_SNAPSHOT_MAX_DIRECTORY_LIMIT. and lets use the same reference of the variable internally;
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##########
@@ -117,23 +119,37 @@ public SnapshotManager(final Configuration conf, final FSDirectory fsdir) {
DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_DIFF_ALLOW_SNAP_ROOT_DESCENDANT,
DFSConfigKeys.
DFS_NAMENODE_SNAPSHOT_DIFF_ALLOW_SNAP_ROOT_DESCENDANT_DEFAULT);
+ this.maxSnapshotLimitPerDirectory = conf.getInt(
+ DFSConfigKeys.
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT,
+ DFSConfigKeys.
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT_DEFAULT);
this.maxSnapshotLimit = conf.getInt(
- DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_MAX_LIMIT,
- DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_MAX_LIMIT_DEFAULT);
+ DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_GLOBAL_LIMIT,
+ DFSConfigKeys.DFS_NAMENODE_SNAPSHOT_GLOBAL_LIMIT_DEFAULT);
LOG.info("Loaded config captureOpenFiles: " + captureOpenFiles
+ ", skipCaptureAccessTimeOnlyChange: "
+ skipCaptureAccessTimeOnlyChange
+ ", snapshotDiffAllowSnapRootDescendant: "
+ snapshotDiffAllowSnapRootDescendant
+ ", maxSnapshotLimit: "
- + maxSnapshotLimit);
+ + maxSnapshotLimit
Review comment:
maxSnapshotLimit -> maxSnapshotDirectroyLimit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#issuecomment-666907287
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 0s | Docker mode activated. |
| -1 :x: | patch | 0m 4s | https://github.com/apache/hadoop/pull/2175 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|----------:|:-------------|
| GITHUB PR | https://github.com/apache/hadoop/pull/2175 |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2175/3/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#issuecomment-667246271
/retest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant commented on pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#issuecomment-668465949
Thanks @szetszwo and @mukul1987 for the review. I have committed this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant merged pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant merged pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] bshashikant commented on a change in pull request #2175: HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable
Posted by GitBox <gi...@apache.org>.
bshashikant commented on a change in pull request #2175:
URL: https://github.com/apache/hadoop/pull/2175#discussion_r464581693
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
##########
@@ -498,9 +498,14 @@
public static final int
DFS_NAMENODE_SNAPSHOT_DIFF_LISTING_LIMIT_DEFAULT = 1000;
- public static final String DFS_NAMENODE_SNAPSHOT_MAX_LIMIT =
- "dfs.namenode.snapshot.max.limit";
+ public static final String
+ DFS_NAMENODE_SNAPSHOT_MAX_LIMIT = "dfs.namenode.snapshot.max.limit";
Review comment:
sure
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
##########
@@ -99,13 +99,15 @@
private boolean allowNestedSnapshots = false;
private int snapshotCounter = 0;
+ private final int maxSnapshotLimitPerDirectory;
Review comment:
sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org