You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by GitBox <gi...@apache.org> on 2020/08/04 03:23:40 UTC
[GitHub] [hadoop] huangtianhua opened a new pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
huangtianhua opened a new pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189
The non-volatile memory NVDIMM is faster than SSD,
it can be used simultaneously with RAM, DISK and SSD.
Storing the data of HDFS on NVDIMM directly will get
better response rate and reliability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333
@huangtianhua and @YaYun-Wang thanks for review. @liuml07 thanks for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483194125
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
Oh, I was thinking that allowing Balancer to move the NVDIMM data is by design since they are not volatile. But if that is *not* case, then we can update Balancer code by replacing `isTransient()` call with `isRAM()` call. Not sure if this makes more sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-674689358
@liuml07 Hi, would you please help to review it then we will modify this ASAP, thanks very much:)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-685405577
@liuml07 and @brahmareddybattula please help to review the latest code again? thanks very much.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r477166546
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
Review comment:
Sorry, I don't understand.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-691761278
@brahmareddybattula Hi brahma, would you review it again, thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483515521
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
Sorry, I made a big mistake in the above reply. Our design idea is: NVDIMM supports mover and balancer, and `isTransient() ` applied in the case. isRAM() is only used to “FsDatasetCache” judgment. So the current code is reasonable, i think it‘s not necessary to modify the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r477150690
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
##########
@@ -88,13 +92,14 @@ The effective storage policy can be retrieved by the "[`storagepolicies -getStor
### Configuration
* **dfs.storage.policy.enabled** - for enabling/disabling the storage policy feature. The default value is `true`.
-* **dfs.storage.default.policy** - Set the default storage policy with the policy name. The default value is `HOT`. All possible policies are defined in enum StoragePolicy, including `LAZY_PERSIST` `ALL_SSD` `ONE_SSD` `HOT` `WARM` `COLD` and `PROVIDED`.
+* **dfs.storage.default.policy** - Set the default storage policy with the policy name. The default value is `HOT`. All possible policies are defined in enum StoragePolicy, including `LAZY_PERSIST` `ALL_SSD` `ONE_SSD` `HOT` `WARM` `COLD` `ALL_NVDIMM` and `PROVIDED`.
Review comment:
sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-681369503
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 47s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 8s | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 0m 22s | root in trunk failed. |
| -1 :x: | compile | 0m 22s | root in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 0m 22s | root in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 1m 48s | hadoop-common in trunk failed. |
| -1 :x: | mvnsite | 0m 22s | hadoop-hdfs in trunk failed. |
| -1 :x: | mvnsite | 0m 22s | hadoop-hdfs-client in trunk failed. |
| +1 :green_heart: | shadedclient | 3m 18s | branch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 22s | hadoop-common in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 23s | hadoop-hdfs in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 22s | hadoop-hdfs-client in trunk failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 23s | hadoop-common in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javadoc | 0m 21s | hadoop-hdfs in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javadoc | 0m 21s | hadoop-hdfs-client in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| +0 :ok: | spotbugs | 6m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| -1 :x: | findbugs | 0m 21s | hadoop-common in trunk failed. |
| -1 :x: | findbugs | 0m 23s | hadoop-hdfs in trunk failed. |
| -1 :x: | findbugs | 0m 22s | hadoop-hdfs-client in trunk failed. |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 22s | hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 22s | hadoop-hdfs in the patch failed. |
| -1 :x: | mvninstall | 0m 22s | hadoop-hdfs-client in the patch failed. |
| -1 :x: | compile | 0m 21s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | cc | 0m 21s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javac | 0m 21s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 0m 22s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | cc | 0m 22s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javac | 0m 22s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 0m 20s | The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 22s | hadoop-common in the patch failed. |
| -1 :x: | mvnsite | 0m 22s | hadoop-hdfs in the patch failed. |
| -1 :x: | mvnsite | 0m 22s | hadoop-hdfs-client in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 0m 22s | patch has no errors when building and testing our client artifacts. |
| -1 :x: | javadoc | 0m 33s | hadoop-common in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 38s | hadoop-hdfs in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 28s | hadoop-hdfs-client in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javadoc | 0m 16s | hadoop-common in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javadoc | 0m 12s | hadoop-hdfs in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javadoc | 0m 12s | hadoop-hdfs-client in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | findbugs | 0m 13s | hadoop-common in the patch failed. |
| -1 :x: | findbugs | 0m 12s | hadoop-hdfs in the patch failed. |
| -1 :x: | findbugs | 0m 15s | hadoop-hdfs-client in the patch failed. |
||| _ Other Tests _ |
| -1 :x: | unit | 0m 16s | hadoop-common in the patch failed. |
| -1 :x: | unit | 0m 14s | hadoop-hdfs in the patch failed. |
| -1 :x: | unit | 0m 14s | hadoop-hdfs-client in the patch failed. |
| +1 :green_heart: | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| | | 21m 22s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux fb204f10a504 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ca8e7a77256 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-mvninstall-root.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/buildtool-branch-checkstyle-root.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-findbugs-hadoop-common-project_hadoop-common.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/buildtool-patch-checkstyle-root.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-mvnsite-hadoop-common-project_hadoop-common.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javadoc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-findbugs-hadoop-common-project_hadoop-common.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/testReport/ |
| Max. process+thread count | 51 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/9/console |
| versions | git=2.17.1 maven=3.6.0 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483515521
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
Sorry, I made a big mistake in the above reply. Our design idea is: NVDIMM supports mover and balancer, and `isTransient() ` applied in the case.` isRAM()` is only used to “FsDatasetCache” judgment. So the current code is reasonable, i think it‘s not necessary to modify the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-691760924
@liuml07 That sounds good, thank you very much:)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486545111
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##########
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
Ok. Got it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483125623
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
Balancer and mover will not move the blocks based on the `isTransient` ( they call getMovableTypes(..))..The blocks which are in NVDIMM shouldn't moved I feel(as this also exists in RAM and no need to move),but as per this change it will move.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697162885
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
| +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 16 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 19s | | trunk passed |
| +1 :green_heart: | compile | 26m 40s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 22m 27s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 41s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 32s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 48s | | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 29s | | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 13s | | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 16s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 10m 3s | | trunk passed |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 37s | | the patch passed |
| +1 :green_heart: | compile | 20m 54s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 20m 54s | [/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt) | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 32 new + 131 unchanged - 32 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 20m 54s | | the patch passed |
| +1 :green_heart: | compile | 16m 48s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 16m 48s | [/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt) | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 37 new + 126 unchanged - 37 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 16m 48s | | the patch passed |
| -0 :warning: | checkstyle | 2m 57s | [/diff-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/diff-checkstyle-root.txt) | root: The patch generated 16 new + 723 unchanged - 6 fixed = 739 total (was 729) |
| +1 :green_heart: | mvnsite | 4m 3s | | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 15s | | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 34s | | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 6s | | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 21s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 22s | | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 21s | | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 96m 8s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not generate ASF License warnings. |
| | | 322m 9s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux b39fec604d95 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ae089f2db72 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/testReport/ |
| Max. process+thread count | 4531 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/17/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula edited a comment on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula edited a comment on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333
@huangtianhua and @YaYun-Wang thanks for contribution. @liuml07 thanks for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-668645247
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698070550
@liuml07 , thanks, and we have updated the code, seems there is a checkstyle about the storage enum name of 'nvdimm' should keep as it is to consistent with other storage types, what do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696754216
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 28s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 55s | trunk passed |
| +1 :green_heart: | compile | 21m 10s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 0s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 55s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 39s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 22s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 50s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 38s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 16s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 53s | the patch passed |
| +1 :green_heart: | compile | 20m 54s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 20m 54s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 35 new + 128 unchanged - 35 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 20m 54s | the patch passed |
| +1 :green_heart: | compile | 18m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 18m 50s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 34 new + 129 unchanged - 34 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 18m 50s | the patch passed |
| -0 :warning: | checkstyle | 2m 58s | root: The patch generated 4 new + 727 unchanged - 4 fixed = 731 total (was 731) |
| +1 :green_heart: | mvnsite | 4m 2s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 6s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 21s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 1s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 48s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 47s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 24s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 115m 39s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 5s | The patch does not generate ASF License warnings. |
| | | 321m 38s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.TestDatanodeLayoutUpgrade |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.TestMultiThreadedHflush |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.TestClientReportBadBlock |
| | hadoop.hdfs.server.namenode.TestListOpenFiles |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestFileConcurrentReader |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.TestWriteRead |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/13/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux c09dbf3d406b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b5d9e2334b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/13/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/13/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/13/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/13/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/13/testReport/ |
| Max. process+thread count | 3558 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/13/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697068485
@liuml07 Sorry, we didn't catch you, what other changes we have to do?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-677511383
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 33m 16s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 23s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 9s | trunk passed |
| +1 :green_heart: | compile | 25m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | compile | 18m 42s | root in trunk failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| +1 :green_heart: | checkstyle | 4m 13s | trunk passed |
| -1 :x: | mvnsite | 0m 48s | hadoop-common in trunk failed. |
| +1 :green_heart: | shadedclient | 24m 50s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 15s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 46s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 36s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 5s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 48s | the patch passed |
| +1 :green_heart: | compile | 20m 36s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 20m 36s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 15 new + 147 unchanged - 15 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 20m 36s | the patch passed |
| +1 :green_heart: | compile | 18m 52s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 18m 52s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 39 new + 123 unchanged - 39 fixed = 162 total (was 162) |
| -1 :x: | javac | 18m 52s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 187 new + 1766 unchanged - 0 fixed = 1953 total (was 1766) |
| -0 :warning: | checkstyle | 2m 57s | root: The patch generated 3 new + 727 unchanged - 4 fixed = 730 total (was 731) |
| +1 :green_heart: | mvnsite | 3m 59s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 4s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 32s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 10s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 9m 2s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 27s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 21s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 96m 38s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 348m 8s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 4a620c0f25c9 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9b9f7ea16a2 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-common.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/diff-compile-javac-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/testReport/ |
| Max. process+thread count | 4564 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/4/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690912424
@liuml07 We have checked for several times, the failure tests are not related with this patch, and I found almost all of the HDFS-* patches are not pass the hadoop-hdfs unit tests, like https://github.com/apache/hadoop/pull/2275 https://github.com/apache/hadoop/pull/2205
So maybe the failure tests are not blocker?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r494037751
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690912424
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716810
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##########
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
> So, NVDIMM is peristent storage and RAM.
yes, that’s right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r475381174
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
##########
@@ -27,15 +27,17 @@ The frameworks provided by Heterogeneous Storage and Archival Storage generalize
Storage Types and Storage Policies
----------------------------------
-### Storage Types: ARCHIVE, DISK, SSD and RAM\_DISK
+### Storage Types: ARCHIVE, DISK, SSD, NVDIMM and RAM\_DISK
The first phase of [Heterogeneous Storage (HDFS-2832)](https://issues.apache.org/jira/browse/HDFS-2832) changed datanode storage model from a single storage, which may correspond to multiple physical storage medias, to a collection of storages with each storage corresponding to a physical storage media. It also added the notion of storage types, DISK and SSD, where DISK is the default storage type.
A new storage type *ARCHIVE*, which has high storage density (petabyte of storage) but little compute power, is added for supporting archival storage.
Another new storage type *RAM\_DISK* is added for supporting writing single replica files in memory.
-### Storage Policies: Hot, Warm, Cold, All\_SSD, One\_SSD, Lazy\_Persist and Provided
+And a new storage type *NVDIMM*, which is a non-volatile memory that will retains the datastored on the memory when the computer is powered down or system crashes and restore the data when the machine powered on, is added for supporting archival storage.
Review comment:
1. `is added for supporting archival storage.` What is the main purpose of this use case? I don't believe this is for "archival storage".
1. Also, this sentence is a bit verbose to me. Do you think we can make it concise, something like this?
```
From Hadoop 3.4, a new storage type *NVDIMM* is added for supporting writing replica files in non-volatile
memory that has the capability to hold saved data even if the power is turned off.
```
If you instead prefer keeping the current phrase, correct the syntax error by `s/retains/retain/` and `s/datastored/data stored`.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
##########
@@ -63,6 +63,12 @@ public static BlockStoragePolicySuite createDefaultSuite(
new StorageType[]{StorageType.DISK},
new StorageType[]{StorageType.DISK},
true); // Cannot be changed on regular files, but inherited.
+ final byte allNVDIMMId = HdfsConstants.StoragePolicy.ALL_NVDIMM.value();
Review comment:
nit: I see other variables are using lower case, could we change this name to `allnvdimmId`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r477268196
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
##########
@@ -63,6 +63,12 @@ public static BlockStoragePolicySuite createDefaultSuite(
new StorageType[]{StorageType.DISK},
new StorageType[]{StorageType.DISK},
true); // Cannot be changed on regular files, but inherited.
+ final byte allNVDIMMId = HdfsConstants.StoragePolicy.ALL_NVDIMM.value();
Review comment:
> Yes I agree the checksum calculation and read is built-in hadoop. I'm thinking of more about: do we need checksum for this storage type? For example, the RAM_DISK does not need checksum as far as I remember. This is RAM, but this also survives the service restarts, so checksum makes sense for data integrity. The code so far looks good regarding this.
Yes, NVDIMM, like ordinary persistent storage type, such as DISK, SSD, etc. requires checksum too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-681615978
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 44s | trunk passed |
| +1 :green_heart: | compile | 19m 9s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 45s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 0s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 3s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 36s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 32s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 4s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 35s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 7m 58s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 44s | the patch passed |
| +1 :green_heart: | compile | 18m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 18m 38s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 12 new + 151 unchanged - 12 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 18m 38s | the patch passed |
| +1 :green_heart: | compile | 17m 50s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 17m 50s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 10 new + 153 unchanged - 10 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 17m 50s | the patch passed |
| -0 :warning: | checkstyle | 2m 57s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) |
| +1 :green_heart: | mvnsite | 4m 4s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 15s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 54s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 39s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 56s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 101m 28s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | The patch does not generate ASF License warnings. |
| | | 297m 7s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.server.balancer.TestBalancer |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/8/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 07820bd5ed78 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5e52955112a |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/8/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/8/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/8/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/8/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/8/testReport/ |
| Max. process+thread count | 4321 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/8/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696758750
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 4s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 16s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 33s | trunk passed |
| +1 :green_heart: | compile | 20m 57s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 3s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 48s | trunk passed |
| +1 :green_heart: | shadedclient | 23m 2s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 10s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 38s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 7s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 49s | the patch passed |
| +1 :green_heart: | compile | 20m 8s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 20m 8s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 30 new + 133 unchanged - 30 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 20m 8s | the patch passed |
| +1 :green_heart: | compile | 17m 39s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 17m 39s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 5 new + 158 unchanged - 5 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 17m 39s | the patch passed |
| -0 :warning: | checkstyle | 3m 2s | root: The patch generated 16 new + 723 unchanged - 6 fixed = 739 total (was 729) |
| +1 :green_heart: | mvnsite | 3m 50s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 22s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 11s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 35s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 42s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 10s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 15s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 113m 52s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | The patch does not generate ASF License warnings. |
| | | 319m 19s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/15/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 56056b587384 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b5d9e2334b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/15/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/15/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/15/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/15/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/15/testReport/ |
| Max. process+thread count | 3049 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/15/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697068485
@liuml07 Sorry, we didn't catch you, what other changes we have to do?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690932868
No this is not a blocker @huangtianhua I will review (again) and commit hopefully this week if there is no other comments. Thanks,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r477304478
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##########
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
The "isTransientStorage" method is still available.
In the original code, “isTransient“ and “isTransientStorage” methods are used to determine whether to suport FsDatasetCache, Persistent, Quota, and Movable. FsDatasetCache will be used When the storage type is persistent. NVDIMM is RAM to some extent, which is fast. However, NVDIMM is a persistent storage type. Then, "isTransient" and "isTransientStorage" used to determine whether to support FsDatasetCache can't meet the requirements. Therefore, we add "isRAM" and "isRAMStorage" methods to decide whether cache is supported or not. And the other functions, such as, Persistent, Quota, and Movable judged by " isTransient" and "isTransientStorage" methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483194125
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
Oh, I was thinking that allowing Balancer to move the NVDIMM data is by design since they are not volatile. But if that is case, then we can update Balancer code by replacing `isTransient()` call with `isRAM()` call. Not sure if this makes more sense?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491859739
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##########
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map<String,Object> storageTypeStats = (Map<String,Object>)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
- assertEquals(3l, storageTypeStats.get("nodesInService"));
+ assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
Hadoop releases before 2.10 are all end of life (EoL). Hadoop 2.10 is the only version using Java 7. We do not need any support, compile or runtime, for Java versions before Java 7.
Hadoop 3.x are all using Java 8+. We do not need any Java 7 support in Hadoop 3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492866266
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##########
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes() throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+ FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+ .setStorageDirectory(
+ new StorageDirectory(
+ StorageLocation.parse("[NVDIMM]"+volDir.getPath())))
+ .setStorageID("storage-id")
+ .setConf(conf)
+ .build();
+ assertEquals("", 3L, volume5.getReserved());
Review comment:
Usually we can, but following original code style here is bad. When it fails, the original code gives up empty string. My code shows your expected value and actual value so you can debug. Please change it.
We can also file a JIRA to update all such cases where `assertEquals` can be improved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r476125679
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
##########
@@ -27,15 +27,17 @@ The frameworks provided by Heterogeneous Storage and Archival Storage generalize
Storage Types and Storage Policies
----------------------------------
-### Storage Types: ARCHIVE, DISK, SSD and RAM\_DISK
+### Storage Types: ARCHIVE, DISK, SSD, NVDIMM and RAM\_DISK
The first phase of [Heterogeneous Storage (HDFS-2832)](https://issues.apache.org/jira/browse/HDFS-2832) changed datanode storage model from a single storage, which may correspond to multiple physical storage medias, to a collection of storages with each storage corresponding to a physical storage media. It also added the notion of storage types, DISK and SSD, where DISK is the default storage type.
A new storage type *ARCHIVE*, which has high storage density (petabyte of storage) but little compute power, is added for supporting archival storage.
Another new storage type *RAM\_DISK* is added for supporting writing single replica files in memory.
-### Storage Policies: Hot, Warm, Cold, All\_SSD, One\_SSD, Lazy\_Persist and Provided
+And a new storage type *NVDIMM*, which is a non-volatile memory that will retains the datastored on the memory when the computer is powered down or system crashes and restore the data when the machine powered on, is added for supporting archival storage.
Review comment:
The original description is indeed inaccurate , and has been modified as suggested.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491857797
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##########
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map<String,Object> storageTypeStats = (Map<String,Object>)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
- assertEquals(3l, storageTypeStats.get("nodesInService"));
+ assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
I have not used Java 7 for a while, but I remember vaguely this is actually supported?
https://docs.oracle.com/javase/specs/jls/se7/html/jls-14.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491852471
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##########
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map<String,Object> storageTypeStats = (Map<String,Object>)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
- assertEquals(3l, storageTypeStats.get("nodesInService"));
+ assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
`storageType` is a parameter of "java.lang.String" , and `switch()` does not support "java.lang.String" before java 1.7. So, will `if-else ` be more appropriate here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r477304478
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##########
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
The "isTransientStorage" method is still available.
In the original code,` isTransient()` and `isTransientStorage` methods are used to determine whether to support FsDatasetCache, Persistent, Quota, and Movable.
FsDatasetCache will be used When the storage type is persistent. NVDIMM is RAM to some extent, which is fast. However, NVDIMM is a persistent storage type. Then, `isTransient()` and `isTransientStorage() ` used to determine whether to support FsDatasetCache can't meet the requirements. Therefore, we add `isRAM()` and `isRAMStorage() ` methods to decide whether cache is supported or not. And the other functions, such as, Persistent, Quota, and Movable judged by`isTransient()` and `isTransientStorage() ` methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486548053
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##########
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
So, NVDIMM is peristent storage and RAM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r494037751
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696757028
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 44s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 32s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 58s | trunk passed |
| +1 :green_heart: | compile | 21m 42s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 16s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 2s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 5s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 51s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 22s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 48s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 41s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 14s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 6s | the patch passed |
| +1 :green_heart: | compile | 20m 55s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 20m 55s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 9 new + 154 unchanged - 9 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 20m 55s | the patch passed |
| +1 :green_heart: | compile | 18m 15s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 18m 15s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 14 new + 149 unchanged - 14 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 18m 15s | the patch passed |
| -0 :warning: | checkstyle | 2m 59s | root: The patch generated 16 new + 725 unchanged - 6 fixed = 741 total (was 731) |
| +1 :green_heart: | mvnsite | 4m 9s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 20s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 29s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 49s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 58s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 10s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 30s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 113m 41s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 11s | The patch does not generate ASF License warnings. |
| | | 319m 59s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestByteBufferPread |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestErasureCodingExerciseAPIs |
| | hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile |
| | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDFSOutputStream |
| | hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStream |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.tools.TestDFSAdminWithHA |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.TestLeaseRecovery |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/14/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 717e2aac5d70 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b5d9e2334b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/14/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/14/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/14/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/14/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/14/testReport/ |
| Max. process+thread count | 4386 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/14/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula merged pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula merged pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696772998
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 14s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 35s | trunk passed |
| +1 :green_heart: | compile | 24m 47s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 19m 58s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 14s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 51s | trunk passed |
| +1 :green_heart: | shadedclient | 23m 24s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 16s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 46s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 56s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 9m 1s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 36s | the patch passed |
| +1 :green_heart: | compile | 24m 13s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 24m 13s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 27 new + 136 unchanged - 27 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 24m 13s | the patch passed |
| +1 :green_heart: | compile | 20m 3s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 20m 3s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 30 new + 133 unchanged - 30 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 20m 3s | the patch passed |
| -0 :warning: | checkstyle | 3m 12s | root: The patch generated 4 new + 725 unchanged - 4 fixed = 729 total (was 729) |
| +1 :green_heart: | mvnsite | 3m 52s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 49s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 16s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 44s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 9m 53s | the patch passed |
||| _ Other Tests _ |
| -1 :x: | unit | 10m 33s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 29s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 132m 31s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 1s | The patch does not generate ASF License warnings. |
| | | 356m 13s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.security.TestRaceWhenRelogin |
| | hadoop.hdfs.TestQuota |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.TestDecommissionWithBackoffMonitor |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 491b66e1dc4c 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b5d9e2334b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/testReport/ |
| Max. process+thread count | 3270 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/12/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula edited a comment on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula edited a comment on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333
@huangtianhua and @YaYun-Wang thanks for contribution. @liuml07 thanks for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r476109273
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
##########
@@ -63,6 +63,12 @@ public static BlockStoragePolicySuite createDefaultSuite(
new StorageType[]{StorageType.DISK},
new StorageType[]{StorageType.DISK},
true); // Cannot be changed on regular files, but inherited.
+ final byte allNVDIMMId = HdfsConstants.StoragePolicy.ALL_NVDIMM.value();
Review comment:
“allnvdimmId” is more applicable in the situation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-668517822
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 52s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 45s | trunk passed |
| +1 :green_heart: | compile | 21m 44s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 18m 34s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 50s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 59s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 16s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 41s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 47s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 19s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 5s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 6s | the patch passed |
| +1 :green_heart: | compile | 21m 44s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 21m 44s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 14 new + 148 unchanged - 14 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 21m 44s | the patch passed |
| +1 :green_heart: | compile | 17m 54s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| -1 :x: | cc | 17m 54s | root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 38 new + 124 unchanged - 38 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 17m 54s | the patch passed |
| -0 :warning: | checkstyle | 2m 58s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) |
| +1 :green_heart: | mvnsite | 3m 53s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 15s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 8s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 8m 34s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 26s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 14s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 98m 33s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 2s | The patch does not generate ASF License warnings. |
| | | 292m 50s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 5b994e364d2f 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ab2b3df2de1 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/1/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/1/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/1/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/1/testReport/ |
| Max. process+thread count | 4289 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482248330
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
Review comment:
As we are adding only "NVDIMM",So I expect one line change here but I missed that you added one more param to enum.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus removed a comment on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus removed a comment on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-668517822
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482249623
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
As this will be co-exists. I was suggesting to change the method name to something like "isNvdimm" which was revlant here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r476644846
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
will RAM_DISK and NVDIMM co-exist..? if co-exist's, why can't we name NVDIM itself..?
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
Review comment:
Let's add only this line
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
##########
@@ -88,13 +92,14 @@ The effective storage policy can be retrieved by the "[`storagepolicies -getStor
### Configuration
* **dfs.storage.policy.enabled** - for enabling/disabling the storage policy feature. The default value is `true`.
-* **dfs.storage.default.policy** - Set the default storage policy with the policy name. The default value is `HOT`. All possible policies are defined in enum StoragePolicy, including `LAZY_PERSIST` `ALL_SSD` `ONE_SSD` `HOT` `WARM` `COLD` and `PROVIDED`.
+* **dfs.storage.default.policy** - Set the default storage policy with the policy name. The default value is `HOT`. All possible policies are defined in enum StoragePolicy, including `LAZY_PERSIST` `ALL_SSD` `ONE_SSD` `HOT` `WARM` `COLD` `ALL_NVDIMM` and `PROVIDED`.
Review comment:
can we maintain same order here also..?
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##########
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
is this will not valid anymore.?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r476144965
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
##########
@@ -63,6 +63,12 @@ public static BlockStoragePolicySuite createDefaultSuite(
new StorageType[]{StorageType.DISK},
new StorageType[]{StorageType.DISK},
true); // Cannot be changed on regular files, but inherited.
+ final byte allNVDIMMId = HdfsConstants.StoragePolicy.ALL_NVDIMM.value();
Review comment:
> The overall change looks good to me, thanks! I will finish the review of testing in 1/2 days and provide more input.
>
> I also will check which `isTransient()` will need to be replaced with `isRAM()`. It seems case by case for all usages. One simple question is for this NVDIMM storage type, we save the checksum file right?
The calculation of checksum in hadoop is the responsibility of clients and datanodes, each storage media including NVDIMM participates in the checksum of data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696788468
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 26s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 35s | trunk passed |
| +1 :green_heart: | compile | 24m 47s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 21m 59s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 25s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 35s | trunk passed |
| +1 :green_heart: | shadedclient | 24m 36s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 44s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 2s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 9m 38s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 12s | the patch passed |
| +1 :green_heart: | compile | 27m 37s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 27m 37s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 22 new + 141 unchanged - 22 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 27m 37s | the patch passed |
| +1 :green_heart: | compile | 24m 0s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 24m 0s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 25 new + 138 unchanged - 25 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 24m 0s | the patch passed |
| -0 :warning: | checkstyle | 3m 34s | root: The patch generated 16 new + 723 unchanged - 6 fixed = 739 total (was 729) |
| +1 :green_heart: | mvnsite | 4m 59s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 19m 21s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 56s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 47s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 11m 58s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 11m 45s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 39s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 107m 50s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 356m 14s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDFSClientRetries |
| | hadoop.hdfs.TestStripedFileAppend |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/16/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux b413f70212ad 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b5d9e2334b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/16/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/16/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/16/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/16/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/16/testReport/ |
| Max. process+thread count | 4004 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/16/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r476214970
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockStoragePolicySuite.java
##########
@@ -63,6 +63,12 @@ public static BlockStoragePolicySuite createDefaultSuite(
new StorageType[]{StorageType.DISK},
new StorageType[]{StorageType.DISK},
true); // Cannot be changed on regular files, but inherited.
+ final byte allNVDIMMId = HdfsConstants.StoragePolicy.ALL_NVDIMM.value();
Review comment:
Yes I agree the checksum calculation and read is built-in hadoop. I'm thinking of more about: do we need checksum for this storage type? For example, the RAM_DISK does not need checksum as far as I remember. This is RAM, but this also survives the service restarts, so checksum makes sense for data integrity. The code so far looks good regarding this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716471
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
> My final query then, why can't have one NVDIMM like one SSD as this also movable and peristent..?
Considering NVDIMM is faster, so NVDIMM does not use `FsDatasetCache()` which SSD needs in the design.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
##########
@@ -77,6 +77,9 @@
/** Returns true if the volume is NOT backed by persistent storage. */
boolean isTransientStorage();
Review comment:
> So, NVDIMM is peristent storage and RAM.
yes, that’s right.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-679794690
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 28s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 20s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 6s | trunk passed |
| +1 :green_heart: | compile | 21m 6s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 58s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 2m 49s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 54s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 39s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 24s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 52s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 48s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 32s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 38s | hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 2m 52s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | cc | 2m 52s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javac | 2m 52s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 2m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | cc | 2m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javac | 2m 27s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 2m 32s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) |
| -1 :x: | mvnsite | 0m 44s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| -1 :x: | shadedclient | 4m 22s | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 47s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 19s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | findbugs | 0m 43s | hadoop-hdfs in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 8m 59s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 2s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 0m 42s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 32s | The patch does not generate ASF License warnings. |
| | | 152m 12s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux ed937425961f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 64f36b9543c |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/testReport/ |
| Max. process+thread count | 1426 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/5/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r485463441
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
@brahmareddybattula would you please have a look for this? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483414755
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
> neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to determine whether FsDatasetCache is needed
I think we have agreed on this.
> I think it is better to use isRAM to determine whether to use mover
So just to be clear, if we use `isRAM()` to determine, both RAM_DISK and NVDIMM will return true and thus will simply disable Balancer and Mover. I was proposing that only when we do not want to move those NVDIMM data replicas around (disk/node) ever. Is that the design here? I'm fine with that but I think this contradicts with the statement that _"It can be regarded as a general hardware device. "_
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-679835368
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 30m 22s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 17s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 35s | trunk passed |
| +1 :green_heart: | compile | 20m 54s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 42s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 5s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 46s | trunk passed |
| +1 :green_heart: | shadedclient | 22m 43s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 12s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 37s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 1s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 38s | hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 2m 58s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | cc | 2m 58s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | javac | 2m 58s | root in the patch failed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. |
| -1 :x: | compile | 2m 31s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | cc | 2m 31s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -1 :x: | javac | 2m 31s | root in the patch failed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01. |
| -0 :warning: | checkstyle | 2m 49s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) |
| -1 :x: | mvnsite | 0m 43s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| -1 :x: | shadedclient | 4m 2s | patch has errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 33s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 4s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | findbugs | 0m 43s | hadoop-hdfs in the patch failed. |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 0s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 5s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 0m 46s | hadoop-hdfs in the patch failed. |
| +1 :green_heart: | asflicense | 0m 30s | The patch does not generate ASF License warnings. |
| | | 182m 10s | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux e925d20cb161 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 64f36b9543c |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| mvninstall | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-compile-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| compile | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| javac | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/diff-checkstyle-root.txt |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt |
| findbugs | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/testReport/ |
| Max. process+thread count | 1978 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/6/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697078022
No more comments on the latest commit.
Let's wait for the QA and run failing tests locally. I'll commit shortly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483125623
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
Balancer and mover will not move the blocks based on the `isTransient` ( they call getMovableTypes(..))..the blocks which NVDIMM shouldn't moved I feel,but as per this change it will move.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-668627236
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 32s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. |
| +0 :ok: | buf | 0m 0s | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 1m 1s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 1s | trunk passed |
| +1 :green_heart: | compile | 20m 18s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 42s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 48s | trunk passed |
| +1 :green_heart: | mvnsite | 3m 54s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 50s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 0s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 14s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 10s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 53s | the patch passed |
| +1 :green_heart: | compile | 19m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 19m 39s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 14 new + 148 unchanged - 14 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 19m 39s | the patch passed |
| +1 :green_heart: | compile | 18m 1s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| -1 :x: | cc | 18m 1s | root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 12 new + 150 unchanged - 12 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 18m 1s | the patch passed |
| -0 :warning: | checkstyle | 2m 52s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) |
| +1 :green_heart: | mvnsite | 3m 55s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 38s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 39s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 53s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 8m 28s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 26s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 12s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 105m 12s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 293m 57s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.namenode.TestINodeAttributeProvider |
| | hadoop.hdfs.server.blockmanagement.TestComputeInvalidateWork |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.server.datanode.TestNNHandlesBlockReportPerStorage |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfoStriped |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.server.namenode.TestINodeFile |
| | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/2/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 94191eced3cd 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e072d33327b |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/2/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/2/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/2/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/2/testReport/ |
| Max. process+thread count | 4309 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/2/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696769048
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 1m 10s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 21s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 3s | trunk passed |
| +1 :green_heart: | compile | 24m 5s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 20m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 12s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 14s | trunk passed |
| +1 :green_heart: | shadedclient | 23m 31s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 9s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 41s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 54s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 43s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 15s | the patch passed |
| +1 :green_heart: | compile | 24m 1s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 24m 1s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 10 new + 153 unchanged - 10 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 24m 1s | the patch passed |
| +1 :green_heart: | compile | 20m 48s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 20m 48s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 6 new + 157 unchanged - 6 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 20m 48s | the patch passed |
| -0 :warning: | checkstyle | 3m 7s | root: The patch generated 4 new + 725 unchanged - 4 fixed = 729 total (was 729) |
| +1 :green_heart: | mvnsite | 4m 11s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 16m 31s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 9s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 39s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 9m 18s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 10m 19s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 134m 19s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 357m 54s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux ffd8b78d5ab4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b5d9e2334b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/11/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/11/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/11/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/11/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/11/testReport/ |
| Max. process+thread count | 2920 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/11/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula merged pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula merged pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486535639
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
ok, By design if you dn't want to move.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492598805
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##########
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes() throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+ FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+ .setStorageDirectory(
+ new StorageDirectory(
+ StorageLocation.parse("[NVDIMM]"+volDir.getPath())))
+ .setStorageID("storage-id")
+ .setConf(conf)
+ .build();
+ assertEquals("", 3L, volume5.getReserved());
Review comment:
In order to be consistent with the original code, the `assertEquals()` here has three parameters, such as, lines 196 and 204 of the original code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r487387683
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
##########
@@ -2298,9 +2298,9 @@ private void cacheBlock(String bpid, long blockId) {
": volume was not an instance of FsVolumeImpl.");
return;
}
- if (volume.isTransientStorage()) {
+ if (volume.isRAMStorage()) {
LOG.warn("Caching not supported on block with id " + blockId +
- " since the volume is backed by RAM.");
+ " since the volume is backed by RAM_DISK or NVDIMM.");
Review comment:
nit: we can make it clear what the volume is.
```
LOG.warn("Caching not supported on block with id {} since the volume "
+ "is backed by {} which is RAM.", blockId, volume);
```
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/net/TestDFSNetworkTopology.java
##########
@@ -182,15 +212,16 @@ public void testGetStorageTypeInfo() throws Exception {
*/
@Test
public void testAddAndRemoveTopology() throws Exception {
- String[] newRack = {"/l1/d1/r1", "/l1/d1/r3", "/l1/d3/r3", "/l1/d3/r3"};
- String[] newHost = {"nhost1", "nhost2", "nhost3", "nhost4"};
+ String[] newRack = {"/l1/d1/r1", "/l1/d1/r3", "/l1/d3/r3", "/l1/d3/r3",
+ "/l1/d3/r4"};
+ String[] newHost = {"nhost1", "nhost2", "nhost3", "nhost4", "nhost5"};
String[] newips = {"30.30.30.30", "31.31.31.31", "32.32.32.32",
- "33.33.33.33"};
+ "33.33.33.33", "33.33.33.34"};
Review comment:
nit: why not use `34.34.34.34`?
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##########
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map<String,Object> storageTypeStats = (Map<String,Object>)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
- assertEquals(3l, storageTypeStats.get("nodesInService"));
+ assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
As this `if-else if-else if-else` getting longer, let's use switch case?
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataDirs.java
##########
@@ -43,14 +43,15 @@ public void testDataDirParsing() throws Throwable {
File dir5 = new File("/dir5");
File dir6 = new File("/dir6");
+ File dir7 = new File("/dir7");
// Verify that a valid string is correctly parsed, and that storage
// type is not case-sensitive and we are able to handle white-space between
// storage type and URI.
String locations1 = "[disk]/dir0,[DISK]/dir1,[sSd]/dir2,[disK]/dir3," +
- "[ram_disk]/dir4,[disk]/dir5, [disk] /dir6, [disk] ";
+ "[ram_disk]/dir4,[disk]/dir5, [disk] /dir6, [disk], [nvdimm]/dir7";
Review comment:
The invalid 8th URI has ending space deliberately for testing. Let's keep it, aka
```
"[ram_disk]/dir4,[disk]/dir5, [disk] /dir6, [disk] , [nvdimm]/dir7";
```
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/net/TestDFSNetworkTopology.java
##########
@@ -120,10 +125,11 @@ public void testGetStorageTypeInfo() throws Exception {
HashMap<String, EnumMap<StorageType, Integer>> d2info =
d2.getChildrenStorageInfo();
assertEquals(1, d2info.keySet().size());
- assertTrue(d2info.get("r3").size() == 3);
+ assertTrue(d2info.get("r3").size() == 4);
Review comment:
nit: replace with `assertEquals(4, d2info.get("r3").size())`;
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##########
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes() throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+ FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+ .setStorageDirectory(
+ new StorageDirectory(
+ StorageLocation.parse("[NVDIMM]"+volDir.getPath())))
+ .setStorageID("storage-id")
+ .setConf(conf)
+ .build();
+ assertEquals("", 3L, volume5.getReserved());
Review comment:
nit: Let's use better assertion statement `assertEquals(3L, volume5.getReserved());`
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/net/TestDFSNetworkTopology.java
##########
@@ -64,37 +64,42 @@ public void setupDatanodes() {
final String[] racks = {
"/l1/d1/r1", "/l1/d1/r1", "/l1/d1/r2", "/l1/d1/r2", "/l1/d1/r2",
- "/l1/d2/r3", "/l1/d2/r3", "/l1/d2/r3",
+ "/l1/d2/r3", "/l1/d2/r3", "/l1/d2/r3", "/l1/d2/r3",
"/l2/d3/r1", "/l2/d3/r2", "/l2/d3/r3", "/l2/d3/r4", "/l2/d3/r5",
"/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1",
- "/l2/d4/r1", "/l2/d4/r1"};
+ "/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r2",
+ "/l3/d5/r1", "/l3/d5/r1", "/l3/d5/r2"};
Review comment:
nit: Add a blank line before every new rack, aka
```
"/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1",
"/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r2",
"/l3/d5/r1", "/l3/d5/r1", "/l3/d5/r2"};
```
can be
```
"/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r1",
"/l2/d4/r1", "/l2/d4/r1", "/l2/d4/r2",
"/l3/d5/r1", "/l3/d5/r1", "/l3/d5/r2"};
```
Same to the `hosts`, please make the last three hosts in a new line so that `racks`, `hosts`, and `types` can be easily read with eyeballs.
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/net/TestDFSNetworkTopology.java
##########
@@ -182,15 +212,16 @@ public void testGetStorageTypeInfo() throws Exception {
*/
@Test
public void testAddAndRemoveTopology() throws Exception {
- String[] newRack = {"/l1/d1/r1", "/l1/d1/r3", "/l1/d3/r3", "/l1/d3/r3"};
- String[] newHost = {"nhost1", "nhost2", "nhost3", "nhost4"};
+ String[] newRack = {"/l1/d1/r1", "/l1/d1/r3", "/l1/d3/r3", "/l1/d3/r3",
+ "/l1/d3/r4"};
+ String[] newHost = {"nhost1", "nhost2", "nhost3", "nhost4", "nhost5"};
String[] newips = {"30.30.30.30", "31.31.31.31", "32.32.32.32",
- "33.33.33.33"};
+ "33.33.33.33", "33.33.33.34"};
StorageType[] newTypes = {StorageType.DISK, StorageType.SSD,
- StorageType.SSD, StorageType.SSD};
- DatanodeDescriptor[] newDD = new DatanodeDescriptor[4];
+ StorageType.SSD, StorageType.SSD, StorageType.NVDIMM};
+ DatanodeDescriptor[] newDD = new DatanodeDescriptor[5];
- for (int i = 0; i<4; i++) {
+ for (int i = 0; i<5; i++) {
Review comment:
nit: let's have space before and after `<` operator
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
##########
@@ -1103,6 +1103,8 @@ public void testSetQuota() throws Exception {
() -> webHdfs.setQuotaByStorageType(path, StorageType.SSD, -100));
LambdaTestUtils.intercept(IllegalArgumentException.class,
() -> webHdfs.setQuotaByStorageType(path, StorageType.RAM_DISK, 100));
+ LambdaTestUtils.intercept(IllegalArgumentException.class,
+ () -> webHdfs.setQuotaByStorageType(path, StorageType.NVDIMM, -100));
Review comment:
Also add a few happy test?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491857797
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##########
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map<String,Object> storageTypeStats = (Map<String,Object>)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
- assertEquals(3l, storageTypeStats.get("nodesInService"));
+ assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
I have not used Java 7 for a while, but I remember vaguely this is actually supported?
https://docs.oracle.com/javase/specs/jls/se7/html/jls-14.html
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##########
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map<String,Object> storageTypeStats = (Map<String,Object>)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
- assertEquals(3l, storageTypeStats.get("nodesInService"));
+ assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
Hadoop releases before 2.10 are all end of life (EoL). Hadoop 2.10 is the only version using Java 7. We do not need any support, compile or runtime, for Java versions before Java 7.
Hadoop 3.x are all using Java 8+. We do not need any Java 7 support in Hadoop 3.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
NVDIMM is special RAM, the data above can be stored persistently. It can be regarded as a general hardware device. We don't have to consider what storage type it is. Therefore, I think it is better to use isRAM to determine whether to use mover . In addition, neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to determine whether FsDatasetCache is needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-693753888
@liuml07 and @brahmareddybattula Sorry to disturb you again, I wonder if there is any update about this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698212333
@huangtianhua and @YaYun-Wang thanks for review. @liuml07 thanks for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696742400
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 36s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 17s | trunk passed |
| +1 :green_heart: | compile | 24m 31s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 20m 46s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 27s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 25s | trunk passed |
| +1 :green_heart: | shadedclient | 25m 57s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 37s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 17s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 3m 3s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 9m 9s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 7s | the patch passed |
| +1 :green_heart: | compile | 22m 40s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 22m 40s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 32 new + 131 unchanged - 32 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 22m 40s | the patch passed |
| +1 :green_heart: | compile | 18m 59s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 18m 59s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 7 new + 156 unchanged - 7 fixed = 163 total (was 163) |
| +1 :green_heart: | javac | 18m 59s | the patch passed |
| -0 :warning: | checkstyle | 2m 57s | root: The patch generated 4 new + 725 unchanged - 4 fixed = 729 total (was 729) |
| +1 :green_heart: | mvnsite | 4m 4s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 14s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 28s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 49s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 37s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 30s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 18s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 97m 36s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | The patch does not generate ASF License warnings. |
| | | 318m 7s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestSnapshotCommands |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.TestFileChecksumCompositeCrc |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestFileChecksum |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/10/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 9e6810db2f4d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6b5d9e2334b |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/10/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/10/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/10/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/10/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/10/testReport/ |
| Max. process+thread count | 3986 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/10/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492866266
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##########
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes() throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+ FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+ .setStorageDirectory(
+ new StorageDirectory(
+ StorageLocation.parse("[NVDIMM]"+volDir.getPath())))
+ .setStorageID("storage-id")
+ .setConf(conf)
+ .build();
+ assertEquals("", 3L, volume5.getReserved());
Review comment:
Usually we can, but following original code style here is bad. When it fails, the original code gives up empty string. My code shows your expected value and actual value so you can debug. Please change it.
We can also file a JIRA to update all such cases where `assertEquals` can be improved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483780758
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
Thanks for clarification, @YaYun-Wang I think now we both are on the same page. @brahmareddybattula Does this make sense to you? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r476679977
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
> why can't we name NVDIM itself..?
I don't think I get the question totally. Yes they can co-exist. You can have some folders being RAM_DISK storage and some folders being NVDIMM storage. RAM_DISK will get data lost after process restart, so it is transient. NVDIMM is not transient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r482341228
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
No. Both RAM_DISK and NVDIMM will return true here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486716471
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
> My final query then, why can't have one NVDIMM like one SSD as this also movable and peristent..?
Considering NVDIMM is faster, so NVDIMM does not use `FsDatasetCache()` which SSD needs in the design.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-698076876
Yeah; for this checkstyle in test we can keep it as-is for the sake of code consistency. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-693756560
Sorry, I have been busy in other stuff. The recent review I have no other comments except a few trivial ones. I'll post that and commit this week.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-668645247
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 35s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 46s | trunk passed |
| +1 :green_heart: | compile | 20m 55s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 17m 34s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | checkstyle | 2m 56s | trunk passed |
| -1 :x: | mvnsite | 1m 11s | hadoop-hdfs in trunk failed. |
| +1 :green_heart: | shadedclient | 20m 46s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 45s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 3m 50s | trunk passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +0 :ok: | spotbugs | 3m 24s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 11s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 59s | the patch passed |
| +1 :green_heart: | compile | 20m 1s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 20m 0s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 33 new + 129 unchanged - 33 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 20m 0s | the patch passed |
| +1 :green_heart: | compile | 17m 34s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| -1 :x: | cc | 17m 34s | root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09 with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 generated 30 new + 132 unchanged - 30 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 17m 34s | the patch passed |
| -0 :warning: | checkstyle | 2m 50s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) |
| +1 :green_heart: | mvnsite | 3m 58s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 15m 33s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 1m 43s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 21s | the patch passed with JDK Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| +1 :green_heart: | findbugs | 9m 58s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 59s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 27s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 101m 28s | hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 1m 6s | The patch generated 2 ASF License warnings. |
| | | 294m 40s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.TestBlockStoragePolicy |
| | hadoop.hdfs.TestDFSInotifyEventInputStream |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.qjournal.TestMiniJournalCluster |
| | hadoop.hdfs.qjournal.client.TestQJMWithFaults |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestQuotaAllowOwner |
| | hadoop.hdfs.qjournal.TestNNWithQJM |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux 2dcb307c6b03 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e072d33327b |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| mvnsite | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_252-8u252-b09-1~18.04-b09.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/testReport/ |
| asflicense | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/artifact/out/patch-asflicense-problems.txt |
| Max. process+thread count | 4099 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/3/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690839476
Will check again later this week.
Ideally we can get a clean QA. Could you check the test failures and make sure they are not related? Thanks,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690829884
@liuml07 and @brahmareddybattula So if all of the codes are ok, would you please to approve? Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
NVDIMM is special RAM, the data above can be stored persistently. It can be regarded as a general hardware device. We don't have to consider what storage type it is, balancer and mover can be applied on NVDIMM, therefore, I think it is better to use isRAM to determine whether to use mover . In addition, neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to determine whether FsDatasetCache is needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r491852471
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
##########
@@ -145,9 +150,11 @@ public void testStorageTypeStatsJMX() throws Exception {
Map<String,Object> storageTypeStats = (Map<String,Object>)entry.get("value");
typesPresent.add(storageType);
if (storageType.equals("ARCHIVE") || storageType.equals("DISK") ) {
- assertEquals(3l, storageTypeStats.get("nodesInService"));
+ assertEquals(3L, storageTypeStats.get("nodesInService"));
Review comment:
`storageType` is a parameter of "java.lang.String" , and `switch()` does not support "java.lang.String" before java 1.7. So, will `if-else ` be more appropriate here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-680017965
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| +0 :ok: | reexec | 0m 30s | Docker mode activated. |
||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. |
| +0 :ok: | buf | 0m 1s | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | markdownlint was not available. |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 16 new or modified test files. |
||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 3m 18s | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 2s | trunk passed |
| +1 :green_heart: | compile | 19m 26s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | compile | 16m 59s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | checkstyle | 3m 2s | trunk passed |
| +1 :green_heart: | mvnsite | 4m 9s | trunk passed |
| +1 :green_heart: | shadedclient | 21m 37s | branch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 35s | trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 1s | trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +0 :ok: | spotbugs | 2m 34s | Used deprecated FindBugs config; considering switching to SpotBugs. |
| +1 :green_heart: | findbugs | 8m 1s | trunk passed |
||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 44s | the patch passed |
| +1 :green_heart: | compile | 18m 57s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| -1 :x: | cc | 18m 57s | root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 14 new + 148 unchanged - 14 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 18m 57s | the patch passed |
| +1 :green_heart: | compile | 16m 52s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| -1 :x: | cc | 16m 52s | root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 13 new + 149 unchanged - 13 fixed = 162 total (was 162) |
| +1 :green_heart: | javac | 16m 52s | the patch passed |
| -0 :warning: | checkstyle | 2m 58s | root: The patch generated 3 new + 725 unchanged - 4 fixed = 728 total (was 729) |
| +1 :green_heart: | mvnsite | 4m 9s | the patch passed |
| +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. |
| +1 :green_heart: | shadedclient | 14m 3s | patch has no errors when building and testing our client artifacts. |
| +1 :green_heart: | javadoc | 2m 35s | the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 |
| +1 :green_heart: | javadoc | 4m 5s | the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| +1 :green_heart: | findbugs | 8m 29s | the patch passed |
||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 24s | hadoop-common in the patch passed. |
| +1 :green_heart: | unit | 2m 20s | hadoop-hdfs-client in the patch passed. |
| -1 :x: | unit | 95m 10s | hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | The patch does not generate ASF License warnings. |
| | | 291m 43s | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.fs.contract.hdfs.TestHDFSContractMultipartUploader |
| | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
| | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier |
| | hadoop.hdfs.TestStripedFileAppend |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/7/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/2189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle cc buflint bufcompat xml markdownlint |
| uname | Linux ec4103513b05 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c0492962355 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/7/artifact/out/diff-compile-cc-root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt |
| cc | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/7/artifact/out/diff-compile-cc-root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt |
| checkstyle | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/7/artifact/out/diff-checkstyle-root.txt |
| unit | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/7/testReport/ |
| Max. process+thread count | 4623 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2189/7/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
hadoop-yetus commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-696742400
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r483410183
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
NVDIMM is RAM, however, the data above can be stored persistently. It can be regarded as a general hardware device. We don't have to consider what storage type it is. Therefore, I think it is better to use isRAM to determine whether to use mover . In addition, neither RAM nor nvdimm need FsDatasetCache, and isTransient() used to determine whether FsDatasetCache is needed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] huangtianhua commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
huangtianhua commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-690829884
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] brahmareddybattula commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
brahmareddybattula commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r486549682
##########
File path: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StorageType.java
##########
@@ -34,28 +34,35 @@
@InterfaceStability.Unstable
public enum StorageType {
// sorted by the speed of the storage types, from fast to slow
- RAM_DISK(true),
- SSD(false),
- DISK(false),
- ARCHIVE(false),
- PROVIDED(false);
+ RAM_DISK(true, true),
+ NVDIMM(false, true),
+ SSD(false, false),
+ DISK(false, false),
+ ARCHIVE(false, false),
+ PROVIDED(false, false);
private final boolean isTransient;
+ private final boolean isRAM;
public static final StorageType DEFAULT = DISK;
public static final StorageType[] EMPTY_ARRAY = {};
private static final StorageType[] VALUES = values();
- StorageType(boolean isTransient) {
+ StorageType(boolean isTransient, boolean isRAM) {
this.isTransient = isTransient;
+ this.isRAM = isRAM;
}
public boolean isTransient() {
return isTransient;
}
+ public boolean isRAM() {
+ return isRAM;
+ }
Review comment:
My final query then, why can't have one NVDIMM like one SSD as this also movable and peristent..?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] liuml07 commented on pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
liuml07 commented on pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#issuecomment-697078022
No more comments on the latest commit.
Let's wait for the QA and run failing tests locally. I'll commit shortly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
[GitHub] [hadoop] YaYun-Wang commented on a change in pull request #2189: HDFS-15025. Applying NVDIMM storage media to HDFS
Posted by GitBox <gi...@apache.org>.
YaYun-Wang commented on a change in pull request #2189:
URL: https://github.com/apache/hadoop/pull/2189#discussion_r492598805
##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsVolumeList.java
##########
@@ -202,6 +205,14 @@ public void testDfsReservedForDifferentStorageTypes() throws IOException {
.setConf(conf)
.build();
assertEquals("", 100L, volume4.getReserved());
+ FsVolumeImpl volume5 = new FsVolumeImplBuilder().setDataset(dataset)
+ .setStorageDirectory(
+ new StorageDirectory(
+ StorageLocation.parse("[NVDIMM]"+volDir.getPath())))
+ .setStorageID("storage-id")
+ .setConf(conf)
+ .build();
+ assertEquals("", 3L, volume5.getReserved());
Review comment:
In order to be consistent with the original code, the `assertEquals()` here has three parameters, such as, lines 196 and 204 of the original code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org