[GitHub] [hadoop] Hexiaoqiao commented on a change in pull request #2288: HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount

[GitBox <gi...@apache.org>]

Hexiaoqiao commented on a change in pull request #2288:
URL: https://github.com/apache/hadoop/pull/2288#discussion_r490219801



##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
##########
@@ -134,6 +134,9 @@
   private final FileIoProvider fileIoProvider;
   private final DataNodeVolumeMetrics metrics;
   private URI baseURI;
+  private boolean enableSameDiskArchival;
+  private final String device;

Review comment:
       what about using `storageID` replace `device`? IMO both of them are in order to index single volume, right?

##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
##########
@@ -412,16 +435,31 @@ long getBlockPoolUsed(String bpid) throws IOException {
    */
   @VisibleForTesting
   public long getCapacity() {
+    long capacity;
     if (configuredCapacity < 0L) {
       long remaining;
       if (cachedCapacity > 0L) {
         remaining = cachedCapacity - getReserved();
       } else {
         remaining = usage.getCapacity() - getReserved();
       }
-      return Math.max(remaining, 0L);
+      capacity = Math.max(remaining, 0L);
+    } else {
+      capacity = configuredCapacity;
+    }
+
+    if (enableSameDiskArchival) {
+      double reservedForArchival = conf.getDouble(

Review comment:
       `reservedForArchival` here is same as `this.reservedForArchival` when init?

##########
File path: hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
##########
@@ -190,6 +193,26 @@
     }
     this.conf = conf;
     this.fileIoProvider = fileIoProvider;
+    this.enableSameDiskArchival =
+        conf.getBoolean(DFSConfigKeys.DFS_DATANODE_ALLOW_SAME_DISK_TIERING,
+            DFSConfigKeys.DFS_DATANODE_ALLOW_SAME_DISK_TIERING_DEFAULT);
+    if (enableSameDiskArchival) {
+      this.device = usage.getMount();
+      reservedForArchive = conf.getDouble(
+          DFSConfigKeys.DFS_DATANODE_RESERVE_FOR_ARCHIVE_PERCENTAGE,
+          DFSConfigKeys.DFS_DATANODE_RESERVE_FOR_ARCHIVE_PERCENTAGE_DEFAULT);
+      if (reservedForArchive >= 1) {
+        FsDatasetImpl.LOG.warn("Value of reserve-for-archival is >= 100% for "
+            + currentDir + ". Setting it to 99%.");
+        reservedForArchive = 0.99;

Review comment:
       Why `reservedForArchive` has to less than 1 here, IIUC it means that this is ARCHIVE device when `reservedForArchive` set to 1. Right?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org