You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by we...@apache.org on 2021/03/31 03:26:21 UTC
[hadoop] branch branch-3.3 updated (56ef164 -> ab451b8)
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a change to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git.
from 56ef164 HADOOP-17222. Create socket address leveraging URI cache (#2817)
new 7f20fad HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.
new 769df1e HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366)
new c4f7b85 HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.
new a407f7d HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li.
new 53b024a HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module
new 8728394 HDFS-15331. Remove invalid exclusions that minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji
new ab451b8 HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago.
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/hadoop/hdfs/ClientGSIContext.java | 2 +-
.../java/org/apache/hadoop/hdfs/DFSClient.java | 2 +-
.../apache/hadoop/hdfs/protocol/HdfsConstants.java | 3 +-
.../router/security/token/package-info.java | 3 +-
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 9 +++-
.../main/java/org/apache/hadoop/hdfs/DFSUtil.java | 2 -
.../BlockPlacementPolicyDefault.java | 55 ++++++++++++++++++++--
.../DatanodeAdminDefaultMonitor.java | 2 +-
.../server/blockmanagement/DatanodeManager.java | 6 +++
.../server/blockmanagement/FSClusterStats.java | 9 ++++
.../server/blockmanagement/StorageTypeStats.java | 16 +++++++
.../hdfs/server/datanode/BPServiceActor.java | 12 +++++
.../snapshot/DirectorySnapshottableFeature.java | 3 +-
.../org/apache/hadoop/hdfs/tools/DFSAdmin.java | 1 -
.../src/main/resources/hdfs-default.xml | 15 +++++-
.../blockmanagement/TestBlockStatsMXBean.java | 52 ++++++++++++++++++++
.../blockmanagement/TestReplicationPolicy.java | 29 ++++++++++++
.../hdfs/server/datanode/TestBPOfferService.java | 24 +++++++++-
hadoop-minicluster/pom.xml | 6 ---
19 files changed, 228 insertions(+), 23 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 07/07: HDFS-15651. Client could not obtain block when DN
CommandProcessingThread exit. Contributed by Aiphago.
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit ab451b8e428defee8836beeca57ec0fc741383f3
Author: He Xiaoqiao <he...@apache.org>
AuthorDate: Wed Nov 4 13:53:46 2020 +0800
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago.
Reviewed-by: He Xiaoqiao <he...@apache.org>
Reviewed-by: Yiqun Lin <yq...@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
---
.../hdfs/server/datanode/BPServiceActor.java | 12 +++++++++++
.../hdfs/server/datanode/TestBPOfferService.java | 24 +++++++++++++++++++++-
2 files changed, 35 insertions(+), 1 deletion(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
index 49156c2..0cc9fac 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
@@ -1312,6 +1312,10 @@ class BPServiceActor implements Runnable {
processQueue();
} catch (Throwable t) {
LOG.error("{} encountered fatal exception and exit.", getName(), t);
+ runningState = RunningState.FAILED;
+ } finally {
+ LOG.warn("Ending command processor service for: " + this);
+ shouldServiceRun = false;
}
}
@@ -1327,6 +1331,7 @@ class BPServiceActor implements Runnable {
dn.getMetrics().incrNumProcessedCommands();
} catch (InterruptedException e) {
LOG.error("{} encountered interrupt and exit.", getName());
+ Thread.currentThread().interrupt();
// ignore unless thread was specifically interrupted.
if (Thread.interrupted()) {
break;
@@ -1398,4 +1403,11 @@ class BPServiceActor implements Runnable {
dn.getMetrics().incrActorCmdQueueLength(1);
}
}
+
+ @VisibleForTesting
+ void stopCommandProcessingThread() {
+ if (commandProcessingThread != null) {
+ commandProcessingThread.interrupt();
+ }
+ }
}
\ No newline at end of file
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
index 9b47884..a344011 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
@@ -1216,4 +1216,26 @@ public class TestBPOfferService {
}
}
}
-}
\ No newline at end of file
+
+ @Test(timeout = 5000)
+ public void testCommandProcessingThreadExit() throws Exception {
+ Configuration conf = new HdfsConfiguration();
+ MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).
+ numDataNodes(1).build();
+ try {
+ List<DataNode> datanodes = cluster.getDataNodes();
+ DataNode dataNode = datanodes.get(0);
+ List<BPOfferService> allBpOs = dataNode.getAllBpOs();
+ BPOfferService bpos = allBpOs.get(0);
+ waitForInitialization(bpos);
+ BPServiceActor actor = bpos.getBPServiceActors().get(0);
+ // Stop and wait util actor exit.
+ actor.stopCommandProcessingThread();
+ GenericTestUtils.waitFor(() -> !actor.isAlive(), 100, 3000);
+ } finally {
+ if (cluster != null) {
+ cluster.shutdown();
+ }
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 04/07: HDFS-15507. [JDK 11] Fix javadoc errors in
hadoop-hdfs-client module. Contributed by Xieming Li.
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit a407f7de5afc5a07e9016788d85dc7e582833e34
Author: Akira Ajisaka <aa...@apache.org>
AuthorDate: Tue Aug 11 13:49:56 2020 +0900
HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li.
(cherry picked from commit 32895f4f7ea51522e779e68a030bfc9515f40798)
---
.../src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java | 2 +-
.../src/main/java/org/apache/hadoop/hdfs/DFSClient.java | 2 +-
.../src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java | 3 ++-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
index 9b324bd..4de9696 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ClientGSIContext.java
@@ -29,7 +29,7 @@ import java.util.concurrent.atomic.LongAccumulator;
/**
* Global State Id context for the client.
- * <p/>
+ * <p>
* This is the client side implementation responsible for receiving
* state alignment info from server(s).
*/
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
index b210f91..9e532f2 100755
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
@@ -1245,7 +1245,7 @@ public class DFSClient implements java.io.Closeable, RemotePeerFactory,
/**
* Same as {@link #create(String, FsPermission, EnumSet, boolean, short, long,
- * addition of Progressable, int, ChecksumOpt, InetSocketAddress[], String)}
+ * Progressable, int, ChecksumOpt, InetSocketAddress[], String)}
* with the storagePolicy that is used to specify a specific storage policy
* instead of inheriting any policy from this new file's parent directory.
* This policy will be persisted in HDFS. A value of null means inheriting
diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
index eb3dc4e..615acad 100755
--- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java
@@ -22,6 +22,7 @@ import java.util.Map;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.client.HdfsClientConfigKeys;
import org.apache.hadoop.util.StringUtils;
@InterfaceAudience.Private
@@ -110,7 +111,7 @@ public final class HdfsConstants {
* period, no other client can write to the file. The writing client can
* periodically renew the lease. When the file is closed, the lease is
* revoked. The lease duration is bound by this soft limit and a
- * {@link HdfsConstants#LEASE_HARDLIMIT_PERIOD hard limit}. Until the
+ * {@link HdfsClientConfigKeys#DFS_LEASE_HARDLIMIT_KEY }. Until the
* soft limit expires, the writer has sole write access to the file. If the
* soft limit expires and the client fails to close the file or renew the
* lease, another client can preempt the lease.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 05/07: HDFS-15508. [JDK 11] Fix javadoc errors in
hadoop-hdfs-rbf module
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 53b024a2b7c1702887081de2a22eea28b62943c9
Author: Akira Ajisaka <aa...@apache.org>
AuthorDate: Mon Aug 3 14:39:13 2020 +0900
HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module
(cherry picked from commit 9062814bab7d6087c584e03b0b16eaa60b31f60a)
---
.../hdfs/server/federation/router/security/token/package-info.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/package-info.java b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/package-info.java
index a51e455..ae65c8f 100644
--- a/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/package-info.java
+++ b/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/security/token/package-info.java
@@ -18,7 +18,8 @@
/**
* Includes implementations of token secret managers.
- * Implementations should extend {@link AbstractDelegationTokenSecretManager}.
+ * Implementations should extend
+ * {@link org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager}.
*/
@InterfaceAudience.Private
@InterfaceStability.Evolving
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 02/07: HDFS-15253 Default checkpoint transfer speed,
50mb per second (#2366)
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 769df1e964299a86421397f1fde348a567ceb366
Author: Karthik Palanisamy <kp...@cloudera.com>
AuthorDate: Wed Oct 7 09:39:04 2020 -0700
HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366)
(cherry picked from commit df4006eb8139a08b92b5a94adfde544de978137c)
---
.../src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 2 +-
hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 75b7389..1cf7cb1 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -983,7 +983,7 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
public static final String DFS_IMAGE_TRANSFER_RATE_KEY =
"dfs.image.transfer.bandwidthPerSec";
- public static final long DFS_IMAGE_TRANSFER_RATE_DEFAULT = 0; //no throttling
+ public static final long DFS_IMAGE_TRANSFER_RATE_DEFAULT = 52428800;
public static final String DFS_IMAGE_TRANSFER_BOOTSTRAP_STANDBY_RATE_KEY =
"dfs.image.transfer-bootstrap-standby.bandwidthPerSec";
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index e63303f..ce9195c 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -1439,13 +1439,13 @@
<property>
<name>dfs.image.transfer.bandwidthPerSec</name>
- <value>0</value>
+ <value>52428800</value>
<description>
Maximum bandwidth used for regular image transfers (instead of
bootstrapping the standby namenode), in bytes per second.
This can help keep normal namenode operations responsive during
checkpointing.
- A default value of 0 indicates that throttling is disabled.
+ A default value is 50mb per second.
The maximum bandwidth used for bootstrapping standby namenode is
configured with dfs.image.transfer-bootstrap-standby.bandwidthPerSec.
Support multiple size unit suffix(case insensitive), as described
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 03/07: HDFS-15506. [JDK 11] Fix javadoc errors in
hadoop-hdfs module. Contributed by Xieming Li.
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit c4f7b8578b14ec026b10e18b139a669f5982497f
Author: Akira Ajisaka <aa...@apache.org>
AuthorDate: Tue Aug 11 11:14:58 2020 +0900
HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.
(cherry picked from commit 7938ebfb9dd12b874ced1798f0c3ad0c160cb465)
---
.../src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 2 +-
.../hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java | 2 --
.../hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java | 2 +-
.../hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java | 3 ++-
.../src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java | 1 -
5 files changed, 4 insertions(+), 6 deletions(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index 1cf7cb1..965ea8d 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -682,7 +682,7 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
600;
/**
* The maximum number of getBlocks RPCs data movement utilities can make to
- * a NameNode per second. Values <= 0 disable throttling. This affects
+ * a NameNode per second. Values <= 0 disable throttling. This affects
* anything that uses a NameNodeConnector, i.e., the Balancer, Mover,
* and StoragePolicySatisfier.
*/
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
index ad61257..efb2a8b 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
@@ -39,7 +39,6 @@ import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_SERVER_HTTPS_TRUSTSTORE_P
import java.io.ByteArrayInputStream;
import java.io.DataInputStream;
-import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.PrintStream;
import java.net.InetAddress;
@@ -1777,7 +1776,6 @@ public class DFSUtil {
* was found.
* @throws ParentNotDirectoryException
* @throws UnresolvedLinkException
- * @throws FileNotFoundException
*/
public static void checkProtectedDescendants(
FSDirectory fsd, INodesInPath iip)
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
index b372e73..dfa7bf4 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
@@ -40,7 +40,7 @@ import java.util.Iterator;
/**
* Checks to see if datanodes have finished DECOMMISSION_INPROGRESS or
* ENTERING_MAINTENANCE state.
- * <p/>
+ * <p>
* Since this is done while holding the namesystem lock,
* the amount of work per monitor tick is limited.
*/
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
index 85e5ae4..234f7b9 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectorySnapshottableFeature.java
@@ -173,7 +173,8 @@ public class DirectorySnapshottableFeature extends DirectoryWithSnapshotFeature
* Add a snapshot.
* @param snapshotRoot Root of the snapshot.
* @param name Name of the snapshot.
- * @param mtime The snapshot creation time set by Time.now().
+ * @param leaseManager
+ * @param captureOpenFiles
* @throws SnapshotException Throw SnapshotException when there is a snapshot
* with the same name already exists or snapshot quota exceeds
*/
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
index 5344d14..b57637d 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
@@ -2184,7 +2184,6 @@ public class DFSAdmin extends FsShell {
/**
* @param argv The parameters passed to this program.
- * @exception Exception if the filesystem does not exist.
* @return 0 on success, non zero on error.
*/
@Override
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 01/07: HDFS-14383. Compute datanode load based on
StoragePolicy. Contributed by Ayush Saxena.
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 7f20fad41912217c03cdd7f2d54c01e40f49d051
Author: Ayush Saxena <ay...@apache.org>
AuthorDate: Mon Oct 19 10:48:47 2020 +0530
HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.
(cherry picked from commit 2e8cafac3b071fe5b943542827fd8a496b137fa9)
---
.../java/org/apache/hadoop/hdfs/DFSConfigKeys.java | 5 ++
.../BlockPlacementPolicyDefault.java | 55 ++++++++++++++++++++--
.../server/blockmanagement/DatanodeManager.java | 6 +++
.../server/blockmanagement/FSClusterStats.java | 9 ++++
.../server/blockmanagement/StorageTypeStats.java | 16 +++++++
.../src/main/resources/hdfs-default.xml | 11 +++++
.../blockmanagement/TestBlockStatsMXBean.java | 52 ++++++++++++++++++++
.../blockmanagement/TestReplicationPolicy.java | 29 ++++++++++++
8 files changed, 180 insertions(+), 3 deletions(-)
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
index f7da4cb..75b7389 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
@@ -239,6 +239,11 @@ public class DFSConfigKeys extends CommonConfigurationKeys {
HdfsClientConfigKeys.DeprecatedKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_KEY;
public static final boolean DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_DEFAULT =
true;
+ public static final String
+ DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_KEY =
+ "dfs.namenode.redundancy.considerLoadByStorageType";
+ public static final boolean
+ DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_DEFAULT = false;
public static final String DFS_NAMENODE_READ_CONSIDERLOAD_KEY =
"dfs.namenode.read.considerLoad";
public static final boolean DFS_NAMENODE_READ_CONSIDERLOAD_DEFAULT =
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
index a1a83b0..5761690 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
@@ -17,6 +17,8 @@
*/
package org.apache.hadoop.hdfs.server.blockmanagement;
+import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_DEFAULT;
+import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_KEY;
import static org.apache.hadoop.util.Time.monotonicNow;
import java.util.*;
@@ -92,7 +94,8 @@ public class BlockPlacementPolicyDefault extends BlockPlacementPolicy {
}
}
- protected boolean considerLoad;
+ protected boolean considerLoad;
+ private boolean considerLoadByStorageType;
protected double considerLoadFactor;
private boolean preferLocalNode;
protected NetworkTopology clusterMap;
@@ -116,6 +119,9 @@ public class BlockPlacementPolicyDefault extends BlockPlacementPolicy {
this.considerLoad = conf.getBoolean(
DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_KEY,
DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_DEFAULT);
+ this.considerLoadByStorageType = conf.getBoolean(
+ DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_KEY,
+ DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_DEFAULT);
this.considerLoadFactor = conf.getDouble(
DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_FACTOR,
DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_FACTOR_DEFAULT);
@@ -976,8 +982,9 @@ public class BlockPlacementPolicyDefault extends BlockPlacementPolicy {
* @return Return true if the datanode should be excluded, otherwise false
*/
boolean excludeNodeByLoad(DatanodeDescriptor node){
- final double maxLoad = considerLoadFactor *
- stats.getInServiceXceiverAverage();
+ double inServiceXceiverCount = getInServiceXceiverAverage(node);
+ final double maxLoad = considerLoadFactor * inServiceXceiverCount;
+
final int nodeLoad = node.getXceiverCount();
if ((nodeLoad > maxLoad) && (maxLoad > 0)) {
logNodeIsNotChosen(node, NodeNotChosenReason.NODE_TOO_BUSY,
@@ -988,6 +995,48 @@ public class BlockPlacementPolicyDefault extends BlockPlacementPolicy {
}
/**
+ * Gets the inServiceXceiver average count for the cluster, if
+ * considerLoadByStorageType is true, then load is calculated only for the
+ * storage types present on the datanode.
+ * @param node the datanode whose storage types are to be taken into account.
+ * @return the InServiceXceiverAverage count.
+ */
+ private double getInServiceXceiverAverage(DatanodeDescriptor node) {
+ double inServiceXceiverCount;
+ if (considerLoadByStorageType) {
+ inServiceXceiverCount =
+ getInServiceXceiverAverageByStorageType(node.getStorageTypes());
+ } else {
+ inServiceXceiverCount = stats.getInServiceXceiverAverage();
+ }
+ return inServiceXceiverCount;
+ }
+
+ /**
+ * Gets the average xceiver count with respect to the storage types.
+ * @param storageTypes the storage types.
+ * @return the average xceiver count wrt the provided storage types.
+ */
+ private double getInServiceXceiverAverageByStorageType(
+ Set<StorageType> storageTypes) {
+ double avgLoad = 0;
+ final Map<StorageType, StorageTypeStats> storageStats =
+ stats.getStorageTypeStats();
+ int numNodes = 0;
+ int numXceiver = 0;
+ for (StorageType s : storageTypes) {
+ StorageTypeStats storageTypeStats = storageStats.get(s);
+ numNodes += storageTypeStats.getNodesInService();
+ numXceiver += storageTypeStats.getNodesInServiceXceiverCount();
+ }
+ if (numNodes != 0) {
+ avgLoad = (double) numXceiver / numNodes;
+ }
+
+ return avgLoad;
+ }
+
+ /**
* Determine if a datanode is good for placing block.
*
* @param node The target datanode
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
index 22750ec..01dfe04 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
@@ -24,6 +24,7 @@ import org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTest
import org.apache.hadoop.thirdparty.com.google.common.base.Preconditions;
import org.apache.hadoop.thirdparty.com.google.common.net.InetAddresses;
+import org.apache.hadoop.fs.StorageType;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.HadoopIllegalArgumentException;
@@ -1978,6 +1979,11 @@ public class DatanodeManager {
}
return avgLoad;
}
+
+ @Override
+ public Map<StorageType, StorageTypeStats> getStorageTypeStats() {
+ return heartbeatManager.getStorageTypeStats();
+ }
};
}
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/FSClusterStats.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/FSClusterStats.java
index 556b7fc..1412295 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/FSClusterStats.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/FSClusterStats.java
@@ -18,6 +18,9 @@
package org.apache.hadoop.hdfs.server.blockmanagement;
import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.fs.StorageType;
+
+import java.util.Map;
/**
* This interface is used for retrieving the load related statistics of
@@ -57,4 +60,10 @@ public interface FSClusterStats {
* writes that are currently occurring on the cluster.
*/
public double getInServiceXceiverAverage();
+
+ /**
+ * Indicates the storage statistics per storage type.
+ * @return storage statistics per storage type.
+ */
+ Map<StorageType, StorageTypeStats> getStorageTypeStats();
}
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java
index c335ec6..f90dbad 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/StorageTypeStats.java
@@ -20,6 +20,7 @@ package org.apache.hadoop.hdfs.server.blockmanagement;
import java.beans.ConstructorProperties;
+import org.apache.hadoop.thirdparty.com.google.common.annotations.VisibleForTesting;
import org.apache.hadoop.classification.InterfaceAudience;
import org.apache.hadoop.classification.InterfaceStability;
import org.apache.hadoop.fs.StorageType;
@@ -39,6 +40,15 @@ public class StorageTypeStats {
private int nodesInService = 0;
private StorageType storageType;
+ @VisibleForTesting
+ void setDataNodesInServiceXceiverCount(int avgXceiverPerDatanode,
+ int numNodesInService) {
+ this.nodesInService = numNodesInService;
+ this.nodesInServiceXceiverCount = numNodesInService * avgXceiverPerDatanode;
+ }
+
+ private int nodesInServiceXceiverCount;
+
@ConstructorProperties({"capacityTotal", "capacityUsed", "capacityNonDfsUsed",
"capacityRemaining", "blockPoolUsed", "nodesInService"})
public StorageTypeStats(
@@ -101,6 +111,10 @@ public class StorageTypeStats {
return nodesInService;
}
+ public int getNodesInServiceXceiverCount() {
+ return nodesInServiceXceiverCount;
+ }
+
StorageTypeStats(StorageType storageType) {
this.storageType = storageType;
}
@@ -131,6 +145,7 @@ public class StorageTypeStats {
void addNode(final DatanodeDescriptor node) {
if (node.isInService()) {
nodesInService++;
+ nodesInServiceXceiverCount += node.getXceiverCount();
}
}
@@ -151,6 +166,7 @@ public class StorageTypeStats {
void subtractNode(final DatanodeDescriptor node) {
if (node.isInService()) {
nodesInService--;
+ nodesInServiceXceiverCount -= node.getXceiverCount();
}
}
}
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index d2cb3c7..e63303f 100755
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -314,6 +314,17 @@
</property>
<property>
+ <name>dfs.namenode.redundancy.considerLoadByStorageType</name>
+ <value>false</value>
+ <description>
+ Decide if chooseTarget considers the target's load with respect to the
+ storage type. Typically to be used when datanodes contain homogenous
+ storage types. Irrelevent if dfs.namenode.redundancy.considerLoad is
+ false.
+ </description>
+ </property>
+
+ <property>
<name>dfs.namenode.redundancy.considerLoad.factor</name>
<value>2.0</value>
<description>The factor by which a node's load can exceed the average
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
index 81549a6..692f8c7 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockStatsMXBean.java
@@ -33,13 +33,18 @@ import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.StorageType;
+import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.hdfs.DFSConfigKeys;
import org.apache.hadoop.hdfs.DFSTestUtil;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.HdfsConfiguration;
import org.apache.hadoop.hdfs.MiniDFSCluster;
import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
+import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.test.GenericTestUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Rule;
@@ -218,4 +223,51 @@ public class TestBlockStatsMXBean {
storageTypeStats = storageTypeStatsMap.get(StorageType.ARCHIVE);
assertEquals(3, storageTypeStats.getNodesInService());
}
+
+ @Test
+ public void testStorageTypeLoad() throws Exception {
+ HeartbeatManager heartbeatManager =
+ cluster.getNamesystem().getBlockManager().getDatanodeManager()
+ .getHeartbeatManager();
+ Map<StorageType, StorageTypeStats> storageTypeStatsMap =
+ heartbeatManager.getStorageTypeStats();
+ DistributedFileSystem dfs = cluster.getFileSystem();
+
+ // Create a file with HOT storage policy.
+ Path hotSpDir = new Path("/HOT");
+ dfs.mkdir(hotSpDir, FsPermission.getDirDefault());
+ dfs.setStoragePolicy(hotSpDir, "HOT");
+ FSDataOutputStream hotSpFileStream =
+ dfs.create(new Path(hotSpDir, "hotFile"));
+ hotSpFileStream.write("Storage Policy Hot".getBytes());
+ hotSpFileStream.hflush();
+
+ // Create a file with COLD storage policy.
+ Path coldSpDir = new Path("/COLD");
+ dfs.mkdir(coldSpDir, FsPermission.getDirDefault());
+ dfs.setStoragePolicy(coldSpDir, "COLD");
+ FSDataOutputStream coldSpFileStream =
+ dfs.create(new Path(coldSpDir, "coldFile"));
+ coldSpFileStream.write("Writing to ARCHIVE storage type".getBytes());
+ coldSpFileStream.hflush();
+
+ // Trigger heartbeats manually to speed up the test.
+ cluster.triggerHeartbeats();
+
+ // The load would be 2*replication since both the
+ // write xceiver & packet responder threads are counted.
+ GenericTestUtils.waitFor(() -> storageTypeStatsMap.get(StorageType.DISK)
+ .getNodesInServiceXceiverCount() == 6, 100, 5000);
+
+ // The count for ARCHIVE should be independent of the value of DISK.
+ GenericTestUtils.waitFor(() -> storageTypeStatsMap.get(StorageType.ARCHIVE)
+ .getNodesInServiceXceiverCount() == 6, 100, 5000);
+
+ // The total count should stay unaffected, that is sum of load from all
+ // datanodes.
+ GenericTestUtils
+ .waitFor(() -> heartbeatManager.getInServiceXceiverCount() == 12, 100,
+ 5000);
+ IOUtils.closeStreams(hotSpFileStream, coldSpFileStream);
+ }
}
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
index 78629fe..cf5f5a1 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
@@ -17,6 +17,7 @@
*/
package org.apache.hadoop.hdfs.server.blockmanagement;
+import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_KEY;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNotEquals;
@@ -1618,5 +1619,33 @@ public class TestReplicationPolicy extends BaseReplicationPolicyTest {
when(node.getXceiverCount()).thenReturn(10);
assertTrue(bppd.excludeNodeByLoad(node));
+ // Enable load check per storage type.
+ conf.setBoolean(DFS_NAMENODE_REDUNDANCY_CONSIDERLOADBYSTORAGETYPE_KEY,
+ true);
+ bppd.initialize(conf, statistics, null, null);
+ Map<StorageType, StorageTypeStats> storageStats = new HashMap<>();
+ StorageTypeStats diskStorageTypeStats =
+ new StorageTypeStats(StorageType.DISK);
+
+ // Set xceiver count as 500 for DISK.
+ diskStorageTypeStats.setDataNodesInServiceXceiverCount(50, 10);
+ storageStats.put(StorageType.DISK, diskStorageTypeStats);
+
+ //Set xceiver count as 900 for ARCHIVE
+ StorageTypeStats archiveStorageTypeStats =
+ new StorageTypeStats(StorageType.ARCHIVE);
+ archiveStorageTypeStats.setDataNodesInServiceXceiverCount(10, 90);
+ storageStats.put(StorageType.ARCHIVE, diskStorageTypeStats);
+
+ when(statistics.getStorageTypeStats()).thenReturn(storageStats);
+ when(node.getXceiverCount()).thenReturn(29);
+ when(node.getStorageTypes()).thenReturn(EnumSet.of(StorageType.DISK));
+ when(statistics.getInServiceXceiverAverage()).thenReturn(0.0);
+ //Added for sanity, the number of datanodes are 100, the average xceiver
+ // shall be (50*100+90*100)/100 = 14
+ when(statistics.getInServiceXceiverAverage()).thenReturn(14.0);
+ when(node.getXceiverCount()).thenReturn(100);
+
+ assertFalse(bppd.excludeNodeByLoad(node));
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org
[hadoop] 06/07: HDFS-15331. Remove invalid exclusions that
minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji
Posted by we...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/hadoop.git
commit 872839416b9eac6602d60cdd755af46730a9a8af
Author: Wanqiang Ji <wa...@gmail.com>
AuthorDate: Wed May 6 04:36:38 2020 +0800
HDFS-15331. Remove invalid exclusions that minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji
(cherry picked from commit e877118454b97ae72d3b52dd256999ca073df91d)
---
hadoop-minicluster/pom.xml | 6 ------
1 file changed, 6 deletions(-)
diff --git a/hadoop-minicluster/pom.xml b/hadoop-minicluster/pom.xml
index 7d64566..1ba646a 100644
--- a/hadoop-minicluster/pom.xml
+++ b/hadoop-minicluster/pom.xml
@@ -67,12 +67,6 @@
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<scope>compile</scope>
- <exclusions>
- <exclusion>
- <groupId>io.kubernetes</groupId>
- <artifactId>client-java</artifactId>
- </exclusion>
- </exclusions>
</dependency>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org