You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by ji...@apache.org on 2015/05/01 09:01:59 UTC
[42/50] [abbrv] hadoop git commit: HDFS-7770. Need document for
storage type label of data node storage locations under dfs.data.dir.
Contributed by Xiaoyu Yao.
HDFS-7770. Need document for storage type label of data node storage locations under dfs.data.dir. Contributed by Xiaoyu Yao.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/de9404f0
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/de9404f0
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/de9404f0
Branch: refs/heads/HDFS-7240
Commit: de9404f02f36bf9a1100c67f41db907d494bb9ed
Parents: f384a06
Author: Akira Ajisaka <aa...@apache.org>
Authored: Thu Apr 30 20:12:32 2015 +0900
Committer: Akira Ajisaka <aa...@apache.org>
Committed: Thu Apr 30 20:12:32 2015 +0900
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++
.../hadoop-hdfs/src/main/resources/hdfs-default.xml | 7 +++++--
.../hadoop-hdfs/src/site/markdown/ArchivalStorage.md | 8 ++++++++
3 files changed, 16 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/de9404f0/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index c538b78..e092dc8 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -603,6 +603,9 @@ Release 2.7.1 - UNRELEASED
HDFS-8204. Mover/Balancer should not schedule two replicas to the same
datanode. (Walter Su via szetszwo)
+ HDFS-7770. Need document for storage type label of data node storage
+ locations under dfs.data.dir. (Xiaoyu Yao via aajisaka)
+
OPTIMIZATIONS
BUG FIXES
http://git-wip-us.apache.org/repos/asf/hadoop/blob/de9404f0/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index 092d5aa..f03c0fb 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -430,8 +430,11 @@
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named
- directories, typically on different devices.
- Directories that do not exist are ignored.
+ directories, typically on different devices. The directories should be tagged
+ with corresponding storage types ([SSD]/[DISK]/[ARCHIVE]/[RAM_DISK]) for HDFS
+ storage policies. The default storage type will be DISK if the directory does
+ not have a storage type tagged explicitly. Directories that do not exist will
+ be created if local filesystem permission allows.
</description>
</property>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/de9404f0/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
index 2038401..d924f80 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ArchivalStorage.md
@@ -95,6 +95,14 @@ The effective storage policy can be retrieved by the "[`dfsadmin -getStoragePoli
### Configuration
* **dfs.storage.policy.enabled** - for enabling/disabling the storage policy feature. The default value is `true`.
+* **dfs.datanode.data.dir** - on each data node, the comma-separated storage locations should be tagged with their storage types. This allows storage policies to place the blocks on different storage types according to policy. For example:
+
+ 1. A datanode storage location /grid/dn/disk0 on DISK should be configured with `[DISK]file:///grid/dn/disk0`
+ 2. A datanode storage location /grid/dn/ssd0 on SSD can should configured with `[SSD]file:///grid/dn/ssd0`
+ 3. A datanode storage location /grid/dn/archive0 on ARCHIVE should be configured with `[ARCHIVE]file:///grid/dn/archive0`
+ 4. A datanode storage location /grid/dn/ram0 on RAM_DISK should be configured with `[RAM_DISK]file:///grid/dn/ram0`
+
+ The default storage type of a datanode storage location will be DISK if it does not have a storage type tagged explicitly.
Mover - A New Data Migration Tool
---------------------------------