You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-commits@hadoop.apache.org by cm...@apache.org on 2014/08/20 01:50:25 UTC
svn commit: r1619012 [26/35] - in
/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project: hadoop-hdfs-httpfs/
hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/client/
hadoop-hdfs-httpfs/src/main/java/org/apache/hadoop/fs/http/server/ hadoop...
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm Tue Aug 19 23:49:39 2014
@@ -77,7 +77,7 @@ HDFS Users Guide
* <<<fetchdt>>>: a utility to fetch DelegationToken and store it in a
file on the local system.
- * Rebalancer: tool to balance the cluster when the data is
+ * Balancer: tool to balance the cluster when the data is
unevenly distributed among DataNodes.
* Upgrade and rollback: after a software upgrade, it is possible
@@ -143,8 +143,8 @@ HDFS Users Guide
** DFSAdmin Command
- The <<<bin/hadoop dfsadmin>>> command supports a few HDFS administration
- related operations. The <<<bin/hadoop dfsadmin -help>>> command lists all the
+ The <<<bin/hdfs dfsadmin>>> command supports a few HDFS administration
+ related operations. The <<<bin/hdfs dfsadmin -help>>> command lists all the
commands currently supported. For e.g.:
* <<<-report>>>: reports basic statistics of HDFS. Some of this
@@ -172,7 +172,7 @@ HDFS Users Guide
of racks and datanodes attached to the tracks as viewed by the
NameNode.
- For command usage, see {{{../hadoop-common/CommandsManual.html#dfsadmin}dfsadmin}}.
+ For command usage, see {{{./HDFSCommands.html#dfsadmin}dfsadmin}}.
* Secondary NameNode
@@ -207,7 +207,7 @@ HDFS Users Guide
primary NameNode if necessary.
For command usage,
- see {{{../hadoop-common/CommandsManual.html#secondarynamenode}secondarynamenode}}.
+ see {{{./HDFSCommands.html#secondarynamenode}secondarynamenode}}.
* Checkpoint Node
@@ -249,7 +249,7 @@ HDFS Users Guide
Multiple checkpoint nodes may be specified in the cluster configuration
file.
- For command usage, see {{{../hadoop-common/CommandsManual.html#namenode}namenode}}.
+ For command usage, see {{{./HDFSCommands.html#namenode}namenode}}.
* Backup Node
@@ -291,7 +291,7 @@ HDFS Users Guide
For a complete discussion of the motivation behind the creation of the
Backup node and Checkpoint node, see {{{https://issues.apache.org/jira/browse/HADOOP-4539}HADOOP-4539}}.
- For command usage, see {{{../hadoop-common/CommandsManual.html#namenode}namenode}}.
+ For command usage, see {{{./HDFSCommands.html#namenode}namenode}}.
* Import Checkpoint
@@ -314,9 +314,9 @@ HDFS Users Guide
verifies that the image in <<<dfs.namenode.checkpoint.dir>>> is consistent,
but does not modify it in any way.
- For command usage, see {{{../hadoop-common/CommandsManual.html#namenode}namenode}}.
+ For command usage, see {{{./HDFSCommands.html#namenode}namenode}}.
-* Rebalancer
+* Balancer
HDFS data might not always be be placed uniformly across the DataNode.
One common reason is addition of new DataNodes to an existing cluster.
@@ -338,10 +338,10 @@ HDFS Users Guide
Due to multiple competing considerations, data might not be uniformly
placed across the DataNodes. HDFS provides a tool for administrators
that analyzes block placement and rebalanaces data across the DataNode.
- A brief administrator's guide for rebalancer as a PDF is attached to
+ A brief administrator's guide for balancer is available at
{{{https://issues.apache.org/jira/browse/HADOOP-1652}HADOOP-1652}}.
- For command usage, see {{{../hadoop-common/CommandsManual.html#balancer}balancer}}.
+ For command usage, see {{{./HDFSCommands.html#balancer}balancer}}.
* Rack Awareness
@@ -368,7 +368,7 @@ HDFS Users Guide
allow any modifications to file system or blocks. Normally the NameNode
leaves Safemode automatically after the DataNodes have reported that
most file system blocks are available. If required, HDFS could be
- placed in Safemode explicitly using <<<bin/hadoop dfsadmin -safemode>>>
+ placed in Safemode explicitly using <<<bin/hdfs dfsadmin -safemode>>>
command. NameNode front page shows whether Safemode is on or off. A
more detailed description and configuration is maintained as JavaDoc
for <<<setSafeMode()>>>.
@@ -383,8 +383,8 @@ HDFS Users Guide
most of the recoverable failures. By default fsck ignores open files
but provides an option to select all files during reporting. The HDFS
fsck command is not a Hadoop shell command. It can be run as
- <<<bin/hadoop fsck>>>. For command usage, see
- {{{../hadoop-common/CommandsManual.html#fsck}fsck}}. fsck can be run on
+ <<<bin/hdfs fsck>>>. For command usage, see
+ {{{./HDFSCommands.html#fsck}fsck}}. fsck can be run on
the whole file system or on a subset of files.
* fetchdt
@@ -395,11 +395,11 @@ HDFS Users Guide
Utility uses either RPC or HTTPS (over Kerberos) to get the token, and
thus requires kerberos tickets to be present before the run (run kinit
to get the tickets). The HDFS fetchdt command is not a Hadoop shell
- command. It can be run as <<<bin/hadoop fetchdt DTfile>>>. After you got
+ command. It can be run as <<<bin/hdfs fetchdt DTfile>>>. After you got
the token you can run an HDFS command without having Kerberos tickets,
by pointing <<<HADOOP_TOKEN_FILE_LOCATION>>> environmental variable to the
delegation token file. For command usage, see
- {{{../hadoop-common/CommandsManual.html#fetchdt}fetchdt}} command.
+ {{{./HDFSCommands.html#fetchdt}fetchdt}} command.
* Recovery Mode
@@ -533,5 +533,4 @@ HDFS Users Guide
* Explore {{{./hdfs-default.xml}hdfs-default.xml}}. It includes
brief description of most of the configuration variables available.
- * {{{../hadoop-common/CommandsManual.html}Hadoop Commands Guide}}:
- Hadoop commands usage.
+ * {{{./HDFSCommands.html}HDFS Commands Guide}}: HDFS commands usage.
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ViewFs.apt.vm Tue Aug 19 23:49:39 2014
@@ -18,8 +18,6 @@
ViewFs Guide
- \[ {{{./index.html}Go Back}} \]
-
%{toc|section=1|fromDepth=0}
* {Introduction}
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/WebHDFS.apt.vm
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/WebHDFS.apt.vm?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/WebHDFS.apt.vm (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/WebHDFS.apt.vm Tue Aug 19 23:49:39 2014
@@ -70,6 +70,21 @@ WebHDFS REST API
* {{{Get Delegation Tokens}<<<GETDELEGATIONTOKENS>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getDelegationTokens)
+ * {{{Get an XAttr}<<<GETXATTRS>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getXAttr)
+
+ * {{{Get multiple XAttrs}<<<GETXATTRS>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getXAttrs)
+
+ * {{{Get all XAttrs}<<<GETXATTRS>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getXAttrs)
+
+ * {{{List all XAttrs}<<<LISTXATTRS>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.listXAttrs)
+
+ * {{{Check access}<<<CHECKACCESS>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.access)
+
* HTTP PUT
* {{{Create and Write to a File}<<<CREATE>>>}}
@@ -102,6 +117,18 @@ WebHDFS REST API
* {{{Cancel Delegation Token}<<<CANCELDELEGATIONTOKEN>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.cancelDelegationToken)
+ * {{{Create Snapshot}<<<CREATESNAPSHOT>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.createSnapshot)
+
+ * {{{Rename Snapshot}<<<RENAMESNAPSHOT>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.renameSnapshot)
+
+ * {{{Set XAttr}<<<SETXATTR>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setXAttr)
+
+ * {{{Remove XAttr}<<<REMOVEXATTR>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.removeXAttr)
+
* HTTP POST
* {{{Append to a File}<<<APPEND>>>}}
@@ -114,6 +141,9 @@ WebHDFS REST API
* {{{Delete a File/Directory}<<<DELETE>>>}}
(see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.delete)
+
+ * {{{Delete Snapshot}<<<DELETESNAPSHOT>>>}}
+ (see {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.deleteSnapshot)
** {FileSystem URIs vs HTTP URLs}
@@ -146,7 +176,7 @@ WebHDFS REST API
*-------------------------------------------------+---------------------------------------------------+
| <<<dfs.webhdfs.enabled >>> | Enable/disable WebHDFS in Namenodes and Datanodes |
*-------------------------------------------------+---------------------------------------------------+
-| <<<dfs.web.authentication.kerberos.principal>>> | The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. The HTTP Kerberos principal MUST start with 'HTTP/' per Kerberos HTTP SPNEGO specification. |
+| <<<dfs.web.authentication.kerberos.principal>>> | The HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. The HTTP Kerberos principal MUST start with 'HTTP/' per Kerberos HTTP SPNEGO specification. A value of "*" will use all HTTP principals found in the keytab. |
*-------------------------------------------------+---------------------------------------------------+
| <<<dfs.web.authentication.kerberos.keytab >>> | The Kerberos keytab file with the credentials for the HTTP Kerberos principal used by Hadoop-Auth in the HTTP endpoint. |
*-------------------------------------------------+---------------------------------------------------+
@@ -367,7 +397,7 @@ Hello, webhdfs user!
* Submit a HTTP PUT request.
+---------------------------------
-curl -i -X PUT "http://<HOST>:<PORT>/<PATH>?op=MKDIRS[&permission=<OCTAL>]"
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=MKDIRS[&permission=<OCTAL>]"
+---------------------------------
The client receives a response with a {{{Boolean JSON Schema}<<<boolean>>> JSON object}}:
@@ -392,7 +422,7 @@ Transfer-Encoding: chunked
* Submit a HTTP PUT request.
+---------------------------------
-curl -i -X PUT "http://<HOST>:<PORT>/<PATH>?op=CREATESYMLINK
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATESYMLINK
&destination=<PATH>[&createParent=<true|false>]"
+---------------------------------
@@ -900,6 +930,279 @@ Transfer-Encoding: chunked
{{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getAclStatus
+** {Check access}
+
+ * Submit a HTTP GET request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CHECKACCESS
+ &fsaction=<FSACTION>
++---------------------------------
+
+ The client receives a response with zero content length:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Length: 0
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.access
+
+
+* {Extended Attributes(XAttrs) Operations}
+
+** {Set XAttr}
+
+ * Submit a HTTP PUT request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=SETXATTR
+ &xattr.name=<XATTRNAME>&xattr.value=<XATTRVALUE>
+ &flag=<FLAG>"
++---------------------------------
+
+ The client receives a response with zero content length:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Length: 0
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.setXAttr
+
+
+** {Remove XAttr}
+
+ * Submit a HTTP PUT request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=REMOVEXATTR
+ &xattr.name=<XATTRNAME>"
++---------------------------------
+
+ The client receives a response with zero content length:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Length: 0
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.removeXAttr
+
+
+** {Get an XAttr}
+
+ * Submit a HTTP GET request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETXATTRS
+ &xattr.name=<XATTRNAME>&encoding=<ENCODING>"
++---------------------------------
+
+ The client receives a response with a {{{XAttrs JSON Schema}<<<XAttrs>>> JSON object}}:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Type: application/json
+Transfer-Encoding: chunked
+
+{
+ "XAttrs": [
+ {
+ "name":"XATTRNAME",
+ "value":"XATTRVALUE"
+ }
+ ]
+}
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getXAttr
+
+
+** {Get multiple XAttrs}
+
+ * Submit a HTTP GET request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETXATTRS
+ &xattr.name=<XATTRNAME1>&xattr.name=<XATTRNAME2>
+ &encoding=<ENCODING>"
++---------------------------------
+
+ The client receives a response with a {{{XAttrs JSON Schema}<<<XAttrs>>> JSON object}}:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Type: application/json
+Transfer-Encoding: chunked
+
+{
+ "XAttrs": [
+ {
+ "name":"XATTRNAME1",
+ "value":"XATTRVALUE1"
+ },
+ {
+ "name":"XATTRNAME2",
+ "value":"XATTRVALUE2"
+ }
+ ]
+}
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getXAttrs
+
+
+** {Get all XAttrs}
+
+ * Submit a HTTP GET request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=GETXATTRS
+ &encoding=<ENCODING>"
++---------------------------------
+
+ The client receives a response with a {{{XAttrs JSON Schema}<<<XAttrs>>> JSON object}}:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Type: application/json
+Transfer-Encoding: chunked
+
+{
+ "XAttrs": [
+ {
+ "name":"XATTRNAME1",
+ "value":"XATTRVALUE1"
+ },
+ {
+ "name":"XATTRNAME2",
+ "value":"XATTRVALUE2"
+ },
+ {
+ "name":"XATTRNAME3",
+ "value":"XATTRVALUE3"
+ }
+ ]
+}
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.getXAttrs
+
+
+** {List all XAttrs}
+
+ * Submit a HTTP GET request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=LISTXATTRS"
++---------------------------------
+
+ The client receives a response with a {{{XAttrNames JSON Schema}<<<XAttrNames>>> JSON object}}:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Type: application/json
+Transfer-Encoding: chunked
+
+{
+ "XAttrNames":"[\"XATTRNAME1\",\"XATTRNAME2\",\"XATTRNAME3\"]"
+}
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.listXAttrs
+
+
+* {Snapshot Operations}
+
+** {Create Snapshot}
+
+ * Submit a HTTP PUT request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATESNAPSHOT[&snapshotname=<SNAPSHOTNAME>]"
++---------------------------------
+
+ The client receives a response with a {{{Path JSON Schema}<<<Path>>> JSON object}}:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Type: application/json
+Transfer-Encoding: chunked
+
+{"Path": "/user/szetszwo/.snapshot/s1"}
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.createSnapshot
+
+
+** {Delete Snapshot}
+
+ * Submit a HTTP DELETE request.
+
++---------------------------------
+curl -i -X DELETE "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=DELETESNAPSHOT&snapshotname=<SNAPSHOTNAME>"
++---------------------------------
+
+ The client receives a response with zero content length:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Length: 0
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.deleteSnapshot
+
+
+** {Rename Snapshot}
+
+ * Submit a HTTP PUT request.
+
++---------------------------------
+curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=RENAMESNAPSHOT
+ &oldsnapshotname=<SNAPSHOTNAME>&snapshotname=<SNAPSHOTNAME>"
++---------------------------------
+
+ The client receives a response with zero content length:
+
++---------------------------------
+HTTP/1.1 200 OK
+Content-Length: 0
++---------------------------------
+
+ []
+
+ See also:
+ {{{../../api/org/apache/hadoop/fs/FileSystem.html}FileSystem}}.renameSnapshot
+
+
* {Delegation Token Operations}
** {Get Delegation Token}
@@ -1174,6 +1477,58 @@ Transfer-Encoding: chunked
+---------------------------------
+** {XAttrs JSON Schema}
+
++---------------------------------
+{
+ "name" : "XAttrs",
+ "properties":
+ {
+ "XAttrs":
+ {
+ "type" : "array",
+ "items":
+ {
+ "type" " "object",
+ "properties":
+ {
+ "name":
+ {
+ "description": "XAttr name.",
+ "type" : "string",
+ "required" : true
+ },
+ "value":
+ {
+ "description": "XAttr value.",
+ "type" : "string"
+ }
+ }
+ }
+ }
+ }
+}
++---------------------------------
+
+
+** {XAttrNames JSON Schema}
+
++---------------------------------
+{
+ "name" : "XAttrNames",
+ "properties":
+ {
+ "XAttrNames":
+ {
+ "description": "XAttr names.",
+ "type" : "string"
+ "required" : true
+ }
+ }
+}
++---------------------------------
+
+
** {Boolean JSON Schema}
+---------------------------------
@@ -1610,6 +1965,83 @@ var tokenProperties =
*----------------+-------------------------------------------------------------------+
+** {XAttr Name}
+
+*----------------+-------------------------------------------------------------------+
+|| Name | <<<xattr.name>>> |
+*----------------+-------------------------------------------------------------------+
+|| Description | The XAttr name of a file/directory. |
+*----------------+-------------------------------------------------------------------+
+|| Type | String |
+*----------------+-------------------------------------------------------------------+
+|| Default Value | \<empty\> |
+*----------------+-------------------------------------------------------------------+
+|| Valid Values | Any string prefixed with user./trusted./system./security.. |
+*----------------+-------------------------------------------------------------------+
+|| Syntax | Any string prefixed with user./trusted./system./security.. |
+*----------------+-------------------------------------------------------------------+
+
+
+** {XAttr Value}
+
+*----------------+-------------------------------------------------------------------+
+|| Name | <<<xattr.value>>> |
+*----------------+-------------------------------------------------------------------+
+|| Description | The XAttr value of a file/directory. |
+*----------------+-------------------------------------------------------------------+
+|| Type | String |
+*----------------+-------------------------------------------------------------------+
+|| Default Value | \<empty\> |
+*----------------+-------------------------------------------------------------------+
+|| Valid Values | An encoded value. |
+*----------------+-------------------------------------------------------------------+
+|| Syntax | Enclosed in double quotes or prefixed with 0x or 0s. |
+*----------------+-------------------------------------------------------------------+
+
+ See also:
+ {{{./ExtendedAttributes.html}Extended Attributes}}
+
+
+** {XAttr set flag}
+
+*----------------+-------------------------------------------------------------------+
+|| Name | <<<flag>>> |
+*----------------+-------------------------------------------------------------------+
+|| Description | The XAttr set flag. |
+*----------------+-------------------------------------------------------------------+
+|| Type | String |
+*----------------+-------------------------------------------------------------------+
+|| Default Value | \<empty\> |
+*----------------+-------------------------------------------------------------------+
+|| Valid Values | CREATE,REPLACE. |
+*----------------+-------------------------------------------------------------------+
+|| Syntax | CREATE,REPLACE. |
+*----------------+-------------------------------------------------------------------+
+
+ See also:
+ {{{./ExtendedAttributes.html}Extended Attributes}}
+
+
+** {XAttr value encoding}
+
+*----------------+-------------------------------------------------------------------+
+|| Name | <<<encoding>>> |
+*----------------+-------------------------------------------------------------------+
+|| Description | The XAttr value encoding. |
+*----------------+-------------------------------------------------------------------+
+|| Type | String |
+*----------------+-------------------------------------------------------------------+
+|| Default Value | \<empty\> |
+*----------------+-------------------------------------------------------------------+
+|| Valid Values | text \| hex \| base64 |
+*----------------+-------------------------------------------------------------------+
+|| Syntax | text \| hex \| base64 |
+*----------------+-------------------------------------------------------------------+
+
+ See also:
+ {{{./ExtendedAttributes.html}Extended Attributes}}
+
+
** {Access Time}
*----------------+-------------------------------------------------------------------+
@@ -1759,6 +2191,25 @@ var tokenProperties =
{{Proxy Users}}
+** {Fs Action}
+
+*----------------+-------------------------------------------------------------------+
+|| Name | <<<fsaction>>> |
+*----------------+-------------------------------------------------------------------+
+|| Description | File system operation read/write/execute |
+*----------------+-------------------------------------------------------------------+
+|| Type | String |
+*----------------+-------------------------------------------------------------------+
+|| Default Value | null (an invalid value) |
+*----------------+-------------------------------------------------------------------+
+|| Valid Values | Strings matching regex pattern \"[rwx-]\{3\}\" |
+*----------------+-------------------------------------------------------------------+
+|| Syntax | \"[rwx-]\{3\}\" |
+*----------------+-------------------------------------------------------------------+
+
+ See also:
+ {{{Check access}<<<CHECKACCESS>>>}},
+
** {Group}
*----------------+-------------------------------------------------------------------+
@@ -1839,6 +2290,26 @@ var tokenProperties =
{{{Open and Read a File}<<<OPEN>>>}}
+** {Old Snapshot Name}
+
+*----------------+-------------------------------------------------------------------+
+|| Name | <<<oldsnapshotname>>> |
+*----------------+-------------------------------------------------------------------+
+|| Description | The old name of the snapshot to be renamed. |
+*----------------+-------------------------------------------------------------------+
+|| Type | String |
+*----------------+-------------------------------------------------------------------+
+|| Default Value | null |
+*----------------+-------------------------------------------------------------------+
+|| Valid Values | An existing snapshot name. |
+*----------------+-------------------------------------------------------------------+
+|| Syntax | Any string. |
+*----------------+-------------------------------------------------------------------+
+
+ See also:
+ {{{Rename Snapshot}<<<RENAMESNAPSHOT>>>}}
+
+
** {Op}
*----------------+-------------------------------------------------------------------+
@@ -1983,6 +2454,29 @@ var tokenProperties =
{{{Set Replication Factor}<<<SETREPLICATION>>>}}
+** {Snapshot Name}
+
+*----------------+-------------------------------------------------------------------+
+|| Name | <<<snapshotname>>> |
+*----------------+-------------------------------------------------------------------+
+|| Description | The name of the snapshot to be created/deleted. |
+|| | Or the new name for snapshot rename. |
+*----------------+-------------------------------------------------------------------+
+|| Type | String |
+*----------------+-------------------------------------------------------------------+
+|| Default Value | null |
+*----------------+-------------------------------------------------------------------+
+|| Valid Values | Any valid snapshot name. |
+*----------------+-------------------------------------------------------------------+
+|| Syntax | Any string. |
+*----------------+-------------------------------------------------------------------+
+
+ See also:
+ {{{Create Snapshot}<<<CREATESNAPSHOT>>>}},
+ {{{Delete Snapshot}<<<DELETESNAPSHOT>>>}},
+ {{{Rename Snapshot}<<<RENAMESNAPSHOT>>>}}
+
+
** {Sources}
*----------------+-------------------------------------------------------------------+
@@ -2042,4 +2536,3 @@ var tokenProperties =
See also:
{{Authentication}}
-
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml Tue Aug 19 23:49:39 2014
@@ -217,7 +217,7 @@
<subsection name="DFSAdmin Commands" id="dfsadminCommands">
<h4><code>dfsadmin -rollingUpgrade</code></h4>
- <source>hdfs dfsadmin -rollingUpgrade <query|start|finalize></source>
+ <source>hdfs dfsadmin -rollingUpgrade <query|prepare|finalize></source>
<p>
Execute a rolling upgrade action.
<ul><li>Options:<table>
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsSnapshots.xml Tue Aug 19 23:49:39 2014
@@ -97,7 +97,9 @@
<li>Listing the files in snapshot <code>s0</code>:
<source>hdfs dfs -ls /foo/.snapshot/s0</source></li>
<li>Copying a file from snapshot <code>s0</code>:
- <source>hdfs dfs -cp /foo/.snapshot/s0/bar /tmp</source></li>
+ <source>hdfs dfs -cp -ptopax /foo/.snapshot/s0/bar /tmp</source>
+ <p>Note that this example uses the preserve option to preserve
+ timestamps, ownership, permission, ACLs and XAttrs.</p></li>
</ul>
</subsection>
</section>
@@ -255,7 +257,35 @@
<tr><td>fromSnapshot</td><td>The name of the starting snapshot.</td></tr>
<tr><td>toSnapshot</td><td>The name of the ending snapshot.</td></tr>
</table></li>
- </ul>
+ <li>Results:
+ <table>
+ <tr><td>+</td><td>The file/directory has been created.</td></tr>
+ <tr><td>-</td><td>The file/directory has been deleted.</td></tr>
+ <tr><td>M</td><td>The file/directory has been modified.</td></tr>
+ <tr><td>R</td><td>The file/directory has been renamed.</td></tr>
+ </table>
+ </li>
+ </ul>
+ <p>
+ A <em>RENAME</em> entry indicates a file/directory has been renamed but
+ is still under the same snapshottable directory. A file/directory is
+ reported as deleted if it was renamed to outside of the snapshottble directory.
+ A file/directory renamed from outside of the snapshottble directory is
+ reported as newly created.
+ </p>
+ <p>
+ The snapshot difference report does not guarantee the same operation sequence.
+ For example, if we rename the directory <em>"/foo"</em> to <em>"/foo2"</em>, and
+ then append new data to the file <em>"/foo2/bar"</em>, the difference report will
+ be:
+ <source>
+ R. /foo -> /foo2
+ M. /foo/bar
+ </source>
+ I.e., the changes on the files/directories under a renamed directory is
+ reported using the original path before the rename (<em>"/foo/bar"</em> in
+ the above example).
+ </p>
<p>
See also the corresponding Java API
<code>SnapshotDiffReport getSnapshotDiffReport(Path path, String fromSnapshot, String toSnapshot)</code>
Propchange: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs/
------------------------------------------------------------------------------
Merged /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs:r1582150-1619000
Merged /hadoop/common/branches/HDFS-2006/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs:r1588992-1596568
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BenchmarkThroughput.java Tue Aug 19 23:49:39 2014
@@ -229,7 +229,7 @@ public class BenchmarkThroughput extends
}
/**
- * @param args
+ * @param args arguments
*/
public static void main(String[] args) throws Exception {
int res = ToolRunner.run(new HdfsConfiguration(),
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/BlockReaderTestUtil.java Tue Aug 19 23:49:39 2014
@@ -33,9 +33,11 @@ import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hdfs.net.Peer;
import org.apache.hadoop.hdfs.net.TcpPeerServer;
+import org.apache.hadoop.hdfs.protocol.DatanodeID;
import org.apache.hadoop.hdfs.protocol.DatanodeInfo;
import org.apache.hadoop.hdfs.protocol.ExtendedBlock;
import org.apache.hadoop.hdfs.protocol.LocatedBlock;
+import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier;
import org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor;
import org.apache.hadoop.hdfs.server.common.HdfsServerConstants;
import org.apache.hadoop.hdfs.server.datanode.CachingStrategy;
@@ -48,6 +50,7 @@ import org.apache.hadoop.hdfs.shortcircu
import org.apache.hadoop.hdfs.shortcircuit.ShortCircuitReplica;
import org.apache.hadoop.hdfs.shortcircuit.ShortCircuitShm;
import org.apache.hadoop.net.NetUtils;
+import org.apache.hadoop.security.token.Token;
import org.apache.log4j.Level;
import org.apache.log4j.LogManager;
@@ -192,7 +195,8 @@ public class BlockReaderTestUtil {
setAllowShortCircuitLocalReads(true).
setRemotePeerFactory(new RemotePeerFactory() {
@Override
- public Peer newConnectedPeer(InetSocketAddress addr)
+ public Peer newConnectedPeer(InetSocketAddress addr,
+ Token<BlockTokenIdentifier> blockToken, DatanodeID datanodeId)
throws IOException {
Peer peer = null;
Socket sock = NetUtils.
@@ -251,4 +255,4 @@ public class BlockReaderTestUtil {
LogManager.getLogger(DataNode.class.getName()).setLevel(
Level.TRACE);
}
-}
\ No newline at end of file
+}
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSClientAdapter.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSClientAdapter.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSClientAdapter.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSClientAdapter.java Tue Aug 19 23:49:39 2014
@@ -54,8 +54,8 @@ public class DFSClientAdapter {
return dfs.dfs;
}
- public static ExtendedBlock getPreviousBlock(DFSClient client, String file) {
- return client.getPreviousBlock(file);
+ public static ExtendedBlock getPreviousBlock(DFSClient client, long fileId) {
+ return client.getPreviousBlock(fileId);
}
public static long getFileId(DFSOutputStream out) {
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java Tue Aug 19 23:49:39 2014
@@ -44,6 +44,9 @@ import org.apache.hadoop.hdfs.protocol.d
import org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.BlockOpResponseProto;
import org.apache.hadoop.hdfs.security.token.block.BlockTokenIdentifier;
import org.apache.hadoop.hdfs.security.token.block.ExportedBlockKeys;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoUnderConstruction;
+import org.apache.hadoop.hdfs.server.blockmanagement.BlockManager;
import org.apache.hadoop.hdfs.server.blockmanagement.BlockManagerTestUtil;
import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeDescriptor;
import org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager;
@@ -82,6 +85,7 @@ import java.util.concurrent.TimeoutExcep
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_RPC_ADDRESS_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_SERVICE_RPC_ADDRESS_KEY;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
/** Utilities for HDFS tests */
public class DFSTestUtil {
@@ -138,8 +142,8 @@ public class DFSTestUtil {
String clusterId = StartupOption.FORMAT.getClusterId();
if(clusterId == null || clusterId.isEmpty())
StartupOption.FORMAT.setClusterId("testClusterID");
-
- NameNode.format(conf);
+ // Use a copy of conf as it can be altered by namenode during format.
+ NameNode.format(new Configuration(conf));
}
/**
@@ -147,16 +151,40 @@ public class DFSTestUtil {
*/
public static Configuration newHAConfiguration(final String logicalName) {
Configuration conf = new Configuration();
- conf.set(DFSConfigKeys.DFS_NAMESERVICES, logicalName);
+ addHAConfiguration(conf, logicalName);
+ return conf;
+ }
+
+ /**
+ * Add a new HA configuration.
+ */
+ public static void addHAConfiguration(Configuration conf,
+ final String logicalName) {
+ String nsIds = conf.get(DFSConfigKeys.DFS_NAMESERVICES);
+ if (nsIds == null) {
+ conf.set(DFSConfigKeys.DFS_NAMESERVICES, logicalName);
+ } else { // append the nsid
+ conf.set(DFSConfigKeys.DFS_NAMESERVICES, nsIds + "," + logicalName);
+ }
conf.set(DFSUtil.addKeySuffixes(DFSConfigKeys.DFS_HA_NAMENODES_KEY_PREFIX,
logicalName), "nn1,nn2");
conf.set(DFSConfigKeys.DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX + "" +
"." + logicalName,
ConfiguredFailoverProxyProvider.class.getName());
conf.setInt(DFSConfigKeys.DFS_REPLICATION_KEY, 1);
- return conf;
}
+ public static void setFakeHttpAddresses(Configuration conf,
+ final String logicalName) {
+ conf.set(DFSUtil.addKeySuffixes(
+ DFSConfigKeys.DFS_NAMENODE_HTTP_ADDRESS_KEY,
+ logicalName, "nn1"), "127.0.0.1:12345");
+ conf.set(DFSUtil.addKeySuffixes(
+ DFSConfigKeys.DFS_NAMENODE_HTTP_ADDRESS_KEY,
+ logicalName, "nn2"), "127.0.0.1:12346");
+ }
+
+
/** class MyFile contains enough information to recreate the contents of
* a single file.
*/
@@ -355,7 +383,7 @@ public class DFSTestUtil {
*/
public static void waitForReplication(MiniDFSCluster cluster, ExtendedBlock b,
int racks, int replicas, int neededReplicas)
- throws IOException, TimeoutException, InterruptedException {
+ throws TimeoutException, InterruptedException {
int curRacks = 0;
int curReplicas = 0;
int curNeededReplicas = 0;
@@ -389,7 +417,7 @@ public class DFSTestUtil {
*/
public static void waitCorruptReplicas(FileSystem fs, FSNamesystem ns,
Path file, ExtendedBlock b, int corruptRepls)
- throws IOException, TimeoutException, InterruptedException {
+ throws TimeoutException, InterruptedException {
int count = 0;
final int ATTEMPTS = 50;
int repls = ns.getBlockManager().numCorruptReplicas(b.getLocalBlock());
@@ -814,7 +842,8 @@ public class DFSTestUtil {
// send the request
new Sender(out).transferBlock(b, new Token<BlockTokenIdentifier>(),
- dfsClient.clientName, new DatanodeInfo[]{datanodes[1]});
+ dfsClient.clientName, new DatanodeInfo[]{datanodes[1]},
+ new StorageType[]{StorageType.DEFAULT});
out.flush();
return BlockOpResponseProto.parseDelimitedFrom(in);
@@ -896,29 +925,47 @@ public class DFSTestUtil {
return getDatanodeDescriptor(ipAddr, DFSConfigKeys.DFS_DATANODE_DEFAULT_PORT,
rackLocation);
}
+
+ public static DatanodeDescriptor getDatanodeDescriptor(String ipAddr,
+ String rackLocation, String hostname) {
+ return getDatanodeDescriptor(ipAddr,
+ DFSConfigKeys.DFS_DATANODE_DEFAULT_PORT, rackLocation, hostname);
+ }
public static DatanodeStorageInfo createDatanodeStorageInfo(
String storageID, String ip) {
- return createDatanodeStorageInfo(storageID, ip, "defaultRack");
+ return createDatanodeStorageInfo(storageID, ip, "defaultRack", "host");
}
+
public static DatanodeStorageInfo[] createDatanodeStorageInfos(String[] racks) {
- return createDatanodeStorageInfos(racks.length, racks);
+ return createDatanodeStorageInfos(racks, null);
+ }
+
+ public static DatanodeStorageInfo[] createDatanodeStorageInfos(String[] racks, String[] hostnames) {
+ return createDatanodeStorageInfos(racks.length, racks, hostnames);
}
- public static DatanodeStorageInfo[] createDatanodeStorageInfos(int n, String... racks) {
+
+ public static DatanodeStorageInfo[] createDatanodeStorageInfos(int n) {
+ return createDatanodeStorageInfos(n, null, null);
+ }
+
+ public static DatanodeStorageInfo[] createDatanodeStorageInfos(
+ int n, String[] racks, String[] hostnames) {
DatanodeStorageInfo[] storages = new DatanodeStorageInfo[n];
for(int i = storages.length; i > 0; ) {
final String storageID = "s" + i;
final String ip = i + "." + i + "." + i + "." + i;
i--;
- final String rack = i < racks.length? racks[i]: "defaultRack";
- storages[i] = createDatanodeStorageInfo(storageID, ip, rack);
+ final String rack = (racks!=null && i < racks.length)? racks[i]: "defaultRack";
+ final String hostname = (hostnames!=null && i < hostnames.length)? hostnames[i]: "host";
+ storages[i] = createDatanodeStorageInfo(storageID, ip, rack, hostname);
}
return storages;
}
public static DatanodeStorageInfo createDatanodeStorageInfo(
- String storageID, String ip, String rack) {
+ String storageID, String ip, String rack, String hostname) {
final DatanodeStorage storage = new DatanodeStorage(storageID);
- final DatanodeDescriptor dn = BlockManagerTestUtil.getDatanodeDescriptor(ip, rack, storage);
+ final DatanodeDescriptor dn = BlockManagerTestUtil.getDatanodeDescriptor(ip, rack, storage, hostname);
return BlockManagerTestUtil.newDatanodeStorageInfo(dn, storage);
}
public static DatanodeDescriptor[] toDatanodeDescriptor(
@@ -931,8 +978,8 @@ public class DFSTestUtil {
}
public static DatanodeDescriptor getDatanodeDescriptor(String ipAddr,
- int port, String rackLocation) {
- DatanodeID dnId = new DatanodeID(ipAddr, "host",
+ int port, String rackLocation, String hostname) {
+ DatanodeID dnId = new DatanodeID(ipAddr, hostname,
UUID.randomUUID().toString(), port,
DFSConfigKeys.DFS_DATANODE_HTTP_DEFAULT_PORT,
DFSConfigKeys.DFS_DATANODE_HTTPS_DEFAULT_PORT,
@@ -940,6 +987,11 @@ public class DFSTestUtil {
return new DatanodeDescriptor(dnId, rackLocation);
}
+ public static DatanodeDescriptor getDatanodeDescriptor(String ipAddr,
+ int port, String rackLocation) {
+ return getDatanodeDescriptor(ipAddr, port, rackLocation, "host");
+ }
+
public static DatanodeRegistration getLocalDatanodeRegistration() {
return new DatanodeRegistration(getLocalDatanodeID(), new StorageInfo(
NodeType.DATA_NODE), new ExportedBlockKeys(), VersionInfo.getVersion());
@@ -1133,6 +1185,13 @@ public class DFSTestUtil {
.setType(AclEntryType.OTHER)
.build());
filesystem.setAcl(pathConcatTarget, aclEntryList);
+ // OP_SET_XATTR
+ filesystem.setXAttr(pathConcatTarget, "user.a1",
+ new byte[]{0x31, 0x32, 0x33});
+ filesystem.setXAttr(pathConcatTarget, "user.a2",
+ new byte[]{0x37, 0x38, 0x39});
+ // OP_REMOVE_XATTR
+ filesystem.removeXAttr(pathConcatTarget, "user.a2");
}
public static void abortStream(DFSOutputStream out) throws IOException {
@@ -1192,7 +1251,20 @@ public class DFSTestUtil {
long c = (val + factor - 1) / factor;
return c * factor;
}
-
+
+ public static void checkComponentsEquals(byte[][] expected, byte[][] actual) {
+ assertEquals("expected: " + DFSUtil.byteArray2PathString(expected)
+ + ", actual: " + DFSUtil.byteArray2PathString(actual), expected.length,
+ actual.length);
+ int i = 0;
+ for (byte[] e : expected) {
+ byte[] actualComponent = actual[i++];
+ assertTrue("expected: " + DFSUtil.bytes2String(e) + ", actual: "
+ + DFSUtil.bytes2String(actualComponent),
+ Arrays.equals(e, actualComponent));
+ }
+ }
+
/**
* A short-circuit test context which makes it easier to get a short-circuit
* configuration and set everything up.
@@ -1231,4 +1303,33 @@ public class DFSTestUtil {
sockDir.close();
}
}
+
+ /**
+ * @return the node which is expected to run the recovery of the
+ * given block, which is known to be under construction inside the
+ * given NameNOde.
+ */
+ public static DatanodeDescriptor getExpectedPrimaryNode(NameNode nn,
+ ExtendedBlock blk) {
+ BlockManager bm0 = nn.getNamesystem().getBlockManager();
+ BlockInfo storedBlock = bm0.getStoredBlock(blk.getLocalBlock());
+ assertTrue("Block " + blk + " should be under construction, " +
+ "got: " + storedBlock,
+ storedBlock instanceof BlockInfoUnderConstruction);
+ BlockInfoUnderConstruction ucBlock =
+ (BlockInfoUnderConstruction)storedBlock;
+ // We expect that the replica with the most recent heart beat will be
+ // the one to be in charge of the synchronization / recovery protocol.
+ final DatanodeStorageInfo[] storages = ucBlock.getExpectedStorageLocations();
+ DatanodeStorageInfo expectedPrimary = storages[0];
+ long mostRecentLastUpdate = expectedPrimary.getDatanodeDescriptor().getLastUpdate();
+ for (int i = 1; i < storages.length; i++) {
+ final long lastUpdate = storages[i].getDatanodeDescriptor().getLastUpdate();
+ if (lastUpdate > mostRecentLastUpdate) {
+ expectedPrimary = storages[i];
+ mostRecentLastUpdate = lastUpdate;
+ }
+ }
+ return expectedPrimary.getDatanodeDescriptor();
+ }
}
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java Tue Aug 19 23:49:39 2014
@@ -19,12 +19,15 @@ package org.apache.hadoop.hdfs;
import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY;
import static org.apache.hadoop.fs.CommonConfigurationKeysPublic.NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY;
+import static org.apache.hadoop.fs.CommonConfigurationKeys.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_DEFAULT;
+import static org.apache.hadoop.fs.CommonConfigurationKeys.IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCKREPORT_INITIAL_DELAY_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_ADDRESS_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HOST_NAME_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_ADDRESS_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_IPC_ADDRESS_KEY;
+import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATA_TRANSFER_PROTECTION_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_HA_LOGROLL_PERIOD_KEY;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_HA_NAMENODES_KEY_PREFIX;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_HA_NAMENODE_ID_KEY;
@@ -83,10 +86,12 @@ import org.apache.hadoop.hdfs.server.com
import org.apache.hadoop.hdfs.server.datanode.DataNode;
import org.apache.hadoop.hdfs.server.datanode.DataNodeTestUtils;
import org.apache.hadoop.hdfs.server.datanode.DataStorage;
+import org.apache.hadoop.hdfs.server.datanode.DatanodeUtil;
import org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter;
import org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.SecureResources;
import org.apache.hadoop.hdfs.server.datanode.SimulatedFSDataset;
import org.apache.hadoop.hdfs.server.datanode.fsdataset.FsDatasetSpi;
+import org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetUtil;
import org.apache.hadoop.hdfs.server.namenode.FSNamesystem;
import org.apache.hadoop.hdfs.server.namenode.NameNode;
import org.apache.hadoop.hdfs.server.namenode.NameNodeAdapter;
@@ -658,73 +663,81 @@ public class MiniDFSCluster {
boolean checkDataNodeHostConfig,
Configuration[] dnConfOverlays)
throws IOException {
- ExitUtil.disableSystemExit();
+ boolean success = false;
+ try {
+ ExitUtil.disableSystemExit();
- synchronized (MiniDFSCluster.class) {
- instanceId = instanceCount++;
- }
+ synchronized (MiniDFSCluster.class) {
+ instanceId = instanceCount++;
+ }
- this.conf = conf;
- base_dir = new File(determineDfsBaseDir());
- data_dir = new File(base_dir, "data");
- this.waitSafeMode = waitSafeMode;
- this.checkExitOnShutdown = checkExitOnShutdown;
-
- int replication = conf.getInt(DFS_REPLICATION_KEY, 3);
- conf.setInt(DFS_REPLICATION_KEY, Math.min(replication, numDataNodes));
- int safemodeExtension = conf.getInt(
- DFS_NAMENODE_SAFEMODE_EXTENSION_TESTING_KEY, 0);
- conf.setInt(DFS_NAMENODE_SAFEMODE_EXTENSION_KEY, safemodeExtension);
- conf.setInt(DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, 3); // 3 second
- conf.setClass(NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY,
- StaticMapping.class, DNSToSwitchMapping.class);
-
- // In an HA cluster, in order for the StandbyNode to perform checkpoints,
- // it needs to know the HTTP port of the Active. So, if ephemeral ports
- // are chosen, disable checkpoints for the test.
- if (!nnTopology.allHttpPortsSpecified() &&
- nnTopology.isHA()) {
- LOG.info("MiniDFSCluster disabling checkpointing in the Standby node " +
- "since no HTTP ports have been specified.");
- conf.setBoolean(DFS_HA_STANDBY_CHECKPOINTS_KEY, false);
- }
- if (!nnTopology.allIpcPortsSpecified() &&
- nnTopology.isHA()) {
- LOG.info("MiniDFSCluster disabling log-roll triggering in the "
- + "Standby node since no IPC ports have been specified.");
- conf.setInt(DFS_HA_LOGROLL_PERIOD_KEY, -1);
- }
+ this.conf = conf;
+ base_dir = new File(determineDfsBaseDir());
+ data_dir = new File(base_dir, "data");
+ this.waitSafeMode = waitSafeMode;
+ this.checkExitOnShutdown = checkExitOnShutdown;
+
+ int replication = conf.getInt(DFS_REPLICATION_KEY, 3);
+ conf.setInt(DFS_REPLICATION_KEY, Math.min(replication, numDataNodes));
+ int safemodeExtension = conf.getInt(
+ DFS_NAMENODE_SAFEMODE_EXTENSION_TESTING_KEY, 0);
+ conf.setInt(DFS_NAMENODE_SAFEMODE_EXTENSION_KEY, safemodeExtension);
+ conf.setInt(DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, 3); // 3 second
+ conf.setClass(NET_TOPOLOGY_NODE_SWITCH_MAPPING_IMPL_KEY,
+ StaticMapping.class, DNSToSwitchMapping.class);
+
+ // In an HA cluster, in order for the StandbyNode to perform checkpoints,
+ // it needs to know the HTTP port of the Active. So, if ephemeral ports
+ // are chosen, disable checkpoints for the test.
+ if (!nnTopology.allHttpPortsSpecified() &&
+ nnTopology.isHA()) {
+ LOG.info("MiniDFSCluster disabling checkpointing in the Standby node " +
+ "since no HTTP ports have been specified.");
+ conf.setBoolean(DFS_HA_STANDBY_CHECKPOINTS_KEY, false);
+ }
+ if (!nnTopology.allIpcPortsSpecified() &&
+ nnTopology.isHA()) {
+ LOG.info("MiniDFSCluster disabling log-roll triggering in the "
+ + "Standby node since no IPC ports have been specified.");
+ conf.setInt(DFS_HA_LOGROLL_PERIOD_KEY, -1);
+ }
- federation = nnTopology.isFederated();
- try {
- createNameNodesAndSetConf(
- nnTopology, manageNameDfsDirs, manageNameDfsSharedDirs,
- enableManagedDfsDirsRedundancy,
- format, startOpt, clusterId, conf);
- } catch (IOException ioe) {
- LOG.error("IOE creating namenodes. Permissions dump:\n" +
- createPermissionsDiagnosisString(data_dir));
- throw ioe;
- }
- if (format) {
- if (data_dir.exists() && !FileUtil.fullyDelete(data_dir)) {
- throw new IOException("Cannot remove data directory: " + data_dir +
+ federation = nnTopology.isFederated();
+ try {
+ createNameNodesAndSetConf(
+ nnTopology, manageNameDfsDirs, manageNameDfsSharedDirs,
+ enableManagedDfsDirsRedundancy,
+ format, startOpt, clusterId, conf);
+ } catch (IOException ioe) {
+ LOG.error("IOE creating namenodes. Permissions dump:\n" +
createPermissionsDiagnosisString(data_dir));
+ throw ioe;
+ }
+ if (format) {
+ if (data_dir.exists() && !FileUtil.fullyDelete(data_dir)) {
+ throw new IOException("Cannot remove data directory: " + data_dir +
+ createPermissionsDiagnosisString(data_dir));
+ }
}
- }
- if (startOpt == StartupOption.RECOVER) {
- return;
- }
+ if (startOpt == StartupOption.RECOVER) {
+ return;
+ }
- // Start the DataNodes
- startDataNodes(conf, numDataNodes, storageType, manageDataDfsDirs,
- dnStartOpt != null ? dnStartOpt : startOpt,
- racks, hosts, simulatedCapacities, setupHostsFile,
- checkDataNodeAddrConfig, checkDataNodeHostConfig, dnConfOverlays);
- waitClusterUp();
- //make sure ProxyUsers uses the latest conf
- ProxyUsers.refreshSuperUserGroupsConfiguration(conf);
+ // Start the DataNodes
+ startDataNodes(conf, numDataNodes, storageType, manageDataDfsDirs,
+ dnStartOpt != null ? dnStartOpt : startOpt,
+ racks, hosts, simulatedCapacities, setupHostsFile,
+ checkDataNodeAddrConfig, checkDataNodeHostConfig, dnConfOverlays);
+ waitClusterUp();
+ //make sure ProxyUsers uses the latest conf
+ ProxyUsers.refreshSuperUserGroupsConfiguration(conf);
+ success = true;
+ } finally {
+ if (!success) {
+ shutdown();
+ }
+ }
}
/**
@@ -758,8 +771,11 @@ public class MiniDFSCluster {
if (!federation && nnTopology.countNameNodes() == 1) {
NNConf onlyNN = nnTopology.getOnlyNameNode();
- // we only had one NN, set DEFAULT_NAME for it
- conf.set(FS_DEFAULT_NAME_KEY, "127.0.0.1:" + onlyNN.getIpcPort());
+ // we only had one NN, set DEFAULT_NAME for it. If not explicitly
+ // specified initially, the port will be 0 to make NN bind to any
+ // available port. It will be set to the right address after
+ // NN is started.
+ conf.set(FS_DEFAULT_NAME_KEY, "hdfs://127.0.0.1:" + onlyNN.getIpcPort());
}
List<String> allNsIds = Lists.newArrayList();
@@ -775,6 +791,7 @@ public class MiniDFSCluster {
int nnCounter = 0;
for (MiniDFSNNTopology.NSConf nameservice : nnTopology.getNameservices()) {
String nsId = nameservice.getId();
+ String lastDefaultFileSystem = null;
Preconditions.checkArgument(
!federation || nsId != null,
@@ -842,6 +859,13 @@ public class MiniDFSCluster {
nnCounterForFormat++;
if (formatThisOne) {
+ // Allow overriding clusterID for specific NNs to test
+ // misconfiguration.
+ if (nn.getClusterId() == null) {
+ StartupOption.FORMAT.setClusterId(clusterId);
+ } else {
+ StartupOption.FORMAT.setClusterId(nn.getClusterId());
+ }
DFSTestUtil.formatNameNode(conf);
}
prevNNDirs = namespaceDirs;
@@ -851,10 +875,19 @@ public class MiniDFSCluster {
for (NNConf nn : nameservice.getNNs()) {
initNameNodeConf(conf, nsId, nn.getNnId(), manageNameDfsDirs,
enableManagedDfsDirsRedundancy, nnCounter);
- createNameNode(nnCounter++, conf, numDataNodes, false, operation,
+ createNameNode(nnCounter, conf, numDataNodes, false, operation,
clusterId, nsId, nn.getNnId());
+ // Record the last namenode uri
+ if (nameNodes[nnCounter] != null && nameNodes[nnCounter].conf != null) {
+ lastDefaultFileSystem =
+ nameNodes[nnCounter].conf.get(FS_DEFAULT_NAME_KEY);
+ }
+ nnCounter++;
+ }
+ if (!federation && lastDefaultFileSystem != null) {
+ // Set the default file system to the actual bind address of NN.
+ conf.set(FS_DEFAULT_NAME_KEY, lastDefaultFileSystem);
}
-
}
}
@@ -903,7 +936,7 @@ public class MiniDFSCluster {
}
}
- private void copyNameDirs(Collection<URI> srcDirs, Collection<URI> dstDirs,
+ public static void copyNameDirs(Collection<URI> srcDirs, Collection<URI> dstDirs,
Configuration dstConf) throws IOException {
URI srcDir = Lists.newArrayList(srcDirs).get(0);
FileSystem dstFS = FileSystem.getLocal(dstConf).getRaw();
@@ -968,7 +1001,8 @@ public class MiniDFSCluster {
operation.setClusterId(clusterId);
}
- // Start the NameNode
+ // Start the NameNode after saving the default file system.
+ String originalDefaultFs = conf.get(FS_DEFAULT_NAME_KEY);
String[] args = createArgs(operation);
NameNode nn = NameNode.createNameNode(args, conf);
if (operation == StartupOption.RECOVER) {
@@ -992,6 +1026,12 @@ public class MiniDFSCluster {
DFS_NAMENODE_HTTP_ADDRESS_KEY);
nameNodes[nnIndex] = new NameNodeInfo(nn, nameserviceId, nnId,
operation, new Configuration(conf));
+ // Restore the default fs name
+ if (originalDefaultFs == null) {
+ conf.set(FS_DEFAULT_NAME_KEY, "");
+ } else {
+ conf.set(FS_DEFAULT_NAME_KEY, originalDefaultFs);
+ }
}
/**
@@ -1279,15 +1319,42 @@ public class MiniDFSCluster {
}
SecureResources secureResources = null;
- if (UserGroupInformation.isSecurityEnabled()) {
+ if (UserGroupInformation.isSecurityEnabled() &&
+ conf.get(DFS_DATA_TRANSFER_PROTECTION_KEY) == null) {
try {
secureResources = SecureDataNodeStarter.getSecureResources(dnConf);
} catch (Exception ex) {
ex.printStackTrace();
}
}
- DataNode dn = DataNode.instantiateDataNode(dnArgs, dnConf,
- secureResources);
+ final int maxRetriesOnSasl = conf.getInt(
+ IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_KEY,
+ IPC_CLIENT_CONNECT_MAX_RETRIES_ON_SASL_DEFAULT);
+ int numRetries = 0;
+ DataNode dn = null;
+ while (true) {
+ try {
+ dn = DataNode.instantiateDataNode(dnArgs, dnConf,
+ secureResources);
+ break;
+ } catch (IOException e) {
+ // Work around issue testing security where rapidly starting multiple
+ // DataNodes using the same principal gets rejected by the KDC as a
+ // replay attack.
+ if (UserGroupInformation.isSecurityEnabled() &&
+ numRetries < maxRetriesOnSasl) {
+ try {
+ Thread.sleep(1000);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ break;
+ }
+ ++numRetries;
+ continue;
+ }
+ throw e;
+ }
+ }
if(dn == null)
throw new IOException("Cannot start DataNode in "
+ dnConf.get(DFS_DATANODE_DATA_DIR_KEY));
@@ -1384,8 +1451,8 @@ public class MiniDFSCluster {
/**
* Finalize cluster for the namenode at the given index
* @see MiniDFSCluster#finalizeCluster(Configuration)
- * @param nnIndex
- * @param conf
+ * @param nnIndex index of the namenode
+ * @param conf configuration
* @throws Exception
*/
public void finalizeCluster(int nnIndex, Configuration conf) throws Exception {
@@ -1698,6 +1765,14 @@ public class MiniDFSCluster {
LOG.warn("Corrupting the block " + blockFile);
return true;
}
+
+ public static boolean changeGenStampOfBlock(int dnIndex, ExtendedBlock blk,
+ long newGenStamp) throws IOException {
+ File blockFile = getBlockFile(dnIndex, blk);
+ File metaFile = FsDatasetUtil.findMetaFile(blockFile);
+ return metaFile.renameTo(new File(DatanodeUtil.getMetaName(
+ blockFile.getAbsolutePath(), newGenStamp)));
+ }
/*
* Shutdown a particular datanode
@@ -2216,7 +2291,7 @@ public class MiniDFSCluster {
* to determine the location of the storage of a DN instance in the mini cluster
* @param dnIndex datanode index
* @param dirIndex directory index.
- * @return
+ * @return storage directory path
*/
private static String getStorageDirPath(int dnIndex, int dirIndex) {
return "data/data" + (2 * dnIndex + 1 + dirIndex);
@@ -2242,8 +2317,8 @@ public class MiniDFSCluster {
}
/**
* Get directory relative to block pool directory in the datanode
- * @param storageDir
- * @return current directory
+ * @param storageDir storage directory
+ * @return current directory in the given storage directory
*/
public static String getBPDir(File storageDir, String bpid, String dirName) {
return getBPDir(storageDir, bpid) + dirName + "/";
@@ -2278,8 +2353,8 @@ public class MiniDFSCluster {
* @return data file corresponding to the block
*/
public static File getBlockFile(File storageDir, ExtendedBlock blk) {
- return new File(getFinalizedDir(storageDir, blk.getBlockPoolId()),
- blk.getBlockName());
+ return new File(DatanodeUtil.idToBlockDir(getFinalizedDir(storageDir,
+ blk.getBlockPoolId()), blk.getBlockId()), blk.getBlockName());
}
/**
@@ -2289,10 +2364,32 @@ public class MiniDFSCluster {
* @return metadata file corresponding to the block
*/
public static File getBlockMetadataFile(File storageDir, ExtendedBlock blk) {
- return new File(getFinalizedDir(storageDir, blk.getBlockPoolId()),
- blk.getBlockName() + "_" + blk.getGenerationStamp() +
- Block.METADATA_EXTENSION);
-
+ return new File(DatanodeUtil.idToBlockDir(getFinalizedDir(storageDir,
+ blk.getBlockPoolId()), blk.getBlockId()), blk.getBlockName() + "_" +
+ blk.getGenerationStamp() + Block.METADATA_EXTENSION);
+ }
+
+ /**
+ * Return all block metadata files in given directory (recursive search)
+ */
+ public static List<File> getAllBlockMetadataFiles(File storageDir) {
+ List<File> results = new ArrayList<File>();
+ File[] files = storageDir.listFiles();
+ if (files == null) {
+ return null;
+ }
+ for (File f : files) {
+ if (f.getName().startsWith("blk_") && f.getName().endsWith(
+ Block.METADATA_EXTENSION)) {
+ results.add(f);
+ } else if (f.isDirectory()) {
+ List<File> subdirResults = getAllBlockMetadataFiles(f);
+ if (subdirResults != null) {
+ results.addAll(subdirResults);
+ }
+ }
+ }
+ return results;
}
/**
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSNNTopology.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSNNTopology.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSNNTopology.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSNNTopology.java Tue Aug 19 23:49:39 2014
@@ -211,6 +211,7 @@ public class MiniDFSNNTopology {
private final String nnId;
private int httpPort;
private int ipcPort;
+ private String clusterId;
public NNConf(String nnId) {
this.nnId = nnId;
@@ -228,6 +229,10 @@ public class MiniDFSNNTopology {
return httpPort;
}
+ String getClusterId() {
+ return clusterId;
+ }
+
public NNConf setHttpPort(int httpPort) {
this.httpPort = httpPort;
return this;
@@ -237,6 +242,11 @@ public class MiniDFSNNTopology {
this.ipcPort = ipcPort;
return this;
}
+
+ public NNConf setClusterId(String clusterId) {
+ this.clusterId = clusterId;
+ return this;
+ }
}
}
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAbandonBlock.java Tue Aug 19 23:49:39 2014
@@ -71,6 +71,7 @@ public class TestAbandonBlock {
fout.write(123);
}
fout.hflush();
+ long fileId = ((DFSOutputStream)fout.getWrappedStream()).getFileId();
// Now abandon the last block
DFSClient dfsclient = DFSClientAdapter.getDFSClient(fs);
@@ -78,11 +79,11 @@ public class TestAbandonBlock {
dfsclient.getNamenode().getBlockLocations(src, 0, Integer.MAX_VALUE);
int orginalNumBlocks = blocks.locatedBlockCount();
LocatedBlock b = blocks.getLastLocatedBlock();
- dfsclient.getNamenode().abandonBlock(b.getBlock(), src,
+ dfsclient.getNamenode().abandonBlock(b.getBlock(), fileId, src,
dfsclient.clientName);
// call abandonBlock again to make sure the operation is idempotent
- dfsclient.getNamenode().abandonBlock(b.getBlock(), src,
+ dfsclient.getNamenode().abandonBlock(b.getBlock(), fileId, src,
dfsclient.clientName);
// And close the file
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockMissingException.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockMissingException.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockMissingException.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockMissingException.java Tue Aug 19 23:49:39 2014
@@ -51,6 +51,8 @@ public class TestBlockMissingException {
long blockSize = 1024L;
int numBlocks = 4;
conf = new HdfsConfiguration();
+ // Set short retry timeouts so this test runs faster
+ conf.setInt(DFSConfigKeys.DFS_CLIENT_RETRY_WINDOW_BASE, 10);
try {
dfs = new MiniDFSCluster.Builder(conf).numDataNodes(NUM_DATANODES).build();
dfs.waitActive();
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderFactory.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderFactory.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderFactory.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderFactory.java Tue Aug 19 23:49:39 2014
@@ -28,12 +28,15 @@ import static org.hamcrest.CoreMatchers.
import java.io.File;
import java.io.IOException;
+import java.nio.channels.ClosedByInterruptException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -409,4 +412,121 @@ public class TestBlockReaderFactory {
getDomainSocketWatcher().isClosed());
cluster.shutdown();
}
+
+ /**
+ * When an InterruptedException is sent to a thread calling
+ * FileChannel#read, the FileChannel is immediately closed and the
+ * thread gets an exception. This effectively means that we might have
+ * someone asynchronously calling close() on the file descriptors we use
+ * in BlockReaderLocal. So when unreferencing a ShortCircuitReplica in
+ * ShortCircuitCache#unref, we should check if the FileChannel objects
+ * are still open. If not, we should purge the replica to avoid giving
+ * it out to any future readers.
+ *
+ * This is a regression test for HDFS-6227: Short circuit read failed
+ * due to ClosedChannelException.
+ *
+ * Note that you may still get ClosedChannelException errors if two threads
+ * are reading from the same replica and an InterruptedException is delivered
+ * to one of them.
+ */
+ @Test(timeout=120000)
+ public void testPurgingClosedReplicas() throws Exception {
+ BlockReaderTestUtil.enableBlockReaderFactoryTracing();
+ final AtomicInteger replicasCreated = new AtomicInteger(0);
+ final AtomicBoolean testFailed = new AtomicBoolean(false);
+ DFSInputStream.tcpReadsDisabledForTesting = true;
+ BlockReaderFactory.createShortCircuitReplicaInfoCallback =
+ new ShortCircuitCache.ShortCircuitReplicaCreator() {
+ @Override
+ public ShortCircuitReplicaInfo createShortCircuitReplicaInfo() {
+ replicasCreated.incrementAndGet();
+ return null;
+ }
+ };
+ TemporarySocketDirectory sockDir = new TemporarySocketDirectory();
+ Configuration conf = createShortCircuitConf(
+ "testPurgingClosedReplicas", sockDir);
+ final MiniDFSCluster cluster =
+ new MiniDFSCluster.Builder(conf).numDataNodes(1).build();
+ cluster.waitActive();
+ final DistributedFileSystem dfs = cluster.getFileSystem();
+ final String TEST_FILE = "/test_file";
+ final int TEST_FILE_LEN = 4095;
+ final int SEED = 0xFADE0;
+ final DistributedFileSystem fs =
+ (DistributedFileSystem)FileSystem.get(cluster.getURI(0), conf);
+ DFSTestUtil.createFile(fs, new Path(TEST_FILE), TEST_FILE_LEN,
+ (short)1, SEED);
+
+ final Semaphore sem = new Semaphore(0);
+ final List<LocatedBlock> locatedBlocks =
+ cluster.getNameNode().getRpcServer().getBlockLocations(
+ TEST_FILE, 0, TEST_FILE_LEN).getLocatedBlocks();
+ final LocatedBlock lblock = locatedBlocks.get(0); // first block
+ final byte[] buf = new byte[TEST_FILE_LEN];
+ Runnable readerRunnable = new Runnable() {
+ @Override
+ public void run() {
+ try {
+ while (true) {
+ BlockReader blockReader = null;
+ try {
+ blockReader = BlockReaderTestUtil.
+ getBlockReader(cluster, lblock, 0, TEST_FILE_LEN);
+ sem.release();
+ try {
+ blockReader.readAll(buf, 0, TEST_FILE_LEN);
+ } finally {
+ sem.acquireUninterruptibly();
+ }
+ } catch (ClosedByInterruptException e) {
+ LOG.info("got the expected ClosedByInterruptException", e);
+ sem.release();
+ break;
+ } finally {
+ if (blockReader != null) blockReader.close();
+ }
+ LOG.info("read another " + TEST_FILE_LEN + " bytes.");
+ }
+ } catch (Throwable t) {
+ LOG.error("getBlockReader failure", t);
+ testFailed.set(true);
+ sem.release();
+ }
+ }
+ };
+ Thread thread = new Thread(readerRunnable);
+ thread.start();
+
+ // While the thread is reading, send it interrupts.
+ // These should trigger a ClosedChannelException.
+ while (thread.isAlive()) {
+ sem.acquireUninterruptibly();
+ thread.interrupt();
+ sem.release();
+ }
+ Assert.assertFalse(testFailed.get());
+
+ // We should be able to read from the file without
+ // getting a ClosedChannelException.
+ BlockReader blockReader = null;
+ try {
+ blockReader = BlockReaderTestUtil.
+ getBlockReader(cluster, lblock, 0, TEST_FILE_LEN);
+ blockReader.readFully(buf, 0, TEST_FILE_LEN);
+ } finally {
+ if (blockReader != null) blockReader.close();
+ }
+ byte expected[] = DFSTestUtil.
+ calculateFileContentsFromSeed(SEED, TEST_FILE_LEN);
+ Assert.assertTrue(Arrays.equals(buf, expected));
+
+ // Another ShortCircuitReplica object should have been created.
+ Assert.assertEquals(2, replicasCreated.get());
+
+ dfs.close();
+ cluster.shutdown();
+ sockDir.close();
+ }
}
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockReaderLocalLegacy.java Tue Aug 19 23:49:39 2014
@@ -64,6 +64,8 @@ public class TestBlockReaderLocalLegacy
conf.set(DFSConfigKeys.DFS_BLOCK_LOCAL_PATH_ACCESS_USER_KEY,
UserGroupInformation.getCurrentUser().getShortUserName());
conf.setBoolean(DFSConfigKeys.DFS_CLIENT_DOMAIN_SOCKET_DATA_TRAFFIC, false);
+ // Set short retry timeouts so this test runs faster
+ conf.setInt(DFSConfigKeys.DFS_CLIENT_RETRY_WINDOW_BASE, 10);
return conf;
}
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientReportBadBlock.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientReportBadBlock.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientReportBadBlock.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientReportBadBlock.java Tue Aug 19 23:49:39 2014
@@ -73,7 +73,8 @@ public class TestClientReportBadBlock {
public void startUpCluster() throws IOException {
// disable block scanner
conf.setInt(DFSConfigKeys.DFS_DATANODE_SCAN_PERIOD_HOURS_KEY, -1);
-
+ // Set short retry timeouts so this test runs faster
+ conf.setInt(DFSConfigKeys.DFS_CLIENT_RETRY_WINDOW_BASE, 10);
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(numDataNodes)
.build();
cluster.waitActive();
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java Tue Aug 19 23:49:39 2014
@@ -88,6 +88,8 @@ public class TestCrcCorruption {
@Test(timeout=50000)
public void testCorruptionDuringWrt() throws Exception {
Configuration conf = new HdfsConfiguration();
+ // Set short retry timeouts so this test runs faster
+ conf.setInt(DFSConfigKeys.DFS_CLIENT_RETRY_WINDOW_BASE, 10);
MiniDFSCluster cluster = null;
try {
@@ -152,7 +154,8 @@ public class TestCrcCorruption {
int numDataNodes = 2;
short replFactor = 2;
Random random = new Random();
-
+ // Set short retry timeouts so this test runs faster
+ conf.setInt(DFSConfigKeys.DFS_CLIENT_RETRY_WINDOW_BASE, 10);
try {
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(numDataNodes).build();
cluster.waitActive();
@@ -334,6 +337,8 @@ public class TestCrcCorruption {
short replFactor = (short)numDataNodes;
Configuration conf = new Configuration();
conf.setInt(DFSConfigKeys.DFS_REPLICATION_KEY, numDataNodes);
+ // Set short retry timeouts so this test runs faster
+ conf.setInt(DFSConfigKeys.DFS_CLIENT_RETRY_WINDOW_BASE, 10);
MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(numDataNodes).build();
try {
Modified: hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
URL: http://svn.apache.org/viewvc/hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java?rev=1619012&r1=1619011&r2=1619012&view=diff
==============================================================================
--- hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java (original)
+++ hadoop/common/branches/HADOOP-10388/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java Tue Aug 19 23:49:39 2014
@@ -19,6 +19,7 @@ package org.apache.hadoop.hdfs;
import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -41,12 +42,18 @@ import org.apache.hadoop.fs.CommonConfig
import org.apache.hadoop.fs.FileContext;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.HAUtil;
import org.apache.hadoop.hdfs.server.namenode.NameNode;
import org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider;
+import org.apache.hadoop.hdfs.server.namenode.ha.IPFailoverProxyProvider;
import org.apache.hadoop.hdfs.server.namenode.ha.HATestUtil;
import org.apache.hadoop.io.IOUtils;
+import org.apache.hadoop.io.retry.DefaultFailoverProxyProvider;
+import org.apache.hadoop.io.retry.FailoverProxyProvider;
import org.apache.hadoop.net.ConnectTimeoutException;
import org.apache.hadoop.net.StandardSocketFactory;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.test.GenericTestUtils;
import org.apache.hadoop.util.StringUtils;
import org.hamcrest.BaseMatcher;
@@ -83,6 +90,11 @@ public class TestDFSClientFailover {
cluster.shutdown();
}
+ @After
+ public void clearConfig() {
+ SecurityUtil.setTokenServiceUseIp(true);
+ }
+
/**
* Make sure that client failover works when an active NN dies and the standby
* takes over.
@@ -172,12 +184,12 @@ public class TestDFSClientFailover {
*/
@Test
public void testLogicalUriShouldNotHavePorts() {
- Configuration conf = new HdfsConfiguration();
- conf.set(DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX + ".foo",
- ConfiguredFailoverProxyProvider.class.getName());
- Path p = new Path("hdfs://foo:12345/");
+ Configuration config = new HdfsConfiguration(conf);
+ String logicalName = HATestUtil.getLogicalHostname(cluster);
+ HATestUtil.setFailoverConfigurations(cluster, config, logicalName);
+ Path p = new Path("hdfs://" + logicalName + ":12345/");
try {
- p.getFileSystem(conf).exists(p);
+ p.getFileSystem(config).exists(p);
fail("Did not fail with fake FS");
} catch (IOException ioe) {
GenericTestUtils.assertExceptionContains(
@@ -278,4 +290,77 @@ public class TestDFSClientFailover {
// Ensure that the logical hostname was never resolved.
Mockito.verify(spyNS, Mockito.never()).lookupAllHostAddr(Mockito.eq(logicalHost));
}
+
+ /** Dummy implementation of plain FailoverProxyProvider */
+ public static class DummyLegacyFailoverProxyProvider<T>
+ implements FailoverProxyProvider<T> {
+ private Class<T> xface;
+ private T proxy;
+ public DummyLegacyFailoverProxyProvider(Configuration conf, URI uri,
+ Class<T> xface) {
+ try {
+ this.proxy = NameNodeProxies.createNonHAProxy(conf,
+ NameNode.getAddress(uri), xface,
+ UserGroupInformation.getCurrentUser(), false).getProxy();
+ this.xface = xface;
+ } catch (IOException ioe) {
+ }
+ }
+
+ @Override
+ public Class<T> getInterface() {
+ return xface;
+ }
+
+ @Override
+ public ProxyInfo<T> getProxy() {
+ return new ProxyInfo<T>(proxy, "dummy");
+ }
+
+ @Override
+ public void performFailover(T currentProxy) {
+ }
+
+ @Override
+ public void close() throws IOException {
+ }
+ }
+
+ /**
+ * Test to verify legacy proxy providers are correctly wrapped.
+ */
+ @Test
+ public void testWrappedFailoverProxyProvider() throws Exception {
+ // setup the config with the dummy provider class
+ Configuration config = new HdfsConfiguration(conf);
+ String logicalName = HATestUtil.getLogicalHostname(cluster);
+ HATestUtil.setFailoverConfigurations(cluster, config, logicalName);
+ config.set(DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX + "." + logicalName,
+ DummyLegacyFailoverProxyProvider.class.getName());
+ Path p = new Path("hdfs://" + logicalName + "/");
+
+ // not to use IP address for token service
+ SecurityUtil.setTokenServiceUseIp(false);
+
+ // Logical URI should be used.
+ assertTrue("Legacy proxy providers should use logical URI.",
+ HAUtil.useLogicalUri(config, p.toUri()));
+ }
+
+ /**
+ * Test to verify IPFailoverProxyProvider is not requiring logical URI.
+ */
+ @Test
+ public void testIPFailoverProxyProviderLogicalUri() throws Exception {
+ // setup the config with the IP failover proxy provider class
+ Configuration config = new HdfsConfiguration(conf);
+ URI nnUri = cluster.getURI(0);
+ config.set(DFS_CLIENT_FAILOVER_PROXY_PROVIDER_KEY_PREFIX + "." +
+ nnUri.getHost(),
+ IPFailoverProxyProvider.class.getName());
+
+ assertFalse("IPFailoverProxyProvider should not use logical URI.",
+ HAUtil.useLogicalUri(config, nnUri));
+ }
+
}