You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Nirmal Kumar <ni...@impetus.co.in> on 2013/12/10 12:00:08 UTC
Unable to run dfsadmin -upgradeProgress status in Apache Hadoop
2.1.0-beta
Hi All,
hadoop dfsadmin -help
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
hadoop dfsadmin is the command to execute DFS administrative commands.
The full syntax is:
hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth <bandwidth>]
[-fetchImage <local directory>]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-help [cmd]]
-report: Reports basic filesystem information and statistics.
-safemode <enter|leave|get|wait>: Safe mode maintenance command.
Safe mode is a Namenode state in which it
1. does not accept changes to the name space (read-only)
2. does not replicate or delete blocks.
Safe mode is entered automatically at Namenode startup, and
leaves safe mode automatically when the configured minimum
percentage of blocks satisfies the minimum replication
condition. Safe mode can also be entered manually, but then
it can only be turned off manually as well.
-saveNamespace: Save current namespace into storage directories and reset edits log.
Requires superuser permissions and safe mode.
-rollEdits: Rolls the edit log.
Requires superuser permissions.
-restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed storage replicas if they become available.
Requires superuser permissions.
-refreshNodes: Updates the namenode with the set of datanodes allowed to connect to the namenode.
Namenode re-reads datanode hostnames from the file defined by
dfs.hosts, dfs.hosts.exclude configuration parameters.
Hosts defined in dfs.hosts are the datanodes that are part of
the cluster. If there are entries in dfs.hosts, only the hosts
in it are allowed to register with the namenode.
Entries in dfs.hosts.exclude are datanodes that need to be
decommissioned. Datanodes complete decommissioning when
all the replicas from them are replicated to other datanodes.
Decommissioned nodes are not automatically shutdown and
are not chosen for writing new replicas.
-finalizeUpgrade: Finalize upgrade of HDFS.
Datanodes delete their previous version working directories,
followed by Namenode doing the same.
This completes the upgrade process.
-metasave <filename>: Save Namenode's primary data structures
to <filename> in the directory specified by hadoop.log.dir property.
<filename> is overwritten if it exists.
<filename> will contain one line for each of the following
1. Datanodes heart beating with Namenode
2. Blocks waiting to be replicated
3. Blocks currrently being replicated
4. Blocks waiting to be deleted
-setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each directory <dirName>.
The directory quota is a long integer that puts a hard limit
on the number of names in the directory tree
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. User is not an administrator, or
3. The directory does not exist or is a file.
Note: A quota of 1 would force the directory to remain empty.
-clrQuota <dirname>...<dirname>: Clear the quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota <quota> for each directory <dirName>.
The space quota is a long integer that puts a hard limit
on the total size of all the files under the directory tree.
The extra space required for replication is also counted. E.g.
a 1GB file with replication of 3 consumes 3GB of the quota.
Quota can also be speciefied with a binary prefix for terabytes,
petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. user is not an administrator, or
3. the directory does not exist or is a file, or
-clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-refreshServiceAcl: Reload the service-level authorization policy file
Namenode will reload the authorization policy file.
-refreshUserToGroupsMappings: Refresh user-to-groups mappings
-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups mappings
-printTopology: Print a tree of the racks and their
nodes as reported by the Namenode
-refreshNamenodes: Takes a datanodehost:port as argument,
For the given datanode, reloads the configuration files,
stops serving the removed block-pools
and starts serving new block-pools
-deleteBlockPool: Arguments are datanodehost:port, blockpool id
and an optional argument "force". If force is passed,
block pool directory for the given blockpool id on the given
datanode is deleted along with its contents, otherwise
the directory is deleted only if it is empty. The command
will fail if datanode is still serving the block pool.
Refer to refreshNamenodes to shutdown a block pool
service on a datanode.
-setBalancerBandwidth <bandwidth>:
Changes the network bandwidth used by each datanode during
HDFS block balancing.
<bandwidth> is the maximum number of bytes per second
that will be used by each datanode. This value overrides
the dfs.balance.bandwidthPerSec parameter.
--- NOTE: The new value is not persistent on the DataNode.---
-fetchImage <local directory>:
Downloads the most recent fsimage from the Name Node and saves it in the specified local directory.
-allowSnapshot <snapshotDir>:
Allow snapshots to be taken on a directory.
-disallowSnapshot <snapshotDir>:
Do not allow snapshots to be taken on a directory any more.
-help [cmd]: Displays help for the given command or all commands if none
is specified.
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Even tried the hdfs command:
[cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Is there is some other option for checking the upgrade progress?
Thanks,
-Nirmal
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop
2.1.0-beta
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Smarty,
I tried the same and didn't get it working so thought of *-help* option.
This command is working with Apache Hadoop 1.2.0 though:
hadoop dfsadmin -upgradeProgress status
There are no upgrades in progress.
But with Apache Hadoop 2.1.0-beta I get the following:
[cloud@Impetus-942 bin]$ hadoop dfsadmin -upgradeProgress status
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Thanks,
-Nirmal
From: Smarty Juice [mailto:smarty.juice@gmail.com]
Sent: Tuesday, December 10, 2013 7:12 PM
To: user@hadoop.apache.org
Subject: Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
hadoop dfsadmin -help
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
hadoop dfsadmin is the command to execute DFS administrative commands.
The full syntax is:
hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth <bandwidth>]
[-fetchImage <local directory>]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-help [cmd]]
-report: Reports basic filesystem information and statistics.
-safemode <enter|leave|get|wait>: Safe mode maintenance command.
Safe mode is a Namenode state in which it
1. does not accept changes to the name space (read-only)
2. does not replicate or delete blocks.
Safe mode is entered automatically at Namenode startup, and
leaves safe mode automatically when the configured minimum
percentage of blocks satisfies the minimum replication
condition. Safe mode can also be entered manually, but then
it can only be turned off manually as well.
-saveNamespace: Save current namespace into storage directories and reset edits log.
Requires superuser permissions and safe mode.
-rollEdits: Rolls the edit log.
Requires superuser permissions.
-restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed storage replicas if they become available.
Requires superuser permissions.
-refreshNodes: Updates the namenode with the set of datanodes allowed to connect to the namenode.
Namenode re-reads datanode hostnames from the file defined by
dfs.hosts, dfs.hosts.exclude configuration parameters.
Hosts defined in dfs.hosts are the datanodes that are part of
the cluster. If there are entries in dfs.hosts, only the hosts
in it are allowed to register with the namenode.
Entries in dfs.hosts.exclude are datanodes that need to be
decommissioned. Datanodes complete decommissioning when
all the replicas from them are replicated to other datanodes.
Decommissioned nodes are not automatically shutdown and
are not chosen for writing new replicas.
-finalizeUpgrade: Finalize upgrade of HDFS.
Datanodes delete their previous version working directories,
followed by Namenode doing the same.
This completes the upgrade process.
-metasave <filename>: Save Namenode's primary data structures
to <filename> in the directory specified by hadoop.log.dir property.
<filename> is overwritten if it exists.
<filename> will contain one line for each of the following
1. Datanodes heart beating with Namenode
2. Blocks waiting to be replicated
3. Blocks currrently being replicated
4. Blocks waiting to be deleted
-setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each directory <dirName>.
The directory quota is a long integer that puts a hard limit
on the number of names in the directory tree
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. User is not an administrator, or
3. The directory does not exist or is a file.
Note: A quota of 1 would force the directory to remain empty.
-clrQuota <dirname>...<dirname>: Clear the quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota <quota> for each directory <dirName>.
The space quota is a long integer that puts a hard limit
on the total size of all the files under the directory tree.
The extra space required for replication is also counted. E.g.
a 1GB file with replication of 3 consumes 3GB of the quota.
Quota can also be speciefied with a binary prefix for terabytes,
petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. user is not an administrator, or
3. the directory does not exist or is a file, or
-clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-refreshServiceAcl: Reload the service-level authorization policy file
Namenode will reload the authorization policy file.
-refreshUserToGroupsMappings: Refresh user-to-groups mappings
-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups mappings
-printTopology: Print a tree of the racks and their
nodes as reported by the Namenode
-refreshNamenodes: Takes a datanodehost:port as argument,
For the given datanode, reloads the configuration files,
stops serving the removed block-pools
and starts serving new block-pools
-deleteBlockPool: Arguments are datanodehost:port, blockpool id
and an optional argument "force". If force is passed,
block pool directory for the given blockpool id on the given
datanode is deleted along with its contents, otherwise
the directory is deleted only if it is empty. The command
will fail if datanode is still serving the block pool.
Refer to refreshNamenodes to shutdown a block pool
service on a datanode.
-setBalancerBandwidth <bandwidth>:
Changes the network bandwidth used by each datanode during
HDFS block balancing.
<bandwidth> is the maximum number of bytes per second
that will be used by each datanode. This value overrides
the dfs.balance.bandwidthPerSec parameter.
--- NOTE: The new value is not persistent on the DataNode.---
-fetchImage <local directory>:
Downloads the most recent fsimage from the Name Node and saves it in the specified local directory.
-allowSnapshot <snapshotDir>:
Allow snapshots to be taken on a directory.
-disallowSnapshot <snapshotDir>:
Do not allow snapshots to be taken on a directory any more.
-help [cmd]: Displays help for the given command or all commands if none
is specified.
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Even tried the hdfs command:
[cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Is there is some other option for checking the upgrade progress?
Thanks,
-Nirmal
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop
2.1.0-beta
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Smarty,
I tried the same and didn't get it working so thought of *-help* option.
This command is working with Apache Hadoop 1.2.0 though:
hadoop dfsadmin -upgradeProgress status
There are no upgrades in progress.
But with Apache Hadoop 2.1.0-beta I get the following:
[cloud@Impetus-942 bin]$ hadoop dfsadmin -upgradeProgress status
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Thanks,
-Nirmal
From: Smarty Juice [mailto:smarty.juice@gmail.com]
Sent: Tuesday, December 10, 2013 7:12 PM
To: user@hadoop.apache.org
Subject: Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
hadoop dfsadmin -help
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
hadoop dfsadmin is the command to execute DFS administrative commands.
The full syntax is:
hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth <bandwidth>]
[-fetchImage <local directory>]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-help [cmd]]
-report: Reports basic filesystem information and statistics.
-safemode <enter|leave|get|wait>: Safe mode maintenance command.
Safe mode is a Namenode state in which it
1. does not accept changes to the name space (read-only)
2. does not replicate or delete blocks.
Safe mode is entered automatically at Namenode startup, and
leaves safe mode automatically when the configured minimum
percentage of blocks satisfies the minimum replication
condition. Safe mode can also be entered manually, but then
it can only be turned off manually as well.
-saveNamespace: Save current namespace into storage directories and reset edits log.
Requires superuser permissions and safe mode.
-rollEdits: Rolls the edit log.
Requires superuser permissions.
-restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed storage replicas if they become available.
Requires superuser permissions.
-refreshNodes: Updates the namenode with the set of datanodes allowed to connect to the namenode.
Namenode re-reads datanode hostnames from the file defined by
dfs.hosts, dfs.hosts.exclude configuration parameters.
Hosts defined in dfs.hosts are the datanodes that are part of
the cluster. If there are entries in dfs.hosts, only the hosts
in it are allowed to register with the namenode.
Entries in dfs.hosts.exclude are datanodes that need to be
decommissioned. Datanodes complete decommissioning when
all the replicas from them are replicated to other datanodes.
Decommissioned nodes are not automatically shutdown and
are not chosen for writing new replicas.
-finalizeUpgrade: Finalize upgrade of HDFS.
Datanodes delete their previous version working directories,
followed by Namenode doing the same.
This completes the upgrade process.
-metasave <filename>: Save Namenode's primary data structures
to <filename> in the directory specified by hadoop.log.dir property.
<filename> is overwritten if it exists.
<filename> will contain one line for each of the following
1. Datanodes heart beating with Namenode
2. Blocks waiting to be replicated
3. Blocks currrently being replicated
4. Blocks waiting to be deleted
-setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each directory <dirName>.
The directory quota is a long integer that puts a hard limit
on the number of names in the directory tree
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. User is not an administrator, or
3. The directory does not exist or is a file.
Note: A quota of 1 would force the directory to remain empty.
-clrQuota <dirname>...<dirname>: Clear the quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota <quota> for each directory <dirName>.
The space quota is a long integer that puts a hard limit
on the total size of all the files under the directory tree.
The extra space required for replication is also counted. E.g.
a 1GB file with replication of 3 consumes 3GB of the quota.
Quota can also be speciefied with a binary prefix for terabytes,
petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. user is not an administrator, or
3. the directory does not exist or is a file, or
-clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-refreshServiceAcl: Reload the service-level authorization policy file
Namenode will reload the authorization policy file.
-refreshUserToGroupsMappings: Refresh user-to-groups mappings
-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups mappings
-printTopology: Print a tree of the racks and their
nodes as reported by the Namenode
-refreshNamenodes: Takes a datanodehost:port as argument,
For the given datanode, reloads the configuration files,
stops serving the removed block-pools
and starts serving new block-pools
-deleteBlockPool: Arguments are datanodehost:port, blockpool id
and an optional argument "force". If force is passed,
block pool directory for the given blockpool id on the given
datanode is deleted along with its contents, otherwise
the directory is deleted only if it is empty. The command
will fail if datanode is still serving the block pool.
Refer to refreshNamenodes to shutdown a block pool
service on a datanode.
-setBalancerBandwidth <bandwidth>:
Changes the network bandwidth used by each datanode during
HDFS block balancing.
<bandwidth> is the maximum number of bytes per second
that will be used by each datanode. This value overrides
the dfs.balance.bandwidthPerSec parameter.
--- NOTE: The new value is not persistent on the DataNode.---
-fetchImage <local directory>:
Downloads the most recent fsimage from the Name Node and saves it in the specified local directory.
-allowSnapshot <snapshotDir>:
Allow snapshots to be taken on a directory.
-disallowSnapshot <snapshotDir>:
Do not allow snapshots to be taken on a directory any more.
-help [cmd]: Displays help for the given command or all commands if none
is specified.
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Even tried the hdfs command:
[cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Is there is some other option for checking the upgrade progress?
Thanks,
-Nirmal
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop
2.1.0-beta
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Smarty,
I tried the same and didn't get it working so thought of *-help* option.
This command is working with Apache Hadoop 1.2.0 though:
hadoop dfsadmin -upgradeProgress status
There are no upgrades in progress.
But with Apache Hadoop 2.1.0-beta I get the following:
[cloud@Impetus-942 bin]$ hadoop dfsadmin -upgradeProgress status
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Thanks,
-Nirmal
From: Smarty Juice [mailto:smarty.juice@gmail.com]
Sent: Tuesday, December 10, 2013 7:12 PM
To: user@hadoop.apache.org
Subject: Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
hadoop dfsadmin -help
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
hadoop dfsadmin is the command to execute DFS administrative commands.
The full syntax is:
hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth <bandwidth>]
[-fetchImage <local directory>]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-help [cmd]]
-report: Reports basic filesystem information and statistics.
-safemode <enter|leave|get|wait>: Safe mode maintenance command.
Safe mode is a Namenode state in which it
1. does not accept changes to the name space (read-only)
2. does not replicate or delete blocks.
Safe mode is entered automatically at Namenode startup, and
leaves safe mode automatically when the configured minimum
percentage of blocks satisfies the minimum replication
condition. Safe mode can also be entered manually, but then
it can only be turned off manually as well.
-saveNamespace: Save current namespace into storage directories and reset edits log.
Requires superuser permissions and safe mode.
-rollEdits: Rolls the edit log.
Requires superuser permissions.
-restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed storage replicas if they become available.
Requires superuser permissions.
-refreshNodes: Updates the namenode with the set of datanodes allowed to connect to the namenode.
Namenode re-reads datanode hostnames from the file defined by
dfs.hosts, dfs.hosts.exclude configuration parameters.
Hosts defined in dfs.hosts are the datanodes that are part of
the cluster. If there are entries in dfs.hosts, only the hosts
in it are allowed to register with the namenode.
Entries in dfs.hosts.exclude are datanodes that need to be
decommissioned. Datanodes complete decommissioning when
all the replicas from them are replicated to other datanodes.
Decommissioned nodes are not automatically shutdown and
are not chosen for writing new replicas.
-finalizeUpgrade: Finalize upgrade of HDFS.
Datanodes delete their previous version working directories,
followed by Namenode doing the same.
This completes the upgrade process.
-metasave <filename>: Save Namenode's primary data structures
to <filename> in the directory specified by hadoop.log.dir property.
<filename> is overwritten if it exists.
<filename> will contain one line for each of the following
1. Datanodes heart beating with Namenode
2. Blocks waiting to be replicated
3. Blocks currrently being replicated
4. Blocks waiting to be deleted
-setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each directory <dirName>.
The directory quota is a long integer that puts a hard limit
on the number of names in the directory tree
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. User is not an administrator, or
3. The directory does not exist or is a file.
Note: A quota of 1 would force the directory to remain empty.
-clrQuota <dirname>...<dirname>: Clear the quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota <quota> for each directory <dirName>.
The space quota is a long integer that puts a hard limit
on the total size of all the files under the directory tree.
The extra space required for replication is also counted. E.g.
a 1GB file with replication of 3 consumes 3GB of the quota.
Quota can also be speciefied with a binary prefix for terabytes,
petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. user is not an administrator, or
3. the directory does not exist or is a file, or
-clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-refreshServiceAcl: Reload the service-level authorization policy file
Namenode will reload the authorization policy file.
-refreshUserToGroupsMappings: Refresh user-to-groups mappings
-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups mappings
-printTopology: Print a tree of the racks and their
nodes as reported by the Namenode
-refreshNamenodes: Takes a datanodehost:port as argument,
For the given datanode, reloads the configuration files,
stops serving the removed block-pools
and starts serving new block-pools
-deleteBlockPool: Arguments are datanodehost:port, blockpool id
and an optional argument "force". If force is passed,
block pool directory for the given blockpool id on the given
datanode is deleted along with its contents, otherwise
the directory is deleted only if it is empty. The command
will fail if datanode is still serving the block pool.
Refer to refreshNamenodes to shutdown a block pool
service on a datanode.
-setBalancerBandwidth <bandwidth>:
Changes the network bandwidth used by each datanode during
HDFS block balancing.
<bandwidth> is the maximum number of bytes per second
that will be used by each datanode. This value overrides
the dfs.balance.bandwidthPerSec parameter.
--- NOTE: The new value is not persistent on the DataNode.---
-fetchImage <local directory>:
Downloads the most recent fsimage from the Name Node and saves it in the specified local directory.
-allowSnapshot <snapshotDir>:
Allow snapshots to be taken on a directory.
-disallowSnapshot <snapshotDir>:
Do not allow snapshots to be taken on a directory any more.
-help [cmd]: Displays help for the given command or all commands if none
is specified.
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Even tried the hdfs command:
[cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Is there is some other option for checking the upgrade progress?
Thanks,
-Nirmal
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop
2.1.0-beta
Posted by Nirmal Kumar <ni...@impetus.co.in>.
Hi Smarty,
I tried the same and didn't get it working so thought of *-help* option.
This command is working with Apache Hadoop 1.2.0 though:
hadoop dfsadmin -upgradeProgress status
There are no upgrades in progress.
But with Apache Hadoop 2.1.0-beta I get the following:
[cloud@Impetus-942 bin]$ hadoop dfsadmin -upgradeProgress status
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Thanks,
-Nirmal
From: Smarty Juice [mailto:smarty.juice@gmail.com]
Sent: Tuesday, December 10, 2013 7:12 PM
To: user@hadoop.apache.org
Subject: Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>> wrote:
Hi All,
hadoop dfsadmin -help
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
hadoop dfsadmin is the command to execute DFS administrative commands.
The full syntax is:
hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanodehost:port blockpoolId [force]]
[-setBalancerBandwidth <bandwidth>]
[-fetchImage <local directory>]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-help [cmd]]
-report: Reports basic filesystem information and statistics.
-safemode <enter|leave|get|wait>: Safe mode maintenance command.
Safe mode is a Namenode state in which it
1. does not accept changes to the name space (read-only)
2. does not replicate or delete blocks.
Safe mode is entered automatically at Namenode startup, and
leaves safe mode automatically when the configured minimum
percentage of blocks satisfies the minimum replication
condition. Safe mode can also be entered manually, but then
it can only be turned off manually as well.
-saveNamespace: Save current namespace into storage directories and reset edits log.
Requires superuser permissions and safe mode.
-rollEdits: Rolls the edit log.
Requires superuser permissions.
-restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed storage replicas if they become available.
Requires superuser permissions.
-refreshNodes: Updates the namenode with the set of datanodes allowed to connect to the namenode.
Namenode re-reads datanode hostnames from the file defined by
dfs.hosts, dfs.hosts.exclude configuration parameters.
Hosts defined in dfs.hosts are the datanodes that are part of
the cluster. If there are entries in dfs.hosts, only the hosts
in it are allowed to register with the namenode.
Entries in dfs.hosts.exclude are datanodes that need to be
decommissioned. Datanodes complete decommissioning when
all the replicas from them are replicated to other datanodes.
Decommissioned nodes are not automatically shutdown and
are not chosen for writing new replicas.
-finalizeUpgrade: Finalize upgrade of HDFS.
Datanodes delete their previous version working directories,
followed by Namenode doing the same.
This completes the upgrade process.
-metasave <filename>: Save Namenode's primary data structures
to <filename> in the directory specified by hadoop.log.dir property.
<filename> is overwritten if it exists.
<filename> will contain one line for each of the following
1. Datanodes heart beating with Namenode
2. Blocks waiting to be replicated
3. Blocks currrently being replicated
4. Blocks waiting to be deleted
-setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each directory <dirName>.
The directory quota is a long integer that puts a hard limit
on the number of names in the directory tree
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. User is not an administrator, or
3. The directory does not exist or is a file.
Note: A quota of 1 would force the directory to remain empty.
-clrQuota <dirname>...<dirname>: Clear the quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota <quota> for each directory <dirName>.
The space quota is a long integer that puts a hard limit
on the total size of all the files under the directory tree.
The extra space required for replication is also counted. E.g.
a 1GB file with replication of 3 consumes 3GB of the quota.
Quota can also be speciefied with a binary prefix for terabytes,
petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
For each directory, attempt to set the quota. An error will be reported if
1. N is not a positive integer, or
2. user is not an administrator, or
3. the directory does not exist or is a file, or
-clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each directory <dirName>.
For each directory, attempt to clear the quota. An error will be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
-refreshServiceAcl: Reload the service-level authorization policy file
Namenode will reload the authorization policy file.
-refreshUserToGroupsMappings: Refresh user-to-groups mappings
-refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups mappings
-printTopology: Print a tree of the racks and their
nodes as reported by the Namenode
-refreshNamenodes: Takes a datanodehost:port as argument,
For the given datanode, reloads the configuration files,
stops serving the removed block-pools
and starts serving new block-pools
-deleteBlockPool: Arguments are datanodehost:port, blockpool id
and an optional argument "force". If force is passed,
block pool directory for the given blockpool id on the given
datanode is deleted along with its contents, otherwise
the directory is deleted only if it is empty. The command
will fail if datanode is still serving the block pool.
Refer to refreshNamenodes to shutdown a block pool
service on a datanode.
-setBalancerBandwidth <bandwidth>:
Changes the network bandwidth used by each datanode during
HDFS block balancing.
<bandwidth> is the maximum number of bytes per second
that will be used by each datanode. This value overrides
the dfs.balance.bandwidthPerSec parameter.
--- NOTE: The new value is not persistent on the DataNode.---
-fetchImage <local directory>:
Downloads the most recent fsimage from the Name Node and saves it in the specified local directory.
-allowSnapshot <snapshotDir>:
Allow snapshots to be taken on a directory.
-disallowSnapshot <snapshotDir>:
Do not allow snapshots to be taken on a directory any more.
-help [cmd]: Displays help for the given command or all commands if none
is specified.
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Even tried the hdfs command:
[cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
upgradeProgress: Unknown command
Usage: java DFSAdmin
Note: Administrative commands can only be run as the HDFS superuser.
[-report]
[-safemode enter | leave | get | wait]
[-allowSnapshot <snapshotDir>]
[-disallowSnapshot <snapshotDir>]
[-saveNamespace]
[-rollEdits]
[-restoreFailedStorage true|false|check]
[-refreshNodes]
[-finalizeUpgrade]
[-metasave filename]
[-refreshServiceAcl]
[-refreshUserToGroupsMappings]
[-refreshSuperUserGroupsConfiguration]
[-printTopology]
[-refreshNamenodes datanodehost:port]
[-deleteBlockPool datanode-host:port blockpoolId [force]]
[-setQuota <quota> <dirname>...<dirname>]
[-clrQuota <dirname>...<dirname>]
[-setSpaceQuota <quota> <dirname>...<dirname>]
[-clrSpaceQuota <dirname>...<dirname>]
[-setBalancerBandwidth <bandwidth in bytes per second>]
[-fetchImage <local directory>]
[-help [cmd]]
Generic options supported are
-conf <configuration file> specify an application configuration file
-D <property=value> use value for given property
-fs <local|namenode:port> specify a namenode
-jt <local|jobtracker:port> specify a job tracker
-files <comma separated list of files> specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars> specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives> specify comma separated archives to be unarchived on the compute machines.
The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]
Is there is some other option for checking the upgrade progress?
Thanks,
-Nirmal
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Posted by Smarty Juice <sm...@gmail.com>.
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> hadoop dfsadmin -help
>
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
>
> Instead use the hdfs command for it.
>
>
>
> hadoop dfsadmin is the command to execute DFS administrative commands.
>
> The full syntax is:
>
>
>
> hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanodehost:port blockpoolId [force]]
>
> [-setBalancerBandwidth <bandwidth>]
>
> [-fetchImage <local directory>]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-help [cmd]]
>
>
>
> -report: Reports basic filesystem information and statistics.
>
>
>
> -safemode <enter|leave|get|wait>: Safe mode maintenance command.
>
> Safe mode is a Namenode state in which it
>
> 1. does not accept changes to the name space
> (read-only)
>
> 2. does not replicate or delete blocks.
>
> Safe mode is entered automatically at Namenode startup, and
>
> leaves safe mode automatically when the configured minimum
>
> percentage of blocks satisfies the minimum replication
>
> condition. Safe mode can also be entered manually, but
> then
>
> it can only be turned off manually as well.
>
>
>
> -saveNamespace: Save current namespace into storage directories and reset
> edits log.
>
> Requires superuser permissions and safe mode.
>
>
>
> -rollEdits: Rolls the edit log.
>
> Requires superuser permissions.
>
>
>
> -restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed
> storage replicas if they become available.
>
> Requires superuser permissions.
>
>
>
> -refreshNodes: Updates the namenode with the set of datanodes allowed to
> connect to the namenode.
>
>
>
> Namenode re-reads datanode hostnames from the file defined
> by
>
> dfs.hosts, dfs.hosts.exclude configuration parameters.
>
> Hosts defined in dfs.hosts are the datanodes that are part
> of
>
> the cluster. If there are entries in dfs.hosts, only the
> hosts
>
> in it are allowed to register with the namenode.
>
>
>
> Entries in dfs.hosts.exclude are datanodes that need to be
>
> decommissioned. Datanodes complete decommissioning when
>
> all the replicas from them are replicated to other
> datanodes.
>
> Decommissioned nodes are not automatically shutdown and
>
> are not chosen for writing new replicas.
>
>
>
> -finalizeUpgrade: Finalize upgrade of HDFS.
>
> Datanodes delete their previous version working
> directories,
>
> followed by Namenode doing the same.
>
> This completes the upgrade process.
>
>
>
> -metasave <filename>: Save Namenode's primary data structures
>
> to <filename> in the directory specified by hadoop.log.dir
> property.
>
> <filename> is overwritten if it exists.
>
> <filename> will contain one line for each of the following
>
> 1. Datanodes heart beating with Namenode
>
> 2. Blocks waiting to be replicated
>
> 3. Blocks currrently being replicated
>
> 4. Blocks waiting to be deleted
>
>
>
> -setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each
> directory <dirName>.
>
> The directory quota is a long integer that puts a hard
> limit
>
> on the number of names in the directory tree
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. User is not an administrator, or
>
> 3. The directory does not exist or is a file.
>
> Note: A quota of 1 would force the directory to remain
> empty.
>
>
>
> -clrQuota <dirname>...<dirname>: Clear the quota for each directory
> <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota
> <quota> for each directory <dirName>.
>
> The space quota is a long integer that puts a hard limit
>
> on the total size of all the files under the directory
> tree.
>
> The extra space required for replication is also counted.
> E.g.
>
> a 1GB file with replication of 3 consumes 3GB of the quota.
>
>
>
> Quota can also be speciefied with a binary prefix for
> terabytes,
>
> petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. user is not an administrator, or
>
> 3. the directory does not exist or is a file, or
>
>
>
> -clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each
> directory <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -refreshServiceAcl: Reload the service-level authorization policy file
>
> Namenode will reload the authorization policy file.
>
>
>
> -refreshUserToGroupsMappings: Refresh user-to-groups mappings
>
>
>
> -refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>
>
>
> -printTopology: Print a tree of the racks and their
>
> nodes as reported by the Namenode
>
>
>
> -refreshNamenodes: Takes a datanodehost:port as argument,
>
> For the given datanode, reloads the configuration files,
>
> stops serving the removed block-pools
>
> and starts serving new block-pools
>
>
>
> -deleteBlockPool: Arguments are datanodehost:port, blockpool id
>
> and an optional argument "force". If force is passed,
>
> block pool directory for the given blockpool id on the
> given
>
> datanode is deleted along with its contents, otherwise
>
> the directory is deleted only if it is empty. The command
>
> will fail if datanode is still serving the block pool.
>
> Refer to refreshNamenodes to shutdown a block pool
>
> service on a datanode.
>
>
>
> -setBalancerBandwidth <bandwidth>:
>
> Changes the network bandwidth used by each datanode during
>
> HDFS block balancing.
>
>
>
> <bandwidth> is the maximum number of bytes per second
>
> that will be used by each datanode. This value overrides
>
> the dfs.balance.bandwidthPerSec parameter.
>
>
>
> --- NOTE: The new value is not persistent on the
> DataNode.---
>
>
>
> -fetchImage <local directory>:
>
> Downloads the most recent fsimage from the Name Node and saves it
> in the specified local directory.
>
>
>
> -allowSnapshot <snapshotDir>:
>
> Allow snapshots to be taken on a directory.
>
>
>
> -disallowSnapshot <snapshotDir>:
>
> Do not allow snapshots to be taken on a directory any more.
>
>
>
> -help [cmd]: Displays help for the given command or all commands if none
>
> is specified.
>
>
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
> Even tried the hdfs command:
>
> [cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
>
> upgradeProgress: Unknown command
>
> Usage: java DFSAdmin
>
> Note: Administrative commands can only be run as the HDFS superuser.
>
> [-report]
>
> [-safemode enter | leave | get | wait]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-finalizeUpgrade]
>
> [-metasave filename]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [-refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanode-host:port blockpoolId [force]]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-setBalancerBandwidth <bandwidth in bytes per second>]
>
> [-fetchImage <local directory>]
>
> [-help [cmd]]
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
>
>
> Is there is some other option for checking the upgrade progress?
>
>
>
> Thanks,
>
> -Nirmal
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Posted by Smarty Juice <sm...@gmail.com>.
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> hadoop dfsadmin -help
>
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
>
> Instead use the hdfs command for it.
>
>
>
> hadoop dfsadmin is the command to execute DFS administrative commands.
>
> The full syntax is:
>
>
>
> hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanodehost:port blockpoolId [force]]
>
> [-setBalancerBandwidth <bandwidth>]
>
> [-fetchImage <local directory>]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-help [cmd]]
>
>
>
> -report: Reports basic filesystem information and statistics.
>
>
>
> -safemode <enter|leave|get|wait>: Safe mode maintenance command.
>
> Safe mode is a Namenode state in which it
>
> 1. does not accept changes to the name space
> (read-only)
>
> 2. does not replicate or delete blocks.
>
> Safe mode is entered automatically at Namenode startup, and
>
> leaves safe mode automatically when the configured minimum
>
> percentage of blocks satisfies the minimum replication
>
> condition. Safe mode can also be entered manually, but
> then
>
> it can only be turned off manually as well.
>
>
>
> -saveNamespace: Save current namespace into storage directories and reset
> edits log.
>
> Requires superuser permissions and safe mode.
>
>
>
> -rollEdits: Rolls the edit log.
>
> Requires superuser permissions.
>
>
>
> -restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed
> storage replicas if they become available.
>
> Requires superuser permissions.
>
>
>
> -refreshNodes: Updates the namenode with the set of datanodes allowed to
> connect to the namenode.
>
>
>
> Namenode re-reads datanode hostnames from the file defined
> by
>
> dfs.hosts, dfs.hosts.exclude configuration parameters.
>
> Hosts defined in dfs.hosts are the datanodes that are part
> of
>
> the cluster. If there are entries in dfs.hosts, only the
> hosts
>
> in it are allowed to register with the namenode.
>
>
>
> Entries in dfs.hosts.exclude are datanodes that need to be
>
> decommissioned. Datanodes complete decommissioning when
>
> all the replicas from them are replicated to other
> datanodes.
>
> Decommissioned nodes are not automatically shutdown and
>
> are not chosen for writing new replicas.
>
>
>
> -finalizeUpgrade: Finalize upgrade of HDFS.
>
> Datanodes delete their previous version working
> directories,
>
> followed by Namenode doing the same.
>
> This completes the upgrade process.
>
>
>
> -metasave <filename>: Save Namenode's primary data structures
>
> to <filename> in the directory specified by hadoop.log.dir
> property.
>
> <filename> is overwritten if it exists.
>
> <filename> will contain one line for each of the following
>
> 1. Datanodes heart beating with Namenode
>
> 2. Blocks waiting to be replicated
>
> 3. Blocks currrently being replicated
>
> 4. Blocks waiting to be deleted
>
>
>
> -setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each
> directory <dirName>.
>
> The directory quota is a long integer that puts a hard
> limit
>
> on the number of names in the directory tree
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. User is not an administrator, or
>
> 3. The directory does not exist or is a file.
>
> Note: A quota of 1 would force the directory to remain
> empty.
>
>
>
> -clrQuota <dirname>...<dirname>: Clear the quota for each directory
> <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota
> <quota> for each directory <dirName>.
>
> The space quota is a long integer that puts a hard limit
>
> on the total size of all the files under the directory
> tree.
>
> The extra space required for replication is also counted.
> E.g.
>
> a 1GB file with replication of 3 consumes 3GB of the quota.
>
>
>
> Quota can also be speciefied with a binary prefix for
> terabytes,
>
> petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. user is not an administrator, or
>
> 3. the directory does not exist or is a file, or
>
>
>
> -clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each
> directory <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -refreshServiceAcl: Reload the service-level authorization policy file
>
> Namenode will reload the authorization policy file.
>
>
>
> -refreshUserToGroupsMappings: Refresh user-to-groups mappings
>
>
>
> -refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>
>
>
> -printTopology: Print a tree of the racks and their
>
> nodes as reported by the Namenode
>
>
>
> -refreshNamenodes: Takes a datanodehost:port as argument,
>
> For the given datanode, reloads the configuration files,
>
> stops serving the removed block-pools
>
> and starts serving new block-pools
>
>
>
> -deleteBlockPool: Arguments are datanodehost:port, blockpool id
>
> and an optional argument "force". If force is passed,
>
> block pool directory for the given blockpool id on the
> given
>
> datanode is deleted along with its contents, otherwise
>
> the directory is deleted only if it is empty. The command
>
> will fail if datanode is still serving the block pool.
>
> Refer to refreshNamenodes to shutdown a block pool
>
> service on a datanode.
>
>
>
> -setBalancerBandwidth <bandwidth>:
>
> Changes the network bandwidth used by each datanode during
>
> HDFS block balancing.
>
>
>
> <bandwidth> is the maximum number of bytes per second
>
> that will be used by each datanode. This value overrides
>
> the dfs.balance.bandwidthPerSec parameter.
>
>
>
> --- NOTE: The new value is not persistent on the
> DataNode.---
>
>
>
> -fetchImage <local directory>:
>
> Downloads the most recent fsimage from the Name Node and saves it
> in the specified local directory.
>
>
>
> -allowSnapshot <snapshotDir>:
>
> Allow snapshots to be taken on a directory.
>
>
>
> -disallowSnapshot <snapshotDir>:
>
> Do not allow snapshots to be taken on a directory any more.
>
>
>
> -help [cmd]: Displays help for the given command or all commands if none
>
> is specified.
>
>
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
> Even tried the hdfs command:
>
> [cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
>
> upgradeProgress: Unknown command
>
> Usage: java DFSAdmin
>
> Note: Administrative commands can only be run as the HDFS superuser.
>
> [-report]
>
> [-safemode enter | leave | get | wait]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-finalizeUpgrade]
>
> [-metasave filename]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [-refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanode-host:port blockpoolId [force]]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-setBalancerBandwidth <bandwidth in bytes per second>]
>
> [-fetchImage <local directory>]
>
> [-help [cmd]]
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
>
>
> Is there is some other option for checking the upgrade progress?
>
>
>
> Thanks,
>
> -Nirmal
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Posted by Smarty Juice <sm...@gmail.com>.
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> hadoop dfsadmin -help
>
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
>
> Instead use the hdfs command for it.
>
>
>
> hadoop dfsadmin is the command to execute DFS administrative commands.
>
> The full syntax is:
>
>
>
> hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanodehost:port blockpoolId [force]]
>
> [-setBalancerBandwidth <bandwidth>]
>
> [-fetchImage <local directory>]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-help [cmd]]
>
>
>
> -report: Reports basic filesystem information and statistics.
>
>
>
> -safemode <enter|leave|get|wait>: Safe mode maintenance command.
>
> Safe mode is a Namenode state in which it
>
> 1. does not accept changes to the name space
> (read-only)
>
> 2. does not replicate or delete blocks.
>
> Safe mode is entered automatically at Namenode startup, and
>
> leaves safe mode automatically when the configured minimum
>
> percentage of blocks satisfies the minimum replication
>
> condition. Safe mode can also be entered manually, but
> then
>
> it can only be turned off manually as well.
>
>
>
> -saveNamespace: Save current namespace into storage directories and reset
> edits log.
>
> Requires superuser permissions and safe mode.
>
>
>
> -rollEdits: Rolls the edit log.
>
> Requires superuser permissions.
>
>
>
> -restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed
> storage replicas if they become available.
>
> Requires superuser permissions.
>
>
>
> -refreshNodes: Updates the namenode with the set of datanodes allowed to
> connect to the namenode.
>
>
>
> Namenode re-reads datanode hostnames from the file defined
> by
>
> dfs.hosts, dfs.hosts.exclude configuration parameters.
>
> Hosts defined in dfs.hosts are the datanodes that are part
> of
>
> the cluster. If there are entries in dfs.hosts, only the
> hosts
>
> in it are allowed to register with the namenode.
>
>
>
> Entries in dfs.hosts.exclude are datanodes that need to be
>
> decommissioned. Datanodes complete decommissioning when
>
> all the replicas from them are replicated to other
> datanodes.
>
> Decommissioned nodes are not automatically shutdown and
>
> are not chosen for writing new replicas.
>
>
>
> -finalizeUpgrade: Finalize upgrade of HDFS.
>
> Datanodes delete their previous version working
> directories,
>
> followed by Namenode doing the same.
>
> This completes the upgrade process.
>
>
>
> -metasave <filename>: Save Namenode's primary data structures
>
> to <filename> in the directory specified by hadoop.log.dir
> property.
>
> <filename> is overwritten if it exists.
>
> <filename> will contain one line for each of the following
>
> 1. Datanodes heart beating with Namenode
>
> 2. Blocks waiting to be replicated
>
> 3. Blocks currrently being replicated
>
> 4. Blocks waiting to be deleted
>
>
>
> -setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each
> directory <dirName>.
>
> The directory quota is a long integer that puts a hard
> limit
>
> on the number of names in the directory tree
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. User is not an administrator, or
>
> 3. The directory does not exist or is a file.
>
> Note: A quota of 1 would force the directory to remain
> empty.
>
>
>
> -clrQuota <dirname>...<dirname>: Clear the quota for each directory
> <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota
> <quota> for each directory <dirName>.
>
> The space quota is a long integer that puts a hard limit
>
> on the total size of all the files under the directory
> tree.
>
> The extra space required for replication is also counted.
> E.g.
>
> a 1GB file with replication of 3 consumes 3GB of the quota.
>
>
>
> Quota can also be speciefied with a binary prefix for
> terabytes,
>
> petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. user is not an administrator, or
>
> 3. the directory does not exist or is a file, or
>
>
>
> -clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each
> directory <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -refreshServiceAcl: Reload the service-level authorization policy file
>
> Namenode will reload the authorization policy file.
>
>
>
> -refreshUserToGroupsMappings: Refresh user-to-groups mappings
>
>
>
> -refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>
>
>
> -printTopology: Print a tree of the racks and their
>
> nodes as reported by the Namenode
>
>
>
> -refreshNamenodes: Takes a datanodehost:port as argument,
>
> For the given datanode, reloads the configuration files,
>
> stops serving the removed block-pools
>
> and starts serving new block-pools
>
>
>
> -deleteBlockPool: Arguments are datanodehost:port, blockpool id
>
> and an optional argument "force". If force is passed,
>
> block pool directory for the given blockpool id on the
> given
>
> datanode is deleted along with its contents, otherwise
>
> the directory is deleted only if it is empty. The command
>
> will fail if datanode is still serving the block pool.
>
> Refer to refreshNamenodes to shutdown a block pool
>
> service on a datanode.
>
>
>
> -setBalancerBandwidth <bandwidth>:
>
> Changes the network bandwidth used by each datanode during
>
> HDFS block balancing.
>
>
>
> <bandwidth> is the maximum number of bytes per second
>
> that will be used by each datanode. This value overrides
>
> the dfs.balance.bandwidthPerSec parameter.
>
>
>
> --- NOTE: The new value is not persistent on the
> DataNode.---
>
>
>
> -fetchImage <local directory>:
>
> Downloads the most recent fsimage from the Name Node and saves it
> in the specified local directory.
>
>
>
> -allowSnapshot <snapshotDir>:
>
> Allow snapshots to be taken on a directory.
>
>
>
> -disallowSnapshot <snapshotDir>:
>
> Do not allow snapshots to be taken on a directory any more.
>
>
>
> -help [cmd]: Displays help for the given command or all commands if none
>
> is specified.
>
>
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
> Even tried the hdfs command:
>
> [cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
>
> upgradeProgress: Unknown command
>
> Usage: java DFSAdmin
>
> Note: Administrative commands can only be run as the HDFS superuser.
>
> [-report]
>
> [-safemode enter | leave | get | wait]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-finalizeUpgrade]
>
> [-metasave filename]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [-refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanode-host:port blockpoolId [force]]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-setBalancerBandwidth <bandwidth in bytes per second>]
>
> [-fetchImage <local directory>]
>
> [-help [cmd]]
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
>
>
> Is there is some other option for checking the upgrade progress?
>
>
>
> Thanks,
>
> -Nirmal
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
Re: Unable to run dfsadmin -upgradeProgress status in Apache Hadoop 2.1.0-beta
Posted by Smarty Juice <sm...@gmail.com>.
Can you please try below instead? let me know if it works
hadoop dfsadmin -upgradeProgress status
On Tue, Dec 10, 2013 at 6:00 AM, Nirmal Kumar <ni...@impetus.co.in>wrote:
> Hi All,
>
>
>
> hadoop dfsadmin -help
>
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
>
> Instead use the hdfs command for it.
>
>
>
> hadoop dfsadmin is the command to execute DFS administrative commands.
>
> The full syntax is:
>
>
>
> hadoop dfsadmin [-report] [-safemode <enter | leave | get | wait>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanodehost:port blockpoolId [force]]
>
> [-setBalancerBandwidth <bandwidth>]
>
> [-fetchImage <local directory>]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-help [cmd]]
>
>
>
> -report: Reports basic filesystem information and statistics.
>
>
>
> -safemode <enter|leave|get|wait>: Safe mode maintenance command.
>
> Safe mode is a Namenode state in which it
>
> 1. does not accept changes to the name space
> (read-only)
>
> 2. does not replicate or delete blocks.
>
> Safe mode is entered automatically at Namenode startup, and
>
> leaves safe mode automatically when the configured minimum
>
> percentage of blocks satisfies the minimum replication
>
> condition. Safe mode can also be entered manually, but
> then
>
> it can only be turned off manually as well.
>
>
>
> -saveNamespace: Save current namespace into storage directories and reset
> edits log.
>
> Requires superuser permissions and safe mode.
>
>
>
> -rollEdits: Rolls the edit log.
>
> Requires superuser permissions.
>
>
>
> -restoreFailedStorage: Set/Unset/Check flag to attempt restore of failed
> storage replicas if they become available.
>
> Requires superuser permissions.
>
>
>
> -refreshNodes: Updates the namenode with the set of datanodes allowed to
> connect to the namenode.
>
>
>
> Namenode re-reads datanode hostnames from the file defined
> by
>
> dfs.hosts, dfs.hosts.exclude configuration parameters.
>
> Hosts defined in dfs.hosts are the datanodes that are part
> of
>
> the cluster. If there are entries in dfs.hosts, only the
> hosts
>
> in it are allowed to register with the namenode.
>
>
>
> Entries in dfs.hosts.exclude are datanodes that need to be
>
> decommissioned. Datanodes complete decommissioning when
>
> all the replicas from them are replicated to other
> datanodes.
>
> Decommissioned nodes are not automatically shutdown and
>
> are not chosen for writing new replicas.
>
>
>
> -finalizeUpgrade: Finalize upgrade of HDFS.
>
> Datanodes delete their previous version working
> directories,
>
> followed by Namenode doing the same.
>
> This completes the upgrade process.
>
>
>
> -metasave <filename>: Save Namenode's primary data structures
>
> to <filename> in the directory specified by hadoop.log.dir
> property.
>
> <filename> is overwritten if it exists.
>
> <filename> will contain one line for each of the following
>
> 1. Datanodes heart beating with Namenode
>
> 2. Blocks waiting to be replicated
>
> 3. Blocks currrently being replicated
>
> 4. Blocks waiting to be deleted
>
>
>
> -setQuota <quota> <dirname>...<dirname>: Set the quota <quota> for each
> directory <dirName>.
>
> The directory quota is a long integer that puts a hard
> limit
>
> on the number of names in the directory tree
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. User is not an administrator, or
>
> 3. The directory does not exist or is a file.
>
> Note: A quota of 1 would force the directory to remain
> empty.
>
>
>
> -clrQuota <dirname>...<dirname>: Clear the quota for each directory
> <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -setSpaceQuota <quota> <dirname>...<dirname>: Set the disk space quota
> <quota> for each directory <dirName>.
>
> The space quota is a long integer that puts a hard limit
>
> on the total size of all the files under the directory
> tree.
>
> The extra space required for replication is also counted.
> E.g.
>
> a 1GB file with replication of 3 consumes 3GB of the quota.
>
>
>
> Quota can also be speciefied with a binary prefix for
> terabytes,
>
> petabytes etc (e.g. 50t is 50TB, 5m is 5MB, 3p is 3PB).
>
> For each directory, attempt to set the quota. An error
> will be reported if
>
> 1. N is not a positive integer, or
>
> 2. user is not an administrator, or
>
> 3. the directory does not exist or is a file, or
>
>
>
> -clrSpaceQuota <dirname>...<dirname>: Clear the disk space quota for each
> directory <dirName>.
>
> For each directory, attempt to clear the quota. An error
> will be reported if
>
> 1. the directory does not exist or is a file, or
>
> 2. user is not an administrator.
>
> It does not fault if the directory has no quota.
>
> -refreshServiceAcl: Reload the service-level authorization policy file
>
> Namenode will reload the authorization policy file.
>
>
>
> -refreshUserToGroupsMappings: Refresh user-to-groups mappings
>
>
>
> -refreshSuperUserGroupsConfiguration: Refresh superuser proxy groups
> mappings
>
>
>
> -printTopology: Print a tree of the racks and their
>
> nodes as reported by the Namenode
>
>
>
> -refreshNamenodes: Takes a datanodehost:port as argument,
>
> For the given datanode, reloads the configuration files,
>
> stops serving the removed block-pools
>
> and starts serving new block-pools
>
>
>
> -deleteBlockPool: Arguments are datanodehost:port, blockpool id
>
> and an optional argument "force". If force is passed,
>
> block pool directory for the given blockpool id on the
> given
>
> datanode is deleted along with its contents, otherwise
>
> the directory is deleted only if it is empty. The command
>
> will fail if datanode is still serving the block pool.
>
> Refer to refreshNamenodes to shutdown a block pool
>
> service on a datanode.
>
>
>
> -setBalancerBandwidth <bandwidth>:
>
> Changes the network bandwidth used by each datanode during
>
> HDFS block balancing.
>
>
>
> <bandwidth> is the maximum number of bytes per second
>
> that will be used by each datanode. This value overrides
>
> the dfs.balance.bandwidthPerSec parameter.
>
>
>
> --- NOTE: The new value is not persistent on the
> DataNode.---
>
>
>
> -fetchImage <local directory>:
>
> Downloads the most recent fsimage from the Name Node and saves it
> in the specified local directory.
>
>
>
> -allowSnapshot <snapshotDir>:
>
> Allow snapshots to be taken on a directory.
>
>
>
> -disallowSnapshot <snapshotDir>:
>
> Do not allow snapshots to be taken on a directory any more.
>
>
>
> -help [cmd]: Displays help for the given command or all commands if none
>
> is specified.
>
>
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
> Even tried the hdfs command:
>
> [cloud@impetus-D912 bin]$ ./hdfs dfsadmin -upgradeProgress status
>
> upgradeProgress: Unknown command
>
> Usage: java DFSAdmin
>
> Note: Administrative commands can only be run as the HDFS superuser.
>
> [-report]
>
> [-safemode enter | leave | get | wait]
>
> [-allowSnapshot <snapshotDir>]
>
> [-disallowSnapshot <snapshotDir>]
>
> [-saveNamespace]
>
> [-rollEdits]
>
> [-restoreFailedStorage true|false|check]
>
> [-refreshNodes]
>
> [-finalizeUpgrade]
>
> [-metasave filename]
>
> [-refreshServiceAcl]
>
> [-refreshUserToGroupsMappings]
>
> [-refreshSuperUserGroupsConfiguration]
>
> [-printTopology]
>
> [-refreshNamenodes datanodehost:port]
>
> [-deleteBlockPool datanode-host:port blockpoolId [force]]
>
> [-setQuota <quota> <dirname>...<dirname>]
>
> [-clrQuota <dirname>...<dirname>]
>
> [-setSpaceQuota <quota> <dirname>...<dirname>]
>
> [-clrSpaceQuota <dirname>...<dirname>]
>
> [-setBalancerBandwidth <bandwidth in bytes per second>]
>
> [-fetchImage <local directory>]
>
> [-help [cmd]]
>
>
>
> Generic options supported are
>
> -conf <configuration file> specify an application configuration file
>
> -D <property=value> use value for given property
>
> -fs <local|namenode:port> specify a namenode
>
> -jt <local|jobtracker:port> specify a job tracker
>
> -files <comma separated list of files> specify comma separated files to
> be copied to the map reduce cluster
>
> -libjars <comma separated list of jars> specify comma separated jar
> files to include in the classpath.
>
> -archives <comma separated list of archives> specify comma separated
> archives to be unarchived on the compute machines.
>
>
>
> The general command line syntax is
>
> bin/hadoop command [genericOptions] [commandOptions]
>
>
>
>
>
> Is there is some other option for checking the upgrade progress?
>
>
>
> Thanks,
>
> -Nirmal
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>