You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Gabriel James <ga...@heliase.com> on 2017/04/20 03:51:11 UTC
Hadoop namespace format user and permissions
Hi All,
First time installation, not an expert, think probably a permissions issue,
not sure best way to resolve, forgive basic questions.
Trying to setup Hadoop 2.7.3 as single node on Ubuntu 16.04 with Java Open
JDK 8. Then will setup a three node cluster.
1. I think the directory paths are correct in hdfs-site.xml
2. Not sure what user namenode uses?
3. Not sure what permissions namenode needs?
4. Not sure what user and group configuration for running as service?
The error below seems to common on the web, but couldn't find a clear
answer, the Hadoop documentation seems quite minimal.
/usr/local/hadoop/bin/hdfs namenode -format
17/04/20 03:27:34 WARN namenode.NameNode: Encountered exception during
format:
java.io.IOException: Cannot create directory /res0/hdfs/name/current
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory
(Storage.java:337)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:564)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:585)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:161)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:992)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
:1434)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
17/04/20 03:27:34 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot create directory /res0/hdfs/name/current
at
org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory
(Storage.java:337)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:564)
at
org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:585)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:161)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:992)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java
:1434)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
17/04/20 03:27:34 INFO util.ExitUtil: Exiting with status 1
17/04/20 03:27:34 INFO namenode.NameNode: SHUTDOWN_MSG:
/usr/local/hadoop/etc/hadoop/Hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
/usr/local/hadoop/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///res0/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///res0/hdfs/data</value>
</property>
</configuration>
/res0/hdfs$ ls -la
drwxr-xr-x 4 root root 4096 Apr 19 05:19 .
drwxr-xr-x 4 root root 4096 Apr 19 05:18 ..
drwxr-xr-x 2 root root 4096 Apr 19 05:18 data
drwxr-xr-x 2 root root 4096 Apr 20 03:27 name
Thanks in advance,
Gabe
RE: Hadoop namespace format user and permissions
Posted by Sidharth Kumar <si...@gmail.com>.
Hi James,
Great to here that your issue is resolved . Please find attached
installation document
Sidharth
Mob: +91 8197555599
LinkedIn: www.linkedin.com/in/sidharthkumar2792
On 20-Apr-2017 11:04 PM, "Gabriel James" <ga...@heliase.com> wrote:
Hi Sidharth,
Thank you for your help. Yes, it was simply the write permission on
directory.
I would be grateful if you could share your document on pseudo mode
installation. Would you also have one for cluster installation?
Kind regards,
Gabriel
*From:* Sidharth Kumar [mailto:sidharthkumar2707@gmail.com]
*Sent:* Friday, April 21, 2017 4:45 AM
*To:* gabriel.james@heliase.com
*Cc:* common-user@hadoop.apache.org <us...@hadoop.apache.org>
*Subject:* Re: Hadoop namespace format user and permissions
Hi James,
Please create a user hadoop or hdfs and change the ownership of directory
to hdfs:hadoop. Hdfs run with hdfs user. This should probably resolve your
issue. If you need I can share document which i made for pseudo mode
installation to help my mates.
Please let me know if issue still persists.
Let's hadooping.....!
Best regards
Sidharth
Mob: +91 8197555599
LinkedIn: www.linkedin.com/in/sidharthkumar2792
On 20-Apr-2017 9:21 AM, "Gabriel James" <ga...@heliase.com> wrote:
Hi All,
First time installation, not an expert, think probably a permissions issue,
not sure best way to resolve, forgive basic questions.
Trying to setup Hadoop 2.7.3 as single node on Ubuntu 16.04 with Java Open
JDK 8. Then will setup a three node cluster.
1. I think the directory paths are correct in hdfs-site.xml
2. Not sure what user namenode uses?
3. Not sure what permissions namenode needs?
4. Not sure what user and group configuration for running as service?
The error below seems to common on the web, but couldn’t find a clear
answer, the Hadoop documentation seems quite minimal.
*/usr/local/hadoop/bin/hdfs namenode -format*
17/04/20 03:27:34 WARN namenode.NameNode: Encountered exception during
format:
java.io.IOException: Cannot create directory /res0/hdfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.
clearDirectory(Storage.java:337)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
NNStorage.java:564)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
NNStorage.java:585)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(
FSImage.java:161)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(
NameNode.java:992)
at org.apache.hadoop.hdfs.server.namenode.NameNode.
createNameNode(NameNode.java:1434)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(
NameNode.java:1559)
17/04/20 03:27:34 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot create directory /res0/hdfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.
clearDirectory(Storage.java:337)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
NNStorage.java:564)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
NNStorage.java:585)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(
FSImage.java:161)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(
NameNode.java:992)
at org.apache.hadoop.hdfs.server.namenode.NameNode.
createNameNode(NameNode.java:1434)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(
NameNode.java:1559)
17/04/20 03:27:34 INFO util.ExitUtil: Exiting with status 1
17/04/20 03:27:34 INFO namenode.NameNode: SHUTDOWN_MSG:
*/usr/local/hadoop/etc/hadoop/Hadoop-env.sh*
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
*/usr/local/hadoop/etc/hadoop/hdfs-site.xml*
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///res0/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///res0/hdfs/data</value>
</property>
</configuration>
*/res0/hdfs$ ls -la*
drwxr-xr-x 4 root root 4096 Apr 19 05:19 .
drwxr-xr-x 4 root root 4096 Apr 19 05:18 ..
drwxr-xr-x 2 root root 4096 Apr 19 05:18 data
drwxr-xr-x 2 root root 4096 Apr 20 03:27 name
Thanks in advance,
Gabe
RE: Hadoop namespace format user and permissions
Posted by Gabriel James <ga...@heliase.com>.
Hi Sidharth,
Thank you for your help. Yes, it was simply the write permission on directory.
I would be grateful if you could share your document on pseudo mode installation. Would you also have one for cluster installation?
Kind regards,
Gabriel
From: Sidharth Kumar [mailto:sidharthkumar2707@gmail.com]
Sent: Friday, April 21, 2017 4:45 AM
To: gabriel.james@heliase.com
Cc: common-user@hadoop.apache.org <us...@hadoop.apache.org>
Subject: Re: Hadoop namespace format user and permissions
Hi James,
Please create a user hadoop or hdfs and change the ownership of directory to hdfs:hadoop. Hdfs run with hdfs user. This should probably resolve your issue. If you need I can share document which i made for pseudo mode installation to help my mates.
Please let me know if issue still persists.
Let's hadooping.....!
Best regards
Sidharth
Mob: +91 8197555599
LinkedIn: www.linkedin.com/in/sidharthkumar2792 <http://www.linkedin.com/in/sidharthkumar2792>
On 20-Apr-2017 9:21 AM, "Gabriel James" <gabriel.james@heliase.com <ma...@heliase.com> > wrote:
Hi All,
First time installation, not an expert, think probably a permissions issue, not sure best way to resolve, forgive basic questions.
Trying to setup Hadoop 2.7.3 as single node on Ubuntu 16.04 with Java Open JDK 8. Then will setup a three node cluster.
1. I think the directory paths are correct in hdfs-site.xml
2. Not sure what user namenode uses?
3. Not sure what permissions namenode needs?
4. Not sure what user and group configuration for running as service?
The error below seems to common on the web, but couldn’t find a clear answer, the Hadoop documentation seems quite minimal.
/usr/local/hadoop/bin/hdfs namenode -format
17/04/20 03:27:34 WARN namenode.NameNode: Encountered exception during format:
java.io.IOException: Cannot create directory /res0/hdfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:337)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:564)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:585)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:161)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:992)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1434)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
17/04/20 03:27:34 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot create directory /res0/hdfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:337)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:564)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:585)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:161)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:992)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1434)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1559)
17/04/20 03:27:34 INFO util.ExitUtil: Exiting with status 1
17/04/20 03:27:34 INFO namenode.NameNode: SHUTDOWN_MSG:
/usr/local/hadoop/etc/hadoop/Hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
/usr/local/hadoop/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///res0/hdfs/name <file:///res0/hdfs/name%3c/value> </value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///res0/hdfs/data <file:///res0/hdfs/data%3c/value> </value>
</property>
</configuration>
/res0/hdfs$ ls -la
drwxr-xr-x 4 root root 4096 Apr 19 05:19 .
drwxr-xr-x 4 root root 4096 Apr 19 05:18 ..
drwxr-xr-x 2 root root 4096 Apr 19 05:18 data
drwxr-xr-x 2 root root 4096 Apr 20 03:27 name
Thanks in advance,
Gabe
Re: Hadoop namespace format user and permissions
Posted by Sidharth Kumar <si...@gmail.com>.
Hi James,
Please create a user hadoop or hdfs and change the ownership of directory
to hdfs:hadoop. Hdfs run with hdfs user. This should probably resolve your
issue. If you need I can share document which i made for pseudo mode
installation to help my mates.
Please let me know if issue still persists.
Let's hadooping.....!
Best regards
Sidharth
Mob: +91 8197555599
LinkedIn: www.linkedin.com/in/sidharthkumar2792
On 20-Apr-2017 9:21 AM, "Gabriel James" <ga...@heliase.com> wrote:
> Hi All,
>
>
>
> First time installation, not an expert, think probably a permissions
> issue, not sure best way to resolve, forgive basic questions.
>
>
>
> Trying to setup Hadoop 2.7.3 as single node on Ubuntu 16.04 with Java Open
> JDK 8. Then will setup a three node cluster.
>
>
>
> 1. I think the directory paths are correct in hdfs-site.xml
> 2. Not sure what user namenode uses?
> 3. Not sure what permissions namenode needs?
> 4. Not sure what user and group configuration for running as service?
>
>
>
> The error below seems to common on the web, but couldn’t find a clear
> answer, the Hadoop documentation seems quite minimal.
>
>
>
> */usr/local/hadoop/bin/hdfs namenode -format*
>
>
>
> 17/04/20 03:27:34 WARN namenode.NameNode: Encountered exception during
> format:
>
> java.io.IOException: Cannot create directory /res0/hdfs/name/current
>
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.
> clearDirectory(Storage.java:337)
>
> at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
> NNStorage.java:564)
>
> at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
> NNStorage.java:585)
>
> at org.apache.hadoop.hdfs.server.namenode.FSImage.format(
> FSImage.java:161)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.format(
> NameNode.java:992)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.
> createNameNode(NameNode.java:1434)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(
> NameNode.java:1559)
>
> 17/04/20 03:27:34 ERROR namenode.NameNode: Failed to start namenode.
>
> java.io.IOException: Cannot create directory /res0/hdfs/name/current
>
> at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.
> clearDirectory(Storage.java:337)
>
> at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
> NNStorage.java:564)
>
> at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(
> NNStorage.java:585)
>
> at org.apache.hadoop.hdfs.server.namenode.FSImage.format(
> FSImage.java:161)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.format(
> NameNode.java:992)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.
> createNameNode(NameNode.java:1434)
>
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(
> NameNode.java:1559)
>
> 17/04/20 03:27:34 INFO util.ExitUtil: Exiting with status 1
>
> 17/04/20 03:27:34 INFO namenode.NameNode: SHUTDOWN_MSG:
>
>
>
> */usr/local/hadoop/etc/hadoop/Hadoop-env.sh*
>
>
>
> export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/
>
>
>
> */usr/local/hadoop/etc/hadoop/hdfs-site.xml*
>
>
>
> <?xml version="1.0" encoding="UTF-8"?>
>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
>
>
> <configuration>
>
> <property>
>
> <name>dfs.replication</name>
>
> <value>1</value>
>
> </property>
>
> <property>
>
> <name>dfs.namenode.name.dir</name>
>
> <value>file:///res0/hdfs/name</value>
>
> </property>
>
> <property>
>
> <name>dfs.datanode.data.dir</name>
>
> <value>file:///res0/hdfs/data</value>
>
> </property>
>
> </configuration>
>
>
>
> */res0/hdfs$ ls -la*
>
>
>
> drwxr-xr-x 4 root root 4096 Apr 19 05:19 .
>
> drwxr-xr-x 4 root root 4096 Apr 19 05:18 ..
>
> drwxr-xr-x 2 root root 4096 Apr 19 05:18 data
>
> drwxr-xr-x 2 root root 4096 Apr 20 03:27 name
>
>
>
> Thanks in advance,
>
>
>
> Gabe
>