You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Pontus Amberg <po...@gmail.com> on 2019/02/12 17:57:52 UTC
Using Ozone as the default file system
I'm playing around with a small Hadoop 3.1.1 cluster and I have applied
Ozone 0.3.0-alpha to all the nodes. The
hadoop-ozone-datanode-plugin-0.3.0-alpha.jar and
hadoop-ozone-filesystem-0.3.0-alpha.jar have been added to the classpath
and I can start and init SCM, OM and create volumes, buckets and add new
keys with values. But when I try to use Ozone as the default file system (
as described here
https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html ) starting
the name nodes fails with the following message:
2019-02-12 17:32:48,467 ERROR
org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check
fs.defaultFS): o3fs://internal.address:9864/volume/bucket is not of scheme
'hdfs'.
I have also tried Hadoop 3.2.0 and I encountered the same error message.
Anyone have a clue what the cause might be?
/Pontus
Re: Using Ozone as the default file system
Posted by Pontus Amberg <po...@gmail.com>.
I'm not sure but it looks as if when "dfs.namenode.rpc-address" is missing
from hdfs-site.xml Hadoop will try deduce the address of the name node from
"fs.defaultFS". That logic seems to take for granted that the
"fs.defaultFS" must start with "hdfs".
Specifying "dfs.namenode.rpc-address" in hdfs-site.xml seems to have fixed
my first problem. Using "o3fs://bucket.volume"
seems to work. Thanks for the help!
/Pontus
On Tue, Feb 12, 2019 at 9:04 PM Bharat Viswanadham <
bviswanadham@hortonworks.com> wrote:
> Hi,
> Could you give a try by setting it as below.
>
> It should be set as o3fs://bucket.volume
>
> <property>
> <name>fs.o3fs.impl</name>
> <value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value>
> </property>
> <property>
> <name>fs.defaultFS</name>
> <value>o3fs://bucket.volume</value>
> </property>
>
> This bug is fixed as part of
> https://issues.apache.org/jira/browse/HDDS-913 which will be released as
> part of 0.4.0
>
>
>
> Thanks,
> Bharat
>
>
> From: Pontus Amberg <po...@gmail.com>
> Date: Tuesday, February 12, 2019 at 9:58 AM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Using Ozone as the default file system
>
> I'm playing around with a small Hadoop 3.1.1 cluster and I have applied
> Ozone 0.3.0-alpha to all the nodes. The
> hadoop-ozone-datanode-plugin-0.3.0-alpha.jar and
> hadoop-ozone-filesystem-0.3.0-alpha.jar have been added to the classpath
> and I can start and init SCM, OM and create volumes, buckets and add new
> keys with values. But when I try to use Ozone as the default file system (
> as described here
> https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html ) starting
> the name nodes fails with the following message:
>
> 2019-02-12 17:32:48,467 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> java.lang.IllegalArgumentException: Invalid URI for NameNode address
> (check fs.defaultFS): o3fs://internal.address:9864/volume/bucket is not of
> scheme 'hdfs'.
>
> I have also tried Hadoop 3.2.0 and I encountered the same error message.
>
> Anyone have a clue what the cause might be?
>
> /Pontus
>
>
Re: Using Ozone as the default file system
Posted by Bharat Viswanadham <bv...@hortonworks.com>.
Hi,
Could you give a try by setting it as below.
It should be set as o3fs://bucket.volume
<property>
<name>fs.o3fs.impl</name>
<value>org.apache.hadoop.fs.ozone.OzoneFileSystem</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>o3fs://bucket.volume</value>
</property>
This bug is fixed as part of https://issues.apache.org/jira/browse/HDDS-913 which will be released as part of 0.4.0
Thanks,
Bharat
From: Pontus Amberg <po...@gmail.com>
Date: Tuesday, February 12, 2019 at 9:58 AM
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
Subject: Using Ozone as the default file system
I'm playing around with a small Hadoop 3.1.1 cluster and I have applied Ozone 0.3.0-alpha to all the nodes. The hadoop-ozone-datanode-plugin-0.3.0-alpha.jar and hadoop-ozone-filesystem-0.3.0-alpha.jar have been added to the classpath and I can start and init SCM, OM and create volumes, buckets and add new keys with values. But when I try to use Ozone as the default file system ( as described here https://hadoop.apache.org/ozone/docs/0.3.0-alpha/ozonefs.html ) starting the name nodes fails with the following message:
2019-02-12 17:32:48,467 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): o3fs://internal.address:9864/volume/bucket is not of scheme 'hdfs'.
I have also tried Hadoop 3.2.0 and I encountered the same error message.
Anyone have a clue what the cause might be?
/Pontus