You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by xeonmailinglist <xe...@gmail.com> on 2015/03/03 11:32:52 UTC

Questions about HDFS federation

Hi,

1. The image below shows 2 YARN  instances using the same federation of 
clouds. Is this possible?

2. I have configured in 2 hosts (hadooop-coc-1, and hadoop-coc-2) a 
federation of HDFS. In the configuration, I have set a namespace in each 
host, and a single data node (see image). The service is running 
properly. You can check the output of the |jps| commands in [1].

The strange part is that, when I list the files, they do not appear in 
hadoop-coc-2. You can check the output in [2]. Why this happens?


hdfsfederation

[1]: jps output

|xubuntu@hadoop-coc-1:~/Programs/hadoop$ jps
21538 NameNode
21773 DataNode

xubuntu@hadoop-coc-2:~/Programs/hadoop$ jps
2342 NameNode
|

[2]: hdfs dfs -ls / output

|xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
15/03/03 05:09:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - xubuntu supergroup          0 2015-03-03 04:47 /input1

xubuntu@hadoop-coc-2:~/Programs/hadoop$ hdfs dfs -ls /
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
15/03/03 05:09:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
|



-- 
--

Re: Questions about HDFS federation

Posted by Sagar Thacker <sa...@gmail.com>.

Please unsubscribe me.

Re: Questions about HDFS federation

Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.

I suggest testing without federation first. That is, running two separate yarn/hdfs instances. Once that works properly, you can introduce federation in your config.

Kihwal
From: xeonmailinglist <xe...@gmail.com>
To: hdfs-dev@hadoop.apache.org
Sent: Tuesday, March 3, 2015 4:32 AM
Subject: Questions about HDFS federation

Hi,
1. The image below shows 2 YARN instances using the same federation of clouds. Is this possible?
2. I have configured in 2 hosts (hadooop-coc-1, and hadoop-coc-2) a federation of HDFS. In the configuration, I have set a namespace in each host, and a single data node (see image). The service is running properly. You can check the output of the jps commands in [1]. The strange part is that, when I list the files, they do not appear in hadoop-coc-2. You can check the output in [2]. Why this happens?

[1]: jps output xubuntu@hadoop-coc-1:~/Programs/hadoop$ jps
21538 NameNode
21773 DataNode

xubuntu@hadoop-coc-2:~/Programs/hadoop$ jps
2342 NameNode
[2]: hdfs dfs -ls / output xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
15/03/03 05:09:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x - xubuntu supergroup 0 2015-03-03 04:47 /input1

xubuntu@hadoop-coc-2:~/Programs/hadoop$ hdfs dfs -ls /
Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
15/03/03 05:09:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

--
--

Re: Questions about HDFS federation

Posted by Kihwal Lee <ki...@yahoo-inc.com.INVALID>.

Federation is for sharing the data node storage spaces, not for sharing the data among multiple clusters. Check out viewfs instead.

From: xeonmailinglist-yahoo <xe...@gmail.com>
To: hdfs-dev@hadoop.apache.org
Sent: Tuesday, March 3, 2015 12:31 PM
Subject: Re: Questions about HDFS federation

I would like to access the same file from hadoop-coc-1, and hadoop-coc-2 using HDFS federation.
Does anyone knows if it is possible to share namespaces between 2 Namenodes in a HDFS federation, so that I can access the same file from different hosts?

Thanks,

On 03-03-2015 10:32, xeonmailinglist wrote:

[1]: jps output xubuntu@hadoop-coc-1:~/Programs/hadoop$ jps
21538 NameNode
21773 DataNode

--
--
--
--

Re: Questions about HDFS federation

Posted by xeonmailinglist-yahoo <xe...@gmail.com>.

I would like to access the same file from hadoop-coc-1, and hadoop-coc-2 
using HDFS federation.
Does anyone knows if it is possible to share namespaces between 2 
Namenodes in a HDFS federation, so that I can access the same file from 
different hosts?

Thanks,



On 03-03-2015 10:32, xeonmailinglist wrote:
>
> Hi,
>
> 1. The image below shows 2 YARN  instances using the same federation 
> of clouds. Is this possible?
>
> 2. I have configured in 2 hosts (hadooop-coc-1, and hadoop-coc-2) a 
> federation of HDFS. In the configuration, I have set a namespace in 
> each host, and a single data node (see image). The service is running 
> properly. You can check the output of the |jps| commands in [1].
>
> The strange part is that, when I list the files, they do not appear in 
> hadoop-coc-2. You can check the output in [2]. Why this happens?
>
>
> hdfsfederation
>
> [1]: jps output
>
> |xubuntu@hadoop-coc-1:~/Programs/hadoop$ jps
> 21538 NameNode
> 21773 DataNode
>
> xubuntu@hadoop-coc-2:~/Programs/hadoop$ jps
> 2342 NameNode
> |
>
> [2]: hdfs dfs -ls / output
>
> |xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -ls /
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
> 15/03/03 05:09:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> Found 1 items
> drwxr-xr-x   - xubuntu supergroup          0 2015-03-03 04:47 /input1
>
> xubuntu@hadoop-coc-2:~/Programs/hadoop$ hdfs dfs -ls /
> Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
> 15/03/03 05:09:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> |
> 
> -- 
> --

-- 
--