Small Issues..!

[Sugandha Naolekar <ha...@gmail.com>]

Hello!

I have a 4 node cluster of hadoop running. Now, there is 5th machine which
is acting as a  client of hadoop. It's not a part of the hadoop
cluster(master/slave config file). Now I have to writer a JAVA  code that
gets executed on this client which will simply put the client ystem's data
into HDFS(and get it replicated over 2 datanodes) and as per my requirement,
I can simply fetch it back on the client machine itself.

For this, I have done following things as of now::

***********************************
-> Among 4 nodes 2 are datanodes and ther oter 2 are namenode and jobtracker
respectively.
***********************************

***********************************
-> Now, to make that code work on client machine, I have designed a UI. Now
here on the client m/c, do i need to install hadoop?
***********************************

***********************************
-> I have installed hadoop on it, and in it's config file, I have specified
only 2 tags.
   1) fs.default.name-> value=namenode's address.
   2) dfs.http.address(namenode's addres)
***********************************

***********************************
Thus, If there is a file in /home/hadoop/test.java on client machine; I will
have 2 get the instance of HDFS fs by Filesystem.get. rt??
***********************************

***********************************
Then, by using Filesystem.util, I will have to simply specify both the
fs::local as src, hdfs as destination, and src path as the
/home/hadoop/test.java and destination as /user/hadoop/. rt??
So it should work ...!
***********************************

***********************************
-> But, it gives me an error as "not able to find src path
/home/hadoop/test.java"

-> Will i have to use RPC classes and methods under hadoop api to do this.??
***********************************

***********************************
 Things don;t seem to be working in any of the ways. Please help me out.
***********************************

Thanks!

Re: Small Issues..!

[Matt Massie <ma...@cloudera.com>]

On Jun 14, 2009, at 11:01 PM, Sugandha Naolekar wrote:

> Hello!
>
> I have a 4 node cluster of hadoop running. Now, there is 5th machine  
> which
> is acting as a  client of hadoop. It's not a part of the hadoop
> cluster(master/slave config file). Now I have to writer a JAVA  code  
> that
> gets executed on this client which will simply put the client  
> ystem's data
> into HDFS(and get it replicated over 2 datanodes) and as per my  
> requirement,
> I can simply fetch it back on the client machine itself.
>
> For this, I have done following things as of now::
>
> ***********************************
> -> Among 4 nodes 2 are datanodes and ther oter 2 are namenode and  
> jobtracker
> respectively.
> ***********************************
>
> ***********************************
> -> Now, to make that code work on client machine, I have designed a  
> UI. Now
> here on the client m/c, do i need to install hadoop?
> ***********************************

You will need to have the same version  of Hadoop installed on any  
client that need to communicate with the Hadoop cluster.

>
> ***********************************
> -> I have installed hadoop on it, and in it's config file, I have  
> specified
> only 2 tags.
>   1) fs.default.name-> value=namenode's address.
>   2) dfs.http.address(namenode's addres)
> ***********************************

I'm assuming you mean that you have Hadoop installed on the client  
with a hadoop-site.xml (or core-site.xml) with the correct  
fs.default.name.  Correct?

>
> ***********************************
> Thus, If there is a file in /home/hadoop/test.java on client  
> machine; I will
> have 2 get the instance of HDFS fs by Filesystem.get. rt??
> ***********************************

Before you begin writing special FileSystem Java code, I would do a  
quick sanity check of the client configuration.

Can you run the command...

% bin/hadoop fs -ls

...without error?

Can you -put files onto HDFS from the client...

% bin/hadoop fs -put <src> <dst>

...without error?

* You should also check your firewall rules between the client and  
NameNode.
* Make sure that the TCP port you specified in fs.default.name is open  
for connection from the client.
* Run "netstat -t -l" to make sure that the NameNode is running and  
listening on the TCP port you specified.

Only when you've ensured that the hadoop commandline works would I  
begin writing custom client code based on the FileSystem class.

>
> ***********************************
> Then, by using Filesystem.util, I will have to simply specify both the
> fs::local as src, hdfs as destination, and src path as the
> /home/hadoop/test.java and destination as /user/hadoop/. rt??
> So it should work ...!
> ***********************************
>
> ***********************************
> -> But, it gives me an error as "not able to find src path
> /home/hadoop/test.java"
>
> -> Will i have to use RPC classes and methods under hadoop api to do  
> this.??
> ***********************************

You should be able to just use the FileSystem class to w/o needing to  
use any RPC classes

FileSystem documentation:
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/fs/FileSystem.html


>
> ***********************************
> Things don;t seem to be working in any of the ways. Please help me  
> out.
> ***********************************
>
> Thanks!