You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by navaz <na...@gmail.com> on 2014/02/27 05:23:25 UTC
Client and NN communication in Hadoop
Hi
I have a Hadoop cluster running with 3 slaves and 1 Master. Slaves are
Datanodes and running Tasktarckers. Namenode is running jobtracker and
secondary namenode. I am running sample mapreduce after loading a file into
HDFS.
According to Hadoop architecture ,before writing a file into HDFS , client
will contact the Namenode and get the location details of DNs and then
client directly write the file into DN. What is this client ? is it an
application running on Namenode ? Is user and client both are different ?
How can I see the messages between client and datanodes?
Appreciate your response.
Thanks & Regards
Abdul Navaz
Graduate Student
University of Houston Main Campus,Texas
Mob: (281) 685-0388
Re: Client and NN communication in Hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
The application that try to write the file, for example the binary "hadoop"
found in the bin dir of hadoop is a hadoop client, whoever wants to read or
write from to hadoop is a client, you can also write hadoop client using
java apis which hadoop provide.
like following is an example of hadoop client which tries to read file from
hadoop
import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class Cat{
public static void main (String [] args) throws Exception{
try{
Path pt=new
Path("hdfs://shashwat:9000/home/shriparv/fileonhdfstoread.txt");
FileSystem filesysObject = FileSystem.get(new
Configuration());
BufferedReader bufferReaderObject=new
BufferedReader(new InputStreamReader(filesysObject.open(pt)));
String line;
line=bufferReaderObject.readLine();
while (line != null){
System.out.println(line);
line=bufferReaderObject.readLine();
}
}catch(Exception e){
}
}
}
Hope this clear some of your doubts
* Warm Regards_**∞_*
* Shashwat Shriparv*
[image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
http://google.com/+ShashwatShriparv]
<http://google.com/+ShashwatShriparv>[image:
http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
On Thu, Feb 27, 2014 at 9:53 AM, navaz <na...@gmail.com> wrote:
> Hi
>
>
>
> I have a Hadoop cluster running with 3 slaves and 1 Master. Slaves are
> Datanodes and running Tasktarckers. Namenode is running jobtracker and
> secondary namenode. I am running sample mapreduce after loading a file into
> HDFS.
>
>
>
> According to Hadoop architecture ,before writing a file into HDFS ,
> client will contact the Namenode and get the location details of DNs and
> then client directly write the file into DN. What is this client ? is it
> an application running on Namenode ? Is user and client both are different
> ? How can I see the messages between client and datanodes?
>
>
>
> Appreciate your response.
>
>
>
>
>
> Thanks & Regards
>
>
>
> Abdul Navaz
>
> Graduate Student
>
> University of Houston Main Campus,Texas
>
> Mob: (281) 685-0388
>
>
>
Re: Client and NN communication in Hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
The application that try to write the file, for example the binary "hadoop"
found in the bin dir of hadoop is a hadoop client, whoever wants to read or
write from to hadoop is a client, you can also write hadoop client using
java apis which hadoop provide.
like following is an example of hadoop client which tries to read file from
hadoop
import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class Cat{
public static void main (String [] args) throws Exception{
try{
Path pt=new
Path("hdfs://shashwat:9000/home/shriparv/fileonhdfstoread.txt");
FileSystem filesysObject = FileSystem.get(new
Configuration());
BufferedReader bufferReaderObject=new
BufferedReader(new InputStreamReader(filesysObject.open(pt)));
String line;
line=bufferReaderObject.readLine();
while (line != null){
System.out.println(line);
line=bufferReaderObject.readLine();
}
}catch(Exception e){
}
}
}
Hope this clear some of your doubts
* Warm Regards_**∞_*
* Shashwat Shriparv*
[image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
http://google.com/+ShashwatShriparv]
<http://google.com/+ShashwatShriparv>[image:
http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
On Thu, Feb 27, 2014 at 9:53 AM, navaz <na...@gmail.com> wrote:
> Hi
>
>
>
> I have a Hadoop cluster running with 3 slaves and 1 Master. Slaves are
> Datanodes and running Tasktarckers. Namenode is running jobtracker and
> secondary namenode. I am running sample mapreduce after loading a file into
> HDFS.
>
>
>
> According to Hadoop architecture ,before writing a file into HDFS ,
> client will contact the Namenode and get the location details of DNs and
> then client directly write the file into DN. What is this client ? is it
> an application running on Namenode ? Is user and client both are different
> ? How can I see the messages between client and datanodes?
>
>
>
> Appreciate your response.
>
>
>
>
>
> Thanks & Regards
>
>
>
> Abdul Navaz
>
> Graduate Student
>
> University of Houston Main Campus,Texas
>
> Mob: (281) 685-0388
>
>
>
Re: Client and NN communication in Hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
The application that try to write the file, for example the binary "hadoop"
found in the bin dir of hadoop is a hadoop client, whoever wants to read or
write from to hadoop is a client, you can also write hadoop client using
java apis which hadoop provide.
like following is an example of hadoop client which tries to read file from
hadoop
import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class Cat{
public static void main (String [] args) throws Exception{
try{
Path pt=new
Path("hdfs://shashwat:9000/home/shriparv/fileonhdfstoread.txt");
FileSystem filesysObject = FileSystem.get(new
Configuration());
BufferedReader bufferReaderObject=new
BufferedReader(new InputStreamReader(filesysObject.open(pt)));
String line;
line=bufferReaderObject.readLine();
while (line != null){
System.out.println(line);
line=bufferReaderObject.readLine();
}
}catch(Exception e){
}
}
}
Hope this clear some of your doubts
* Warm Regards_**∞_*
* Shashwat Shriparv*
[image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
http://google.com/+ShashwatShriparv]
<http://google.com/+ShashwatShriparv>[image:
http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
On Thu, Feb 27, 2014 at 9:53 AM, navaz <na...@gmail.com> wrote:
> Hi
>
>
>
> I have a Hadoop cluster running with 3 slaves and 1 Master. Slaves are
> Datanodes and running Tasktarckers. Namenode is running jobtracker and
> secondary namenode. I am running sample mapreduce after loading a file into
> HDFS.
>
>
>
> According to Hadoop architecture ,before writing a file into HDFS ,
> client will contact the Namenode and get the location details of DNs and
> then client directly write the file into DN. What is this client ? is it
> an application running on Namenode ? Is user and client both are different
> ? How can I see the messages between client and datanodes?
>
>
>
> Appreciate your response.
>
>
>
>
>
> Thanks & Regards
>
>
>
> Abdul Navaz
>
> Graduate Student
>
> University of Houston Main Campus,Texas
>
> Mob: (281) 685-0388
>
>
>
Re: Client and NN communication in Hadoop
Posted by shashwat shriparv <dw...@gmail.com>.
The application that try to write the file, for example the binary "hadoop"
found in the bin dir of hadoop is a hadoop client, whoever wants to read or
write from to hadoop is a client, you can also write hadoop client using
java apis which hadoop provide.
like following is an example of hadoop client which tries to read file from
hadoop
import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class Cat{
public static void main (String [] args) throws Exception{
try{
Path pt=new
Path("hdfs://shashwat:9000/home/shriparv/fileonhdfstoread.txt");
FileSystem filesysObject = FileSystem.get(new
Configuration());
BufferedReader bufferReaderObject=new
BufferedReader(new InputStreamReader(filesysObject.open(pt)));
String line;
line=bufferReaderObject.readLine();
while (line != null){
System.out.println(line);
line=bufferReaderObject.readLine();
}
}catch(Exception e){
}
}
}
Hope this clear some of your doubts
* Warm Regards_**∞_*
* Shashwat Shriparv*
[image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
http://google.com/+ShashwatShriparv]
<http://google.com/+ShashwatShriparv>[image:
http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
On Thu, Feb 27, 2014 at 9:53 AM, navaz <na...@gmail.com> wrote:
> Hi
>
>
>
> I have a Hadoop cluster running with 3 slaves and 1 Master. Slaves are
> Datanodes and running Tasktarckers. Namenode is running jobtracker and
> secondary namenode. I am running sample mapreduce after loading a file into
> HDFS.
>
>
>
> According to Hadoop architecture ,before writing a file into HDFS ,
> client will contact the Namenode and get the location details of DNs and
> then client directly write the file into DN. What is this client ? is it
> an application running on Namenode ? Is user and client both are different
> ? How can I see the messages between client and datanodes?
>
>
>
> Appreciate your response.
>
>
>
>
>
> Thanks & Regards
>
>
>
> Abdul Navaz
>
> Graduate Student
>
> University of Houston Main Campus,Texas
>
> Mob: (281) 685-0388
>
>
>