You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Raj Hadoop <ha...@yahoo.com> on 2013/04/17 06:49:34 UTC
Basic Doubt in Hadoop
Hi,
I am new to Hadoop. I started reading the standard Wordcount program. I got this basic doubt in Hadoop.
After the Map - Reduce is done, where is the output generated? Does the reducer ouput sit on individual DataNodes ? Please advise.
Thanks,
Raj
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
You are correct, map outputs are stored in LFS not in HDFS.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Ramesh R Nair <ra...@gmail.com>
Date: Wed, 17 Apr 2013 13:06:32
To: <us...@hadoop.apache.org>; <be...@gmail.com>
Subject: Re: Basic Doubt in Hadoop
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by maisnam ns <ma...@gmail.com>.
@Bejoy
Adding a little bit here,the ouput of map task writes first to a memory
buffer, and when contents reaches a threshold a background thread will
write the contents to disk.
Niranjan Singh
On Wed, Apr 17, 2013 at 1:06 PM, Ramesh R Nair <ra...@gmail.com>wrote:
> Hi Bejoy,
>
> Regarding the output of Map phase, does Hadoop store it in local fs or
> in HDFS.
> I believe it is in the former. Correct me if I am wrong.
>
> Regards
> Ramesh
>
>
> On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
>
>> The data is in HDFS in case of WordCount MR sample.
>>
>> In hdfs, you have the metadata in NameNode and actual data as blocks
>> replicated across DataNodes.
>>
>> In case of reducer, If a reducer is running on a particular node then you
>> have one replica of the blocks in the same node (If there is no space
>> issues) and rest replicas on other nodes.
>> Regards
>> Bejoy KS
>>
>> Sent from remote device, Please excuse typos
>> ------------------------------
>> *From: * Raj Hadoop <ha...@yahoo.com>
>> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
>> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
>> *ReplyTo: * user@hadoop.apache.org
>> *Subject: *Basic Doubt in Hadoop
>>
>> Hi,
>>
>> I am new to Hadoop. I started reading the standard Wordcount program. I
>> got this basic doubt in Hadoop.
>>
>> After the Map - Reduce is done, where is the output generated? Does the
>> reducer ouput sit on individual DataNodes ? Please advise.
>>
>>
>> Thanks,
>> Raj
>>
>
>
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
You are correct, map outputs are stored in LFS not in HDFS.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Ramesh R Nair <ra...@gmail.com>
Date: Wed, 17 Apr 2013 13:06:32
To: <us...@hadoop.apache.org>; <be...@gmail.com>
Subject: Re: Basic Doubt in Hadoop
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
You are correct, map outputs are stored in LFS not in HDFS.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Ramesh R Nair <ra...@gmail.com>
Date: Wed, 17 Apr 2013 13:06:32
To: <us...@hadoop.apache.org>; <be...@gmail.com>
Subject: Re: Basic Doubt in Hadoop
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by maisnam ns <ma...@gmail.com>.
@Bejoy
Adding a little bit here,the ouput of map task writes first to a memory
buffer, and when contents reaches a threshold a background thread will
write the contents to disk.
Niranjan Singh
On Wed, Apr 17, 2013 at 1:06 PM, Ramesh R Nair <ra...@gmail.com>wrote:
> Hi Bejoy,
>
> Regarding the output of Map phase, does Hadoop store it in local fs or
> in HDFS.
> I believe it is in the former. Correct me if I am wrong.
>
> Regards
> Ramesh
>
>
> On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
>
>> The data is in HDFS in case of WordCount MR sample.
>>
>> In hdfs, you have the metadata in NameNode and actual data as blocks
>> replicated across DataNodes.
>>
>> In case of reducer, If a reducer is running on a particular node then you
>> have one replica of the blocks in the same node (If there is no space
>> issues) and rest replicas on other nodes.
>> Regards
>> Bejoy KS
>>
>> Sent from remote device, Please excuse typos
>> ------------------------------
>> *From: * Raj Hadoop <ha...@yahoo.com>
>> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
>> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
>> *ReplyTo: * user@hadoop.apache.org
>> *Subject: *Basic Doubt in Hadoop
>>
>> Hi,
>>
>> I am new to Hadoop. I started reading the standard Wordcount program. I
>> got this basic doubt in Hadoop.
>>
>> After the Map - Reduce is done, where is the output generated? Does the
>> reducer ouput sit on individual DataNodes ? Please advise.
>>
>>
>> Thanks,
>> Raj
>>
>
>
Re: Basic Doubt in Hadoop
Posted by maisnam ns <ma...@gmail.com>.
@Bejoy
Adding a little bit here,the ouput of map task writes first to a memory
buffer, and when contents reaches a threshold a background thread will
write the contents to disk.
Niranjan Singh
On Wed, Apr 17, 2013 at 1:06 PM, Ramesh R Nair <ra...@gmail.com>wrote:
> Hi Bejoy,
>
> Regarding the output of Map phase, does Hadoop store it in local fs or
> in HDFS.
> I believe it is in the former. Correct me if I am wrong.
>
> Regards
> Ramesh
>
>
> On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
>
>> The data is in HDFS in case of WordCount MR sample.
>>
>> In hdfs, you have the metadata in NameNode and actual data as blocks
>> replicated across DataNodes.
>>
>> In case of reducer, If a reducer is running on a particular node then you
>> have one replica of the blocks in the same node (If there is no space
>> issues) and rest replicas on other nodes.
>> Regards
>> Bejoy KS
>>
>> Sent from remote device, Please excuse typos
>> ------------------------------
>> *From: * Raj Hadoop <ha...@yahoo.com>
>> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
>> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
>> *ReplyTo: * user@hadoop.apache.org
>> *Subject: *Basic Doubt in Hadoop
>>
>> Hi,
>>
>> I am new to Hadoop. I started reading the standard Wordcount program. I
>> got this basic doubt in Hadoop.
>>
>> After the Map - Reduce is done, where is the output generated? Does the
>> reducer ouput sit on individual DataNodes ? Please advise.
>>
>>
>> Thanks,
>> Raj
>>
>
>
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
You are correct, map outputs are stored in LFS not in HDFS.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Ramesh R Nair <ra...@gmail.com>
Date: Wed, 17 Apr 2013 13:06:32
To: <us...@hadoop.apache.org>; <be...@gmail.com>
Subject: Re: Basic Doubt in Hadoop
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by maisnam ns <ma...@gmail.com>.
@Bejoy
Adding a little bit here,the ouput of map task writes first to a memory
buffer, and when contents reaches a threshold a background thread will
write the contents to disk.
Niranjan Singh
On Wed, Apr 17, 2013 at 1:06 PM, Ramesh R Nair <ra...@gmail.com>wrote:
> Hi Bejoy,
>
> Regarding the output of Map phase, does Hadoop store it in local fs or
> in HDFS.
> I believe it is in the former. Correct me if I am wrong.
>
> Regards
> Ramesh
>
>
> On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
>
>> The data is in HDFS in case of WordCount MR sample.
>>
>> In hdfs, you have the metadata in NameNode and actual data as blocks
>> replicated across DataNodes.
>>
>> In case of reducer, If a reducer is running on a particular node then you
>> have one replica of the blocks in the same node (If there is no space
>> issues) and rest replicas on other nodes.
>> Regards
>> Bejoy KS
>>
>> Sent from remote device, Please excuse typos
>> ------------------------------
>> *From: * Raj Hadoop <ha...@yahoo.com>
>> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
>> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
>> *ReplyTo: * user@hadoop.apache.org
>> *Subject: *Basic Doubt in Hadoop
>>
>> Hi,
>>
>> I am new to Hadoop. I started reading the standard Wordcount program. I
>> got this basic doubt in Hadoop.
>>
>> After the Map - Reduce is done, where is the output generated? Does the
>> reducer ouput sit on individual DataNodes ? Please advise.
>>
>>
>> Thanks,
>> Raj
>>
>
>
Re: Basic Doubt in Hadoop
Posted by Ramesh R Nair <ra...@gmail.com>.
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by Ramesh R Nair <ra...@gmail.com>.
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by Ramesh R Nair <ra...@gmail.com>.
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by Ramesh R Nair <ra...@gmail.com>.
Hi Bejoy,
Regarding the output of Map phase, does Hadoop store it in local fs or
in HDFS.
I believe it is in the former. Correct me if I am wrong.
Regards
Ramesh
On Wed, Apr 17, 2013 at 10:30 AM, <be...@gmail.com> wrote:
> The data is in HDFS in case of WordCount MR sample.
>
> In hdfs, you have the metadata in NameNode and actual data as blocks
> replicated across DataNodes.
>
> In case of reducer, If a reducer is running on a particular node then you
> have one replica of the blocks in the same node (If there is no space
> issues) and rest replicas on other nodes.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Raj Hadoop <ha...@yahoo.com>
> *Date: *Tue, 16 Apr 2013 21:49:34 -0700 (PDT)
> *To: *user@hadoop.apache.org<us...@hadoop.apache.org>
> *ReplyTo: * user@hadoop.apache.org
> *Subject: *Basic Doubt in Hadoop
>
> Hi,
>
> I am new to Hadoop. I started reading the standard Wordcount program. I
> got this basic doubt in Hadoop.
>
> After the Map - Reduce is done, where is the output generated? Does the
> reducer ouput sit on individual DataNodes ? Please advise.
>
>
> Thanks,
> Raj
>
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
The data is in HDFS in case of WordCount MR sample.
In hdfs, you have the metadata in NameNode and actual data as blocks replicated across DataNodes.
In case of reducer, If a reducer is running on a particular node then you have one replica of the blocks in the same node (If there is no space issues) and rest replicas on other nodes.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Raj Hadoop <ha...@yahoo.com>
Date: Tue, 16 Apr 2013 21:49:34
To: user@hadoop.apache.org<us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Basic Doubt in Hadoop
Hi,
I am new to Hadoop. I started reading the standard Wordcount program. I got this basic doubt in Hadoop.
After the Map - Reduce is done, where is the output generated? Does the reducer ouput sit on individual DataNodes ? Please advise.
Thanks,
Raj
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
The data is in HDFS in case of WordCount MR sample.
In hdfs, you have the metadata in NameNode and actual data as blocks replicated across DataNodes.
In case of reducer, If a reducer is running on a particular node then you have one replica of the blocks in the same node (If there is no space issues) and rest replicas on other nodes.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Raj Hadoop <ha...@yahoo.com>
Date: Tue, 16 Apr 2013 21:49:34
To: user@hadoop.apache.org<us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Basic Doubt in Hadoop
Hi,
I am new to Hadoop. I started reading the standard Wordcount program. I got this basic doubt in Hadoop.
After the Map - Reduce is done, where is the output generated? Does the reducer ouput sit on individual DataNodes ? Please advise.
Thanks,
Raj
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
The data is in HDFS in case of WordCount MR sample.
In hdfs, you have the metadata in NameNode and actual data as blocks replicated across DataNodes.
In case of reducer, If a reducer is running on a particular node then you have one replica of the blocks in the same node (If there is no space issues) and rest replicas on other nodes.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Raj Hadoop <ha...@yahoo.com>
Date: Tue, 16 Apr 2013 21:49:34
To: user@hadoop.apache.org<us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Basic Doubt in Hadoop
Hi,
I am new to Hadoop. I started reading the standard Wordcount program. I got this basic doubt in Hadoop.
After the Map - Reduce is done, where is the output generated? Does the reducer ouput sit on individual DataNodes ? Please advise.
Thanks,
Raj
Re: Basic Doubt in Hadoop
Posted by be...@gmail.com.
The data is in HDFS in case of WordCount MR sample.
In hdfs, you have the metadata in NameNode and actual data as blocks replicated across DataNodes.
In case of reducer, If a reducer is running on a particular node then you have one replica of the blocks in the same node (If there is no space issues) and rest replicas on other nodes.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Raj Hadoop <ha...@yahoo.com>
Date: Tue, 16 Apr 2013 21:49:34
To: user@hadoop.apache.org<us...@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Basic Doubt in Hadoop
Hi,
I am new to Hadoop. I started reading the standard Wordcount program. I got this basic doubt in Hadoop.
After the Map - Reduce is done, where is the output generated? Does the reducer ouput sit on individual DataNodes ? Please advise.
Thanks,
Raj