You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by praveenesh kumar <pr...@gmail.com> on 2011/12/26 12:35:33 UTC
Secondary Namenode on hadoop 0.20.205 ?
Hey people,
How can we setup another machine in the cluster as Secondary Namenode
in hadoop 0.20.205 ?
Can a DN also act as SNN, any pros and cons of having this configuration ?
Thanks,
Praveenesh
Re: Secondary Namenode on hadoop 0.20.205 ?
Posted by Harsh J <ha...@cloudera.com>.
Yes, checkpoints are helpful when your original NN image goes corrupt (very very rare, if you use dual or more dfs.name.dir points to be safe).
On 27-Dec-2011, at 12:33 PM, praveenesh kumar wrote:
> Cool.
> I just did that..
> So now I am seeing my fsimage file on SNN's hadoop.tmp.dir...
> So incase my NN went down.. I can take this image file from SNN and paste
> it at NN's *dfs.name.dir/current/fsimage *
> and I can have NN up based on last snapshot that SNN had, right ?
>
> Thanks,
> Praveenesh
>
> On Tue, Dec 27, 2011 at 12:20 PM, Harsh J <ha...@cloudera.com> wrote:
>> The link Uma passed already covered that question:
> http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/[dfs.http.address
> in hdfs-site.xml pointing to NN_HOST:50070 should d.]
>>
>> Also, if you are using the tarball start/stop scripts, putting in the
> hostname for SNN in the conf/masters list is sufficient to get it
> auto-started there.
>>
>> On 27-Dec-2011, at 11:36 AM, praveenesh kumar wrote:
>>
>>> Thanks..But, my 1st question is still unanswered.
>>> I have a 8 DN/TT machines and 1 NN machine.
>>> I want to set one of my DN/TT machine as SNN.
>>> How I have to configure my conf/*.xml files to achieve this ?
>>>
>>> Thanks,
>>> Praveenesh
>>>
>>> On Mon, Dec 26, 2011 at 8:44 PM, Harsh J <ha...@cloudera.com> wrote:
>>>> (Answering beyond Uma's reply)
>>>>
>>>>> Can a DN also act as SNN, any pros and cons of having this
> configuration ?
>>>>
>>>> You can run SNN on a regular slave box if you can't have a dedicate a
> box, it shouldn't be an issue for small clusters -- Do ensure its disk
> configuration is proper, and its allocated near to the same heap as the
> NameNode is.
>>>>
>>>> For large clusters where the fsimage and periodic edits file sizes are
> larger, it would be worth placing it on a separate box given SNN's
> interactions.
>>>>
>>>> On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote:
>>>>
>>>>> Hey Praveenesh,
>>>>>
>>>>> You can start secondary namenode also by just giving the option
> ./hadoop secondarynamenode
>>>>>
>>>>> DN can not act as seconday namenode. The basic work for seconday
> namenode is to do checkpointing and getting the edits insync with Namenode
> till last checkpointing period. DN is to store the real data blocks
> physically.
>>>>> you need to configure correct namenode http address also for the
> secondaryNN, so that it can connect NN for checkpointing operations.
>>>>>
> http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode
>>>>> You can configure secondary node IP in masters file, start-dfs.sh
> itself will start the SNN automatically as it starts DN and NN as well.
>>>>>
>>>>> also you can see
> http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/
>>>>>
>>>>> Regards,
>>>>> Uma
>>>>> ________________________________________
>>>>> From: praveenesh kumar [praveenesh@gmail.com]
>>>>> Sent: Monday, December 26, 2011 5:05 PM
>>>>> To: common-user@hadoop.apache.org
>>>>> Subject: Secondary Namenode on hadoop 0.20.205 ?
>>>>>
>>>>> Hey people,
>>>>>
>>>>> How can we setup another machine in the cluster as Secondary Namenode
>>>>> in hadoop 0.20.205 ?
>>>>> Can a DN also act as SNN, any pros and cons of having this
> configuration ?
>>>>>
>>>>> Thanks,
>>>>> Praveenesh
>>>>
>>
Re: Secondary Namenode on hadoop 0.20.205 ?
Posted by praveenesh kumar <pr...@gmail.com>.
Cool.
I just did that..
So now I am seeing my fsimage file on SNN's hadoop.tmp.dir...
So incase my NN went down.. I can take this image file from SNN and paste
it at NN's *dfs.name.dir/current/fsimage *
and I can have NN up based on last snapshot that SNN had, right ?
Thanks,
Praveenesh
On Tue, Dec 27, 2011 at 12:20 PM, Harsh J <ha...@cloudera.com> wrote:
> The link Uma passed already covered that question:
http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/[dfs.http.address
in hdfs-site.xml pointing to NN_HOST:50070 should d.]
>
> Also, if you are using the tarball start/stop scripts, putting in the
hostname for SNN in the conf/masters list is sufficient to get it
auto-started there.
>
> On 27-Dec-2011, at 11:36 AM, praveenesh kumar wrote:
>
>> Thanks..But, my 1st question is still unanswered.
>> I have a 8 DN/TT machines and 1 NN machine.
>> I want to set one of my DN/TT machine as SNN.
>> How I have to configure my conf/*.xml files to achieve this ?
>>
>> Thanks,
>> Praveenesh
>>
>> On Mon, Dec 26, 2011 at 8:44 PM, Harsh J <ha...@cloudera.com> wrote:
>>> (Answering beyond Uma's reply)
>>>
>>>> Can a DN also act as SNN, any pros and cons of having this
configuration ?
>>>
>>> You can run SNN on a regular slave box if you can't have a dedicate a
box, it shouldn't be an issue for small clusters -- Do ensure its disk
configuration is proper, and its allocated near to the same heap as the
NameNode is.
>>>
>>> For large clusters where the fsimage and periodic edits file sizes are
larger, it would be worth placing it on a separate box given SNN's
interactions.
>>>
>>> On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote:
>>>
>>>> Hey Praveenesh,
>>>>
>>>> You can start secondary namenode also by just giving the option
./hadoop secondarynamenode
>>>>
>>>> DN can not act as seconday namenode. The basic work for seconday
namenode is to do checkpointing and getting the edits insync with Namenode
till last checkpointing period. DN is to store the real data blocks
physically.
>>>> you need to configure correct namenode http address also for the
secondaryNN, so that it can connect NN for checkpointing operations.
>>>>
http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode
>>>> You can configure secondary node IP in masters file, start-dfs.sh
itself will start the SNN automatically as it starts DN and NN as well.
>>>>
>>>> also you can see
http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/
>>>>
>>>> Regards,
>>>> Uma
>>>> ________________________________________
>>>> From: praveenesh kumar [praveenesh@gmail.com]
>>>> Sent: Monday, December 26, 2011 5:05 PM
>>>> To: common-user@hadoop.apache.org
>>>> Subject: Secondary Namenode on hadoop 0.20.205 ?
>>>>
>>>> Hey people,
>>>>
>>>> How can we setup another machine in the cluster as Secondary Namenode
>>>> in hadoop 0.20.205 ?
>>>> Can a DN also act as SNN, any pros and cons of having this
configuration ?
>>>>
>>>> Thanks,
>>>> Praveenesh
>>>
>
Re: Secondary Namenode on hadoop 0.20.205 ?
Posted by Harsh J <ha...@cloudera.com>.
The link Uma passed already covered that question: http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ [dfs.http.address in hdfs-site.xml pointing to NN_HOST:50070 should do.]
Also, if you are using the tarball start/stop scripts, putting in the hostname for SNN in the conf/masters list is sufficient to get it auto-started there.
On 27-Dec-2011, at 11:36 AM, praveenesh kumar wrote:
> Thanks..But, my 1st question is still unanswered.
> I have a 8 DN/TT machines and 1 NN machine.
> I want to set one of my DN/TT machine as SNN.
> How I have to configure my conf/*.xml files to achieve this ?
>
> Thanks,
> Praveenesh
>
> On Mon, Dec 26, 2011 at 8:44 PM, Harsh J <ha...@cloudera.com> wrote:
>> (Answering beyond Uma's reply)
>>
>>> Can a DN also act as SNN, any pros and cons of having this configuration ?
>>
>> You can run SNN on a regular slave box if you can't have a dedicate a box, it shouldn't be an issue for small clusters -- Do ensure its disk configuration is proper, and its allocated near to the same heap as the NameNode is.
>>
>> For large clusters where the fsimage and periodic edits file sizes are larger, it would be worth placing it on a separate box given SNN's interactions.
>>
>> On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote:
>>
>>> Hey Praveenesh,
>>>
>>> You can start secondary namenode also by just giving the option ./hadoop secondarynamenode
>>>
>>> DN can not act as seconday namenode. The basic work for seconday namenode is to do checkpointing and getting the edits insync with Namenode till last checkpointing period. DN is to store the real data blocks physically.
>>> you need to configure correct namenode http address also for the secondaryNN, so that it can connect NN for checkpointing operations.
>>> http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode
>>> You can configure secondary node IP in masters file, start-dfs.sh itself will start the SNN automatically as it starts DN and NN as well.
>>>
>>> also you can see http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/
>>>
>>> Regards,
>>> Uma
>>> ________________________________________
>>> From: praveenesh kumar [praveenesh@gmail.com]
>>> Sent: Monday, December 26, 2011 5:05 PM
>>> To: common-user@hadoop.apache.org
>>> Subject: Secondary Namenode on hadoop 0.20.205 ?
>>>
>>> Hey people,
>>>
>>> How can we setup another machine in the cluster as Secondary Namenode
>>> in hadoop 0.20.205 ?
>>> Can a DN also act as SNN, any pros and cons of having this configuration ?
>>>
>>> Thanks,
>>> Praveenesh
>>
Re: Secondary Namenode on hadoop 0.20.205 ?
Posted by praveenesh kumar <pr...@gmail.com>.
Thanks..But, my 1st question is still unanswered.
I have a 8 DN/TT machines and 1 NN machine.
I want to set one of my DN/TT machine as SNN.
How I have to configure my conf/*.xml files to achieve this ?
Thanks,
Praveenesh
On Mon, Dec 26, 2011 at 8:44 PM, Harsh J <ha...@cloudera.com> wrote:
> (Answering beyond Uma's reply)
>
>> Can a DN also act as SNN, any pros and cons of having this configuration ?
>
> You can run SNN on a regular slave box if you can't have a dedicate a box, it shouldn't be an issue for small clusters -- Do ensure its disk configuration is proper, and its allocated near to the same heap as the NameNode is.
>
> For large clusters where the fsimage and periodic edits file sizes are larger, it would be worth placing it on a separate box given SNN's interactions.
>
> On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote:
>
>> Hey Praveenesh,
>>
>> You can start secondary namenode also by just giving the option ./hadoop secondarynamenode
>>
>> DN can not act as seconday namenode. The basic work for seconday namenode is to do checkpointing and getting the edits insync with Namenode till last checkpointing period. DN is to store the real data blocks physically.
>> you need to configure correct namenode http address also for the secondaryNN, so that it can connect NN for checkpointing operations.
>> http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode
>> You can configure secondary node IP in masters file, start-dfs.sh itself will start the SNN automatically as it starts DN and NN as well.
>>
>> also you can see http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/
>>
>> Regards,
>> Uma
>> ________________________________________
>> From: praveenesh kumar [praveenesh@gmail.com]
>> Sent: Monday, December 26, 2011 5:05 PM
>> To: common-user@hadoop.apache.org
>> Subject: Secondary Namenode on hadoop 0.20.205 ?
>>
>> Hey people,
>>
>> How can we setup another machine in the cluster as Secondary Namenode
>> in hadoop 0.20.205 ?
>> Can a DN also act as SNN, any pros and cons of having this configuration ?
>>
>> Thanks,
>> Praveenesh
>
Re: Secondary Namenode on hadoop 0.20.205 ?
Posted by Harsh J <ha...@cloudera.com>.
(Answering beyond Uma's reply)
> Can a DN also act as SNN, any pros and cons of having this configuration ?
You can run SNN on a regular slave box if you can't have a dedicate a box, it shouldn't be an issue for small clusters -- Do ensure its disk configuration is proper, and its allocated near to the same heap as the NameNode is.
For large clusters where the fsimage and periodic edits file sizes are larger, it would be worth placing it on a separate box given SNN's interactions.
On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote:
> Hey Praveenesh,
>
> You can start secondary namenode also by just giving the option ./hadoop secondarynamenode
>
> DN can not act as seconday namenode. The basic work for seconday namenode is to do checkpointing and getting the edits insync with Namenode till last checkpointing period. DN is to store the real data blocks physically.
> you need to configure correct namenode http address also for the secondaryNN, so that it can connect NN for checkpointing operations.
> http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode
> You can configure secondary node IP in masters file, start-dfs.sh itself will start the SNN automatically as it starts DN and NN as well.
>
> also you can see http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/
>
> Regards,
> Uma
> ________________________________________
> From: praveenesh kumar [praveenesh@gmail.com]
> Sent: Monday, December 26, 2011 5:05 PM
> To: common-user@hadoop.apache.org
> Subject: Secondary Namenode on hadoop 0.20.205 ?
>
> Hey people,
>
> How can we setup another machine in the cluster as Secondary Namenode
> in hadoop 0.20.205 ?
> Can a DN also act as SNN, any pros and cons of having this configuration ?
>
> Thanks,
> Praveenesh
RE: Secondary Namenode on hadoop 0.20.205 ?
Posted by Uma Maheswara Rao G <ma...@huawei.com>.
Hey Praveenesh,
You can start secondary namenode also by just giving the option ./hadoop secondarynamenode
DN can not act as seconday namenode. The basic work for seconday namenode is to do checkpointing and getting the edits insync with Namenode till last checkpointing period. DN is to store the real data blocks physically.
you need to configure correct namenode http address also for the secondaryNN, so that it can connect NN for checkpointing operations.
http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode
You can configure secondary node IP in masters file, start-dfs.sh itself will start the SNN automatically as it starts DN and NN as well.
also you can see http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/
Regards,
Uma
________________________________________
From: praveenesh kumar [praveenesh@gmail.com]
Sent: Monday, December 26, 2011 5:05 PM
To: common-user@hadoop.apache.org
Subject: Secondary Namenode on hadoop 0.20.205 ?
Hey people,
How can we setup another machine in the cluster as Secondary Namenode
in hadoop 0.20.205 ?
Can a DN also act as SNN, any pros and cons of having this configuration ?
Thanks,
Praveenesh