You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ChingShen <ch...@gmail.com> on 2010/09/09 08:36:41 UTC
Questions about BN and CN
Hi all,
I got the Backup node(BN) that includes all the checkpoint responsibilities,
and it maintains an up-to-date namespace state, which is always in
sync with the active NN.
Q1. In which situation do we need a CN?
Q2. If the NameNode machine fails, which different manual
intervention between BN and CN?
Thanks.
Shen
Re: How to disable secondary node
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
If a CheckpointNode or a BackupNode is used then the SecondaryNameNode should be disabled.
--Konstantin
On 9/9/2010 7:59 AM, Edward Capriolo wrote:
> It is a bad idea to permanently disable 2nn. The edits file grows very
> very large and will not be processed until the name node restart. We
> had a 12GB edit file that took 40 minutes of downtime to process.
>
> On Thu, Sep 9, 2010 at 3:08 AM, Jeff Zhang<zj...@gmail.com> wrote:
>> then, do not start secondary namenode
>> (
>> comment the last line of start-dfs.sh
>> # "$bin"/hadoop-daemons.sh --config $HADOOP_CONF_DIR --hosts masters
>> start secondarynamenode
>> )
>>
>>
>>
>> On Wed, Sep 8, 2010 at 11:36 PM, akhilesh.kumar
>> <ak...@zapak.co.in> wrote:
>>> Hi ,
>>> Can any one help me in disabling secondary namenode feature of hdfs .
>>>
>>> Regards
>>> Akhilesh
>>>
>>> -----Original Message-----
>>> From: ChingShen [mailto:chingshenchen@gmail.com]
>>> Sent: Thursday, September 09, 2010 12:07 PM
>>> To: common-user@hadoop.apache.org
>>> Subject: Questions about BN and CN
>>>
>>> Hi all,
>>>
>>> I got the Backup node(BN) that includes all the checkpoint
>>> responsibilities,
>>> and it maintains an up-to-date namespace state, which is always in
>>> sync with the active NN.
>>>
>>> Q1. In which situation do we need a CN?
>>>
>>> Q2. If the NameNode machine fails, which different manual
>>> intervention between BN and CN?
>>>
>>> Thanks.
>>>
>>> Shen
>>>
>>>
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
Re: How to disable secondary node
Posted by Edward Capriolo <ed...@gmail.com>.
It is a bad idea to permanently disable 2nn. The edits file grows very
very large and will not be processed until the name node restart. We
had a 12GB edit file that took 40 minutes of downtime to process.
On Thu, Sep 9, 2010 at 3:08 AM, Jeff Zhang <zj...@gmail.com> wrote:
> then, do not start secondary namenode
> (
> comment the last line of start-dfs.sh
> # "$bin"/hadoop-daemons.sh --config $HADOOP_CONF_DIR --hosts masters
> start secondarynamenode
> )
>
>
>
> On Wed, Sep 8, 2010 at 11:36 PM, akhilesh.kumar
> <ak...@zapak.co.in> wrote:
>> Hi ,
>> Can any one help me in disabling secondary namenode feature of hdfs .
>>
>> Regards
>> Akhilesh
>>
>> -----Original Message-----
>> From: ChingShen [mailto:chingshenchen@gmail.com]
>> Sent: Thursday, September 09, 2010 12:07 PM
>> To: common-user@hadoop.apache.org
>> Subject: Questions about BN and CN
>>
>> Hi all,
>>
>> I got the Backup node(BN) that includes all the checkpoint
>> responsibilities,
>> and it maintains an up-to-date namespace state, which is always in
>> sync with the active NN.
>>
>> Q1. In which situation do we need a CN?
>>
>> Q2. If the NameNode machine fails, which different manual
>> intervention between BN and CN?
>>
>> Thanks.
>>
>> Shen
>>
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: How to disable secondary node
Posted by Jeff Zhang <zj...@gmail.com>.
then, do not start secondary namenode
(
comment the last line of start-dfs.sh
# "$bin"/hadoop-daemons.sh --config $HADOOP_CONF_DIR --hosts masters
start secondarynamenode
)
On Wed, Sep 8, 2010 at 11:36 PM, akhilesh.kumar
<ak...@zapak.co.in> wrote:
> Hi ,
> Can any one help me in disabling secondary namenode feature of hdfs .
>
> Regards
> Akhilesh
>
> -----Original Message-----
> From: ChingShen [mailto:chingshenchen@gmail.com]
> Sent: Thursday, September 09, 2010 12:07 PM
> To: common-user@hadoop.apache.org
> Subject: Questions about BN and CN
>
> Hi all,
>
> I got the Backup node(BN) that includes all the checkpoint
> responsibilities,
> and it maintains an up-to-date namespace state, which is always in
> sync with the active NN.
>
> Q1. In which situation do we need a CN?
>
> Q2. If the NameNode machine fails, which different manual
> intervention between BN and CN?
>
> Thanks.
>
> Shen
>
>
--
Best Regards
Jeff Zhang
How to disable secondary node
Posted by "akhilesh.kumar" <ak...@zapak.co.in>.
Hi ,
Can any one help me in disabling secondary namenode feature of hdfs .
Regards
Akhilesh
-----Original Message-----
From: ChingShen [mailto:chingshenchen@gmail.com]
Sent: Thursday, September 09, 2010 12:07 PM
To: common-user@hadoop.apache.org
Subject: Questions about BN and CN
Hi all,
I got the Backup node(BN) that includes all the checkpoint
responsibilities,
and it maintains an up-to-date namespace state, which is always in
sync with the active NN.
Q1. In which situation do we need a CN?
Q2. If the NameNode machine fails, which different manual
intervention between BN and CN?
Thanks.
Shen
Re: Questions about BN and CN
Posted by Todd Lipcon <to...@cloudera.com>.
On Thu, Sep 23, 2010 at 6:20 PM, Konstantin Shvachko <sh...@yahoo-inc.com>wrote:
> Hi Shen,
>
> Why do we need CheckpointNode?
> 1. First of all it is a compatible replacement of SecondaryNameNode.
> 2. Checkpointing is also needed for periodically compacting edits.
> You can do it with CN or BN, but CN is more lightweight.
> I assume there could be cases when streaming edits to BN over network
> can be slower than writing them to disk, so you might want to turn BN
> off for performance reasons.
>
Also, if the BN hangs, it will hang edits on the primary node as well, since
synchronous RPCs are used to push edits, right Konst? Would be worth testing
a "kill -STOP" on the BN while performing operations on the primary.
> 3. Also in current implementation NN allows only one BN, but multiple CNs.
> So if the single BN dies the checkpointing will stall.
> You can prevent it by starting two CNs instead, or one BN and one CN.
> But I agree with you CN is just a subset of BN by its functionality.
>
> Thanks,
> Konstantin
>
>
> On 9/22/2010 5:50 PM, ChingShen wrote:
>
>> Thanks Konstantin,
>>
>> But, my main question is that because the CN can only provide an old
>> state
>> of the namespace, so why do we need it? I think the BN is best solution.
>>
>> Shen
>>
>> On Thu, Sep 23, 2010 at 5:20 AM, Konstantin Shvachko<shv@yahoo-inc.com
>> >wrote:
>>
>> The CheckpointNode creates checkpoints of the namespace, but does not
>>> keep
>>> an up-to-date state of the namespace in memory.
>>> If primary NN fails CheckpointNode can only provide an old state of the
>>> namespace
>>> created during latest checkpoint.
>>> Also CheckpointNode is a replacement for SecondaryNameNode in earlier
>>> releases.
>>>
>>> BackupNode does checkpoints too, but in addition keeps an up-to-date
>>> state
>>> of the namespace in its memory.
>>> When the primary NN dies you can ask BackupNode to save namespace, which
>>> will
>>> create the up-to-date image, and then start NN instead of BN on the node
>>> BN
>>> was running using that saved image directly or start NN on a different
>>> node
>>> using importCheckpoint from the saved inage directory.
>>>
>>> See the guide here.
>>>
>>>
>>> http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_user_guide.html#Checkpoint+Node
>>>
>>> Thanks,
>>> --Konstantin
>>>
>>>
>>> On 9/8/2010 11:36 PM, ChingShen wrote:
>>>
>>> Hi all,
>>>>
>>>> I got the Backup node(BN) that includes all the checkpoint
>>>> responsibilities,
>>>> and it maintains an up-to-date namespace state, which is always in
>>>> sync with the active NN.
>>>>
>>>> Q1. In which situation do we need a CN?
>>>>
>>>> Q2. If the NameNode machine fails, which different manual
>>>> intervention between BN and CN?
>>>>
>>>> Thanks.
>>>>
>>>> Shen
>>>>
>>>>
>>>>
>>>
>>
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Questions about BN and CN
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
Hi Shen,
Why do we need CheckpointNode?
1. First of all it is a compatible replacement of SecondaryNameNode.
2. Checkpointing is also needed for periodically compacting edits.
You can do it with CN or BN, but CN is more lightweight.
I assume there could be cases when streaming edits to BN over network
can be slower than writing them to disk, so you might want to turn BN
off for performance reasons.
3. Also in current implementation NN allows only one BN, but multiple CNs.
So if the single BN dies the checkpointing will stall.
You can prevent it by starting two CNs instead, or one BN and one CN.
But I agree with you CN is just a subset of BN by its functionality.
Thanks,
Konstantin
On 9/22/2010 5:50 PM, ChingShen wrote:
> Thanks Konstantin,
>
> But, my main question is that because the CN can only provide an old state
> of the namespace, so why do we need it? I think the BN is best solution.
>
> Shen
>
> On Thu, Sep 23, 2010 at 5:20 AM, Konstantin Shvachko<sh...@yahoo-inc.com>wrote:
>
>> The CheckpointNode creates checkpoints of the namespace, but does not keep
>> an up-to-date state of the namespace in memory.
>> If primary NN fails CheckpointNode can only provide an old state of the
>> namespace
>> created during latest checkpoint.
>> Also CheckpointNode is a replacement for SecondaryNameNode in earlier
>> releases.
>>
>> BackupNode does checkpoints too, but in addition keeps an up-to-date state
>> of the namespace in its memory.
>> When the primary NN dies you can ask BackupNode to save namespace, which
>> will
>> create the up-to-date image, and then start NN instead of BN on the node BN
>> was running using that saved image directly or start NN on a different node
>> using importCheckpoint from the saved inage directory.
>>
>> See the guide here.
>>
>> http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_user_guide.html#Checkpoint+Node
>>
>> Thanks,
>> --Konstantin
>>
>>
>> On 9/8/2010 11:36 PM, ChingShen wrote:
>>
>>> Hi all,
>>>
>>> I got the Backup node(BN) that includes all the checkpoint
>>> responsibilities,
>>> and it maintains an up-to-date namespace state, which is always in
>>> sync with the active NN.
>>>
>>> Q1. In which situation do we need a CN?
>>>
>>> Q2. If the NameNode machine fails, which different manual
>>> intervention between BN and CN?
>>>
>>> Thanks.
>>>
>>> Shen
>>>
>>>
>>
>
Re: Questions about BN and CN
Posted by ChingShen <ch...@gmail.com>.
Thanks Konstantin,
But, my main question is that because the CN can only provide an old state
of the namespace, so why do we need it? I think the BN is best solution.
Shen
On Thu, Sep 23, 2010 at 5:20 AM, Konstantin Shvachko <sh...@yahoo-inc.com>wrote:
> The CheckpointNode creates checkpoints of the namespace, but does not keep
> an up-to-date state of the namespace in memory.
> If primary NN fails CheckpointNode can only provide an old state of the
> namespace
> created during latest checkpoint.
> Also CheckpointNode is a replacement for SecondaryNameNode in earlier
> releases.
>
> BackupNode does checkpoints too, but in addition keeps an up-to-date state
> of the namespace in its memory.
> When the primary NN dies you can ask BackupNode to save namespace, which
> will
> create the up-to-date image, and then start NN instead of BN on the node BN
> was running using that saved image directly or start NN on a different node
> using importCheckpoint from the saved inage directory.
>
> See the guide here.
>
> http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_user_guide.html#Checkpoint+Node
>
> Thanks,
> --Konstantin
>
>
> On 9/8/2010 11:36 PM, ChingShen wrote:
>
>> Hi all,
>>
>> I got the Backup node(BN) that includes all the checkpoint
>> responsibilities,
>> and it maintains an up-to-date namespace state, which is always in
>> sync with the active NN.
>>
>> Q1. In which situation do we need a CN?
>>
>> Q2. If the NameNode machine fails, which different manual
>> intervention between BN and CN?
>>
>> Thanks.
>>
>> Shen
>>
>>
>
Re: Questions about BN and CN
Posted by Konstantin Shvachko <sh...@yahoo-inc.com>.
The CheckpointNode creates checkpoints of the namespace, but does not keep
an up-to-date state of the namespace in memory.
If primary NN fails CheckpointNode can only provide an old state of the namespace
created during latest checkpoint.
Also CheckpointNode is a replacement for SecondaryNameNode in earlier releases.
BackupNode does checkpoints too, but in addition keeps an up-to-date state
of the namespace in its memory.
When the primary NN dies you can ask BackupNode to save namespace, which will
create the up-to-date image, and then start NN instead of BN on the node BN
was running using that saved image directly or start NN on a different node
using importCheckpoint from the saved inage directory.
See the guide here.
http://hadoop.apache.org/hdfs/docs/r0.21.0/hdfs_user_guide.html#Checkpoint+Node
Thanks,
--Konstantin
On 9/8/2010 11:36 PM, ChingShen wrote:
> Hi all,
>
> I got the Backup node(BN) that includes all the checkpoint responsibilities,
> and it maintains an up-to-date namespace state, which is always in
> sync with the active NN.
>
> Q1. In which situation do we need a CN?
>
> Q2. If the NameNode machine fails, which different manual
> intervention between BN and CN?
>
> Thanks.
>
> Shen
>
Re: Questions about BN and CN
Posted by ChingShen <ch...@gmail.com>.
Does anyone know that in what situation do we need a CN rather than BN?
Shen