You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by "Leidle, Rob" <le...@amazon.com> on 2014/11/28 21:06:21 UTC

Re: possible bug in puppet installation?

++dev@bigtop.apache.org

From: <Leidle>, "Leidle, Rob" <le...@amazon.com>>
Reply-To: "user@bigtop.apache.org<ma...@bigtop.apache.org>" <us...@bigtop.apache.org>>
Date: Friday, November 28, 2014 at 11:40 AM
To: "user@bigtop.apache.org<ma...@bigtop.apache.org>" <us...@bigtop.apache.org>>
Subject: possible bug in puppet installation?

I am running into an issue with the puppet installation, and I think it is a bug (although I don’t want to submit a patch for it until I make sure I understand the issue completely). When I do not specify a secondary name node I am seeing both the namenode and the secondary name node being installed and configured. The bug, I believe, is in the cluster.pp file:

https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifests/cluster.pp

On line 224 the logic for secondary namenode is present:

if ($hadoop_ha == "disabled") { hadoop::secondarynamenode { "secondary namenode":
namenode_host => $hadoop_namenode_host,
namenode_port => $hadoop_namenode_port,
auth => $hadoop_security_authentication,
}
}

So, I think this is in error and the code should only execute if $hadoop_ha is not equal to “disabled”. Ie change the equals to a not equals. Can someone who understands these puppet scripts chime in and let me know if this is the appropriate patch to make? Or am I not understanding something about these installations?

Re: possible bug in puppet installation?

Posted by Konstantin Boudnik <co...@apache.org>.
That's what I said. if HA is disabled then you'll get the secondary node
installed/configured/started.

On Sat, Nov 29, 2014 at 01:23AM, Leidle, Rob wrote:
> Thank you for the reply. I would think that is how it would work. However,
> isn╧t that script starting the secondary-namenode service when ha is
> Ёdisabled╡? That╧s the behavior I am seeing. I have no configured for HA
> yet secondary namenode is being configured and started by the puppet
> scripts.
> 
> On 11/28/14, 4:07 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> 
> >The logic here is that in HA environment you can go without a secondary
> >node
> >because your standby will carry on a copy of your primary's editlogs. As
> >an
> >optimization you can cut off a checkpointing overhead.
> >
> >Hope it helps,
> >  Cos
> >
> >On Fri, Nov 28, 2014 at 08:06PM, Leidle, Rob wrote:
> >> I am running into an issue with the puppet installation, and I think it
> >>is a bug (although I don╧t want to submit a patch for it until I make
> >>sure I understand the issue completely). When I do not specify a
> >>secondary name node I am seeing both the namenode and the secondary name
> >>node being installed and configured. The bug, I believe, is in the
> >>cluster.pp file:
> >> 
> >> 
> >>https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifes
> >>ts/cluster.pp
> >> 
> >> On line 224 the logic for secondary namenode is present:
> >> 
> >> if ($hadoop_ha == "disabled") { hadoop::secondarynamenode { "secondary
> >>namenode":
> >> namenode_host => $hadoop_namenode_host,
> >> namenode_port => $hadoop_namenode_port,
> >> auth => $hadoop_security_authentication,
> >> }
> >> }
> >> 
> >> So, I think this is in error and the code should only execute if
> >>$hadoop_ha is not equal to Ёdisabled╡. Ie change the equals to a not
> >>equals. Can someone who understands these puppet scripts chime in and
> >>let me know if this is the appropriate patch to make? Or am I not
> >>understanding something about these installations?
> 

Re: possible bug in puppet installation?

Posted by Konstantin Boudnik <co...@apache.org>.
That's what I said. if HA is disabled then you'll get the secondary node
installed/configured/started.

On Sat, Nov 29, 2014 at 01:23AM, Leidle, Rob wrote:
> Thank you for the reply. I would think that is how it would work. However,
> isn╧t that script starting the secondary-namenode service when ha is
> Ёdisabled╡? That╧s the behavior I am seeing. I have no configured for HA
> yet secondary namenode is being configured and started by the puppet
> scripts.
> 
> On 11/28/14, 4:07 PM, "Konstantin Boudnik" <co...@apache.org> wrote:
> 
> >The logic here is that in HA environment you can go without a secondary
> >node
> >because your standby will carry on a copy of your primary's editlogs. As
> >an
> >optimization you can cut off a checkpointing overhead.
> >
> >Hope it helps,
> >  Cos
> >
> >On Fri, Nov 28, 2014 at 08:06PM, Leidle, Rob wrote:
> >> I am running into an issue with the puppet installation, and I think it
> >>is a bug (although I don╧t want to submit a patch for it until I make
> >>sure I understand the issue completely). When I do not specify a
> >>secondary name node I am seeing both the namenode and the secondary name
> >>node being installed and configured. The bug, I believe, is in the
> >>cluster.pp file:
> >> 
> >> 
> >>https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifes
> >>ts/cluster.pp
> >> 
> >> On line 224 the logic for secondary namenode is present:
> >> 
> >> if ($hadoop_ha == "disabled") { hadoop::secondarynamenode { "secondary
> >>namenode":
> >> namenode_host => $hadoop_namenode_host,
> >> namenode_port => $hadoop_namenode_port,
> >> auth => $hadoop_security_authentication,
> >> }
> >> }
> >> 
> >> So, I think this is in error and the code should only execute if
> >>$hadoop_ha is not equal to Ёdisabled╡. Ie change the equals to a not
> >>equals. Can someone who understands these puppet scripts chime in and
> >>let me know if this is the appropriate patch to make? Or am I not
> >>understanding something about these installations?
> 

Re: possible bug in puppet installation?

Posted by "Leidle, Rob" <le...@amazon.com>.
You are correct. That is what I was confused about and it made sense after I researched the purpose of secondary namenode. 



> On Nov 29, 2014, at 3:44 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> 
>> On Fri, Nov 28, 2014 at 5:23 PM, Leidle, Rob <le...@amazon.com> wrote:
>> Thank you for the reply. I would think that is how it would work. However,
>> isn¹t that script starting the secondary-namenode service when ha is
>> ³disabled²? That¹s the behavior I am seeing. I have no configured for HA
>> yet secondary namenode is being configured and started by the puppet
>> scripts.
> 
> I think you're being confused by the terminology here. Seconadary name
> node is a very old piece of HDFS and it has little to do with HA. When HA
> was implemented in HDFS a notion of a stand-by namemode was introduced.
> With that a need for a secondary namenode disappeared.
> 
> Thanks,
> Roman.

Re: possible bug in puppet installation?

Posted by "Leidle, Rob" <le...@amazon.com>.
You are correct. That is what I was confused about and it made sense after I researched the purpose of secondary namenode. 



> On Nov 29, 2014, at 3:44 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> 
>> On Fri, Nov 28, 2014 at 5:23 PM, Leidle, Rob <le...@amazon.com> wrote:
>> Thank you for the reply. I would think that is how it would work. However,
>> isn¹t that script starting the secondary-namenode service when ha is
>> ³disabled²? That¹s the behavior I am seeing. I have no configured for HA
>> yet secondary namenode is being configured and started by the puppet
>> scripts.
> 
> I think you're being confused by the terminology here. Seconadary name
> node is a very old piece of HDFS and it has little to do with HA. When HA
> was implemented in HDFS a notion of a stand-by namemode was introduced.
> With that a need for a secondary namenode disappeared.
> 
> Thanks,
> Roman.

Re: possible bug in puppet installation?

Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Nov 28, 2014 at 5:23 PM, Leidle, Rob <le...@amazon.com> wrote:
> Thank you for the reply. I would think that is how it would work. However,
> isn¹t that script starting the secondary-namenode service when ha is
> ³disabled²? That¹s the behavior I am seeing. I have no configured for HA
> yet secondary namenode is being configured and started by the puppet
> scripts.

I think you're being confused by the terminology here. Seconadary name
node is a very old piece of HDFS and it has little to do with HA. When HA
was implemented in HDFS a notion of a stand-by namemode was introduced.
With that a need for a secondary namenode disappeared.

Thanks,
Roman.

Re: possible bug in puppet installation?

Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Nov 28, 2014 at 5:23 PM, Leidle, Rob <le...@amazon.com> wrote:
> Thank you for the reply. I would think that is how it would work. However,
> isn¹t that script starting the secondary-namenode service when ha is
> ³disabled²? That¹s the behavior I am seeing. I have no configured for HA
> yet secondary namenode is being configured and started by the puppet
> scripts.

I think you're being confused by the terminology here. Seconadary name
node is a very old piece of HDFS and it has little to do with HA. When HA
was implemented in HDFS a notion of a stand-by namemode was introduced.
With that a need for a secondary namenode disappeared.

Thanks,
Roman.

Re: possible bug in puppet installation?

Posted by "Leidle, Rob" <le...@amazon.com>.
Thank you for the reply. I would think that is how it would work. However,
isn¹t that script starting the secondary-namenode service when ha is
³disabled²? That¹s the behavior I am seeing. I have no configured for HA
yet secondary namenode is being configured and started by the puppet
scripts.

On 11/28/14, 4:07 PM, "Konstantin Boudnik" <co...@apache.org> wrote:

>The logic here is that in HA environment you can go without a secondary
>node
>because your standby will carry on a copy of your primary's editlogs. As
>an
>optimization you can cut off a checkpointing overhead.
>
>Hope it helps,
>  Cos
>
>On Fri, Nov 28, 2014 at 08:06PM, Leidle, Rob wrote:
>> I am running into an issue with the puppet installation, and I think it
>>is a bug (although I don¹t want to submit a patch for it until I make
>>sure I understand the issue completely). When I do not specify a
>>secondary name node I am seeing both the namenode and the secondary name
>>node being installed and configured. The bug, I believe, is in the
>>cluster.pp file:
>> 
>> 
>>https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifes
>>ts/cluster.pp
>> 
>> On line 224 the logic for secondary namenode is present:
>> 
>> if ($hadoop_ha == "disabled") { hadoop::secondarynamenode { "secondary
>>namenode":
>> namenode_host => $hadoop_namenode_host,
>> namenode_port => $hadoop_namenode_port,
>> auth => $hadoop_security_authentication,
>> }
>> }
>> 
>> So, I think this is in error and the code should only execute if
>>$hadoop_ha is not equal to ³disabled². Ie change the equals to a not
>>equals. Can someone who understands these puppet scripts chime in and
>>let me know if this is the appropriate patch to make? Or am I not
>>understanding something about these installations?


Re: possible bug in puppet installation?

Posted by "Leidle, Rob" <le...@amazon.com>.
Thank you for the reply. I would think that is how it would work. However,
isn¹t that script starting the secondary-namenode service when ha is
³disabled²? That¹s the behavior I am seeing. I have no configured for HA
yet secondary namenode is being configured and started by the puppet
scripts.

On 11/28/14, 4:07 PM, "Konstantin Boudnik" <co...@apache.org> wrote:

>The logic here is that in HA environment you can go without a secondary
>node
>because your standby will carry on a copy of your primary's editlogs. As
>an
>optimization you can cut off a checkpointing overhead.
>
>Hope it helps,
>  Cos
>
>On Fri, Nov 28, 2014 at 08:06PM, Leidle, Rob wrote:
>> I am running into an issue with the puppet installation, and I think it
>>is a bug (although I don¹t want to submit a patch for it until I make
>>sure I understand the issue completely). When I do not specify a
>>secondary name node I am seeing both the namenode and the secondary name
>>node being installed and configured. The bug, I believe, is in the
>>cluster.pp file:
>> 
>> 
>>https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifes
>>ts/cluster.pp
>> 
>> On line 224 the logic for secondary namenode is present:
>> 
>> if ($hadoop_ha == "disabled") { hadoop::secondarynamenode { "secondary
>>namenode":
>> namenode_host => $hadoop_namenode_host,
>> namenode_port => $hadoop_namenode_port,
>> auth => $hadoop_security_authentication,
>> }
>> }
>> 
>> So, I think this is in error and the code should only execute if
>>$hadoop_ha is not equal to ³disabled². Ie change the equals to a not
>>equals. Can someone who understands these puppet scripts chime in and
>>let me know if this is the appropriate patch to make? Or am I not
>>understanding something about these installations?


Re: possible bug in puppet installation?

Posted by Konstantin Boudnik <co...@apache.org>.
The logic here is that in HA environment you can go without a secondary node
because your standby will carry on a copy of your primary's editlogs. As an
optimization you can cut off a checkpointing overhead.

Hope it helps,
  Cos

On Fri, Nov 28, 2014 at 08:06PM, Leidle, Rob wrote:
> I am running into an issue with the puppet installation, and I think it is a bug (although I don’t want to submit a patch for it until I make sure I understand the issue completely). When I do not specify a secondary name node I am seeing both the namenode and the secondary name node being installed and configured. The bug, I believe, is in the cluster.pp file:
> 
> https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifests/cluster.pp
> 
> On line 224 the logic for secondary namenode is present:
> 
> if ($hadoop_ha == "disabled") { hadoop::secondarynamenode { "secondary namenode":
> namenode_host => $hadoop_namenode_host,
> namenode_port => $hadoop_namenode_port,
> auth => $hadoop_security_authentication,
> }
> }
> 
> So, I think this is in error and the code should only execute if $hadoop_ha is not equal to “disabled”. Ie change the equals to a not equals. Can someone who understands these puppet scripts chime in and let me know if this is the appropriate patch to make? Or am I not understanding something about these installations?

Re: possible bug in puppet installation?

Posted by Konstantin Boudnik <co...@apache.org>.
The logic here is that in HA environment you can go without a secondary node
because your standby will carry on a copy of your primary's editlogs. As an
optimization you can cut off a checkpointing overhead.

Hope it helps,
  Cos

On Fri, Nov 28, 2014 at 08:06PM, Leidle, Rob wrote:
> I am running into an issue with the puppet installation, and I think it is a bug (although I don’t want to submit a patch for it until I make sure I understand the issue completely). When I do not specify a secondary name node I am seeing both the namenode and the secondary name node being installed and configured. The bug, I believe, is in the cluster.pp file:
> 
> https://github.com/apache/bigtop/blob/master/bigtop-deploy/puppet/manifests/cluster.pp
> 
> On line 224 the logic for secondary namenode is present:
> 
> if ($hadoop_ha == "disabled") { hadoop::secondarynamenode { "secondary namenode":
> namenode_host => $hadoop_namenode_host,
> namenode_port => $hadoop_namenode_port,
> auth => $hadoop_security_authentication,
> }
> }
> 
> So, I think this is in error and the code should only execute if $hadoop_ha is not equal to “disabled”. Ie change the equals to a not equals. Can someone who understands these puppet scripts chime in and let me know if this is the appropriate patch to make? Or am I not understanding something about these installations?