You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by YouPeng Yang <yy...@gmail.com> on 2013/02/24 16:09:24 UTC
One NameNode keeps Rolling Edit Log on HDFS Federation
Hi All
I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
I find that it is Rolling Edit Log continuously on one namenode,however
the other one changes nothing.
My question :
1. Is it the right situation.
2. I have thought that the two namenodes shoud keep concurrency.
why they get differences.
3. is there any advanced docs about HDFS Federation.
thanks.
Regards.
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Mike,Harsh
Thank you for your reply.
I need to time to digest the above stuff.
Thanks
Regards
2013/2/25 Harsh J <ha...@cloudera.com>
> Mike,
>
> I don't see how SPOF comes into the picture when HA is already present
> in the releases that also carry Federation and each NN (federated or
> non) can be assigned further Standby-NN roles. At this point we can
> stop using the word "SPOF" completely for HDFS. Would do great good
> for avoiding further FUD around this :)
>
> The federated NameNodes are namespace-divided; and typically I think
> users may use it to divide applications into using different
> namespaces each, while using the same set of compute and data nodes.
> So if HA isn't deployed, an application may be lost but not "portion
> of the cluster", since DNs will remain unaffected and so would the
> other namespaces.
>
> On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
> <mi...@hotmail.com> wrote:
> > I think part of the confusion stems from the fact that federation of
> name nodes only splits the very large cluster in to smaller portions of the
> same cluster. If you lose a federated name node, you only lose a portion of
> the cluster not the whole thing. So now instead of one SPOF, you have two
> SPOFs.
> >
> > The advantage of Federation is that you reduce the amount of memory
> required for the NN.
> >
> >
> > Sent from a remote device. Please excuse any typos...
> >
> > Mike Segel
> >
> > On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> >
> >> Hi,
> >>
> >> Federated namenodes are independent of one another (except that they
> >> both get reports from all the/common DNs in the cluster). It is
> >> natural to see one roll its edit logs based on its own rate of
> >> metadata growth, as compared to the other. Their edits, image, etc.
> >> everything is independent - they also do not know about/talk to each
> >> other.
> >>
> >> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> >> be found in PDFs attached to
> >> https://issues.apache.org/jira/browse/HDFS-1052.
> >>
> >> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <
> yypvsxf19870706@gmail.com> wrote:
> >>> Hi All
> >>>
> >>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> >>> I find that it is Rolling Edit Log continuously on one
> namenode,however
> >>> the other one changes nothing.
> >>>
> >>> My question :
> >>> 1. Is it the right situation.
> >>> 2. I have thought that the two namenodes shoud keep concurrency.
> >>> why they get differences.
> >>> 3. is there any advanced docs about HDFS Federation.
> >>>
> >>> thanks.
> >>>
> >>> Regards.
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Mike,Harsh
Thank you for your reply.
I need to time to digest the above stuff.
Thanks
Regards
2013/2/25 Harsh J <ha...@cloudera.com>
> Mike,
>
> I don't see how SPOF comes into the picture when HA is already present
> in the releases that also carry Federation and each NN (federated or
> non) can be assigned further Standby-NN roles. At this point we can
> stop using the word "SPOF" completely for HDFS. Would do great good
> for avoiding further FUD around this :)
>
> The federated NameNodes are namespace-divided; and typically I think
> users may use it to divide applications into using different
> namespaces each, while using the same set of compute and data nodes.
> So if HA isn't deployed, an application may be lost but not "portion
> of the cluster", since DNs will remain unaffected and so would the
> other namespaces.
>
> On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
> <mi...@hotmail.com> wrote:
> > I think part of the confusion stems from the fact that federation of
> name nodes only splits the very large cluster in to smaller portions of the
> same cluster. If you lose a federated name node, you only lose a portion of
> the cluster not the whole thing. So now instead of one SPOF, you have two
> SPOFs.
> >
> > The advantage of Federation is that you reduce the amount of memory
> required for the NN.
> >
> >
> > Sent from a remote device. Please excuse any typos...
> >
> > Mike Segel
> >
> > On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> >
> >> Hi,
> >>
> >> Federated namenodes are independent of one another (except that they
> >> both get reports from all the/common DNs in the cluster). It is
> >> natural to see one roll its edit logs based on its own rate of
> >> metadata growth, as compared to the other. Their edits, image, etc.
> >> everything is independent - they also do not know about/talk to each
> >> other.
> >>
> >> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> >> be found in PDFs attached to
> >> https://issues.apache.org/jira/browse/HDFS-1052.
> >>
> >> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <
> yypvsxf19870706@gmail.com> wrote:
> >>> Hi All
> >>>
> >>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> >>> I find that it is Rolling Edit Log continuously on one
> namenode,however
> >>> the other one changes nothing.
> >>>
> >>> My question :
> >>> 1. Is it the right situation.
> >>> 2. I have thought that the two namenodes shoud keep concurrency.
> >>> why they get differences.
> >>> 3. is there any advanced docs about HDFS Federation.
> >>>
> >>> thanks.
> >>>
> >>> Regards.
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Mike,Harsh
Thank you for your reply.
I need to time to digest the above stuff.
Thanks
Regards
2013/2/25 Harsh J <ha...@cloudera.com>
> Mike,
>
> I don't see how SPOF comes into the picture when HA is already present
> in the releases that also carry Federation and each NN (federated or
> non) can be assigned further Standby-NN roles. At this point we can
> stop using the word "SPOF" completely for HDFS. Would do great good
> for avoiding further FUD around this :)
>
> The federated NameNodes are namespace-divided; and typically I think
> users may use it to divide applications into using different
> namespaces each, while using the same set of compute and data nodes.
> So if HA isn't deployed, an application may be lost but not "portion
> of the cluster", since DNs will remain unaffected and so would the
> other namespaces.
>
> On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
> <mi...@hotmail.com> wrote:
> > I think part of the confusion stems from the fact that federation of
> name nodes only splits the very large cluster in to smaller portions of the
> same cluster. If you lose a federated name node, you only lose a portion of
> the cluster not the whole thing. So now instead of one SPOF, you have two
> SPOFs.
> >
> > The advantage of Federation is that you reduce the amount of memory
> required for the NN.
> >
> >
> > Sent from a remote device. Please excuse any typos...
> >
> > Mike Segel
> >
> > On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> >
> >> Hi,
> >>
> >> Federated namenodes are independent of one another (except that they
> >> both get reports from all the/common DNs in the cluster). It is
> >> natural to see one roll its edit logs based on its own rate of
> >> metadata growth, as compared to the other. Their edits, image, etc.
> >> everything is independent - they also do not know about/talk to each
> >> other.
> >>
> >> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> >> be found in PDFs attached to
> >> https://issues.apache.org/jira/browse/HDFS-1052.
> >>
> >> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <
> yypvsxf19870706@gmail.com> wrote:
> >>> Hi All
> >>>
> >>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> >>> I find that it is Rolling Edit Log continuously on one
> namenode,however
> >>> the other one changes nothing.
> >>>
> >>> My question :
> >>> 1. Is it the right situation.
> >>> 2. I have thought that the two namenodes shoud keep concurrency.
> >>> why they get differences.
> >>> 3. is there any advanced docs about HDFS Federation.
> >>>
> >>> thanks.
> >>>
> >>> Regards.
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Mike,Harsh
Thank you for your reply.
I need to time to digest the above stuff.
Thanks
Regards
2013/2/25 Harsh J <ha...@cloudera.com>
> Mike,
>
> I don't see how SPOF comes into the picture when HA is already present
> in the releases that also carry Federation and each NN (federated or
> non) can be assigned further Standby-NN roles. At this point we can
> stop using the word "SPOF" completely for HDFS. Would do great good
> for avoiding further FUD around this :)
>
> The federated NameNodes are namespace-divided; and typically I think
> users may use it to divide applications into using different
> namespaces each, while using the same set of compute and data nodes.
> So if HA isn't deployed, an application may be lost but not "portion
> of the cluster", since DNs will remain unaffected and so would the
> other namespaces.
>
> On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
> <mi...@hotmail.com> wrote:
> > I think part of the confusion stems from the fact that federation of
> name nodes only splits the very large cluster in to smaller portions of the
> same cluster. If you lose a federated name node, you only lose a portion of
> the cluster not the whole thing. So now instead of one SPOF, you have two
> SPOFs.
> >
> > The advantage of Federation is that you reduce the amount of memory
> required for the NN.
> >
> >
> > Sent from a remote device. Please excuse any typos...
> >
> > Mike Segel
> >
> > On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> >
> >> Hi,
> >>
> >> Federated namenodes are independent of one another (except that they
> >> both get reports from all the/common DNs in the cluster). It is
> >> natural to see one roll its edit logs based on its own rate of
> >> metadata growth, as compared to the other. Their edits, image, etc.
> >> everything is independent - they also do not know about/talk to each
> >> other.
> >>
> >> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> >> be found in PDFs attached to
> >> https://issues.apache.org/jira/browse/HDFS-1052.
> >>
> >> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <
> yypvsxf19870706@gmail.com> wrote:
> >>> Hi All
> >>>
> >>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> >>> I find that it is Rolling Edit Log continuously on one
> namenode,however
> >>> the other one changes nothing.
> >>>
> >>> My question :
> >>> 1. Is it the right situation.
> >>> 2. I have thought that the two namenodes shoud keep concurrency.
> >>> why they get differences.
> >>> 3. is there any advanced docs about HDFS Federation.
> >>>
> >>> thanks.
> >>>
> >>> Regards.
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Mike,
I don't see how SPOF comes into the picture when HA is already present
in the releases that also carry Federation and each NN (federated or
non) can be assigned further Standby-NN roles. At this point we can
stop using the word "SPOF" completely for HDFS. Would do great good
for avoiding further FUD around this :)
The federated NameNodes are namespace-divided; and typically I think
users may use it to divide applications into using different
namespaces each, while using the same set of compute and data nodes.
So if HA isn't deployed, an application may be lost but not "portion
of the cluster", since DNs will remain unaffected and so would the
other namespaces.
On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
<mi...@hotmail.com> wrote:
> I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
>
> The advantage of Federation is that you reduce the amount of memory required for the NN.
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> Federated namenodes are independent of one another (except that they
>> both get reports from all the/common DNs in the cluster). It is
>> natural to see one roll its edit logs based on its own rate of
>> metadata growth, as compared to the other. Their edits, image, etc.
>> everything is independent - they also do not know about/talk to each
>> other.
>>
>> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
>> be found in PDFs attached to
>> https://issues.apache.org/jira/browse/HDFS-1052.
>>
>> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one namenode,however
>>> the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>
>>
>>
>> --
>> Harsh J
>>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Mike,
I don't see how SPOF comes into the picture when HA is already present
in the releases that also carry Federation and each NN (federated or
non) can be assigned further Standby-NN roles. At this point we can
stop using the word "SPOF" completely for HDFS. Would do great good
for avoiding further FUD around this :)
The federated NameNodes are namespace-divided; and typically I think
users may use it to divide applications into using different
namespaces each, while using the same set of compute and data nodes.
So if HA isn't deployed, an application may be lost but not "portion
of the cluster", since DNs will remain unaffected and so would the
other namespaces.
On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
<mi...@hotmail.com> wrote:
> I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
>
> The advantage of Federation is that you reduce the amount of memory required for the NN.
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> Federated namenodes are independent of one another (except that they
>> both get reports from all the/common DNs in the cluster). It is
>> natural to see one roll its edit logs based on its own rate of
>> metadata growth, as compared to the other. Their edits, image, etc.
>> everything is independent - they also do not know about/talk to each
>> other.
>>
>> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
>> be found in PDFs attached to
>> https://issues.apache.org/jira/browse/HDFS-1052.
>>
>> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one namenode,however
>>> the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>
>>
>>
>> --
>> Harsh J
>>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Mike,
I don't see how SPOF comes into the picture when HA is already present
in the releases that also carry Federation and each NN (federated or
non) can be assigned further Standby-NN roles. At this point we can
stop using the word "SPOF" completely for HDFS. Would do great good
for avoiding further FUD around this :)
The federated NameNodes are namespace-divided; and typically I think
users may use it to divide applications into using different
namespaces each, while using the same set of compute and data nodes.
So if HA isn't deployed, an application may be lost but not "portion
of the cluster", since DNs will remain unaffected and so would the
other namespaces.
On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
<mi...@hotmail.com> wrote:
> I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
>
> The advantage of Federation is that you reduce the amount of memory required for the NN.
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> Federated namenodes are independent of one another (except that they
>> both get reports from all the/common DNs in the cluster). It is
>> natural to see one roll its edit logs based on its own rate of
>> metadata growth, as compared to the other. Their edits, image, etc.
>> everything is independent - they also do not know about/talk to each
>> other.
>>
>> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
>> be found in PDFs attached to
>> https://issues.apache.org/jira/browse/HDFS-1052.
>>
>> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one namenode,however
>>> the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>
>>
>>
>> --
>> Harsh J
>>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Mike,
I don't see how SPOF comes into the picture when HA is already present
in the releases that also carry Federation and each NN (federated or
non) can be assigned further Standby-NN roles. At this point we can
stop using the word "SPOF" completely for HDFS. Would do great good
for avoiding further FUD around this :)
The federated NameNodes are namespace-divided; and typically I think
users may use it to divide applications into using different
namespaces each, while using the same set of compute and data nodes.
So if HA isn't deployed, an application may be lost but not "portion
of the cluster", since DNs will remain unaffected and so would the
other namespaces.
On Sun, Feb 24, 2013 at 11:23 PM, Michel Segel
<mi...@hotmail.com> wrote:
> I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
>
> The advantage of Federation is that you reduce the amount of memory required for the NN.
>
>
> Sent from a remote device. Please excuse any typos...
>
> Mike Segel
>
> On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
>
>> Hi,
>>
>> Federated namenodes are independent of one another (except that they
>> both get reports from all the/common DNs in the cluster). It is
>> natural to see one roll its edit logs based on its own rate of
>> metadata growth, as compared to the other. Their edits, image, etc.
>> everything is independent - they also do not know about/talk to each
>> other.
>>
>> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
>> be found in PDFs attached to
>> https://issues.apache.org/jira/browse/HDFS-1052.
>>
>> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one namenode,however
>>> the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>
>>
>>
>> --
>> Harsh J
>>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Michel Segel <mi...@hotmail.com>.
I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
The advantage of Federation is that you reduce the amount of memory required for the NN.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> Federated namenodes are independent of one another (except that they
> both get reports from all the/common DNs in the cluster). It is
> natural to see one roll its edit logs based on its own rate of
> metadata growth, as compared to the other. Their edits, image, etc.
> everything is independent - they also do not know about/talk to each
> other.
>
> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> be found in PDFs attached to
> https://issues.apache.org/jira/browse/HDFS-1052.
>
> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one namenode,however
>> the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Michel Segel <mi...@hotmail.com>.
I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
The advantage of Federation is that you reduce the amount of memory required for the NN.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> Federated namenodes are independent of one another (except that they
> both get reports from all the/common DNs in the cluster). It is
> natural to see one roll its edit logs based on its own rate of
> metadata growth, as compared to the other. Their edits, image, etc.
> everything is independent - they also do not know about/talk to each
> other.
>
> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> be found in PDFs attached to
> https://issues.apache.org/jira/browse/HDFS-1052.
>
> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one namenode,however
>> the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Michel Segel <mi...@hotmail.com>.
I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
The advantage of Federation is that you reduce the amount of memory required for the NN.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> Federated namenodes are independent of one another (except that they
> both get reports from all the/common DNs in the cluster). It is
> natural to see one roll its edit logs based on its own rate of
> metadata growth, as compared to the other. Their edits, image, etc.
> everything is independent - they also do not know about/talk to each
> other.
>
> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> be found in PDFs attached to
> https://issues.apache.org/jira/browse/HDFS-1052.
>
> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one namenode,however
>> the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Michel Segel <mi...@hotmail.com>.
I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two SPOFs.
The advantage of Federation is that you reduce the amount of memory required for the NN.
Sent from a remote device. Please excuse any typos...
Mike Segel
On Feb 24, 2013, at 10:48 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> Federated namenodes are independent of one another (except that they
> both get reports from all the/common DNs in the cluster). It is
> natural to see one roll its edit logs based on its own rate of
> metadata growth, as compared to the other. Their edits, image, etc.
> everything is independent - they also do not know about/talk to each
> other.
>
> Given the above, I'd say (1) is true and (2) is invalid, and (3) can
> be found in PDFs attached to
> https://issues.apache.org/jira/browse/HDFS-1052.
>
> On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one namenode,however
>> the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>
>
>
> --
> Harsh J
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Federated namenodes are independent of one another (except that they
both get reports from all the/common DNs in the cluster). It is
natural to see one roll its edit logs based on its own rate of
metadata growth, as compared to the other. Their edits, image, etc.
everything is independent - they also do not know about/talk to each
other.
Given the above, I'd say (1) is true and (2) is invalid, and (3) can
be found in PDFs attached to
https://issues.apache.org/jira/browse/HDFS-1052.
On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one namenode,however
> the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Federated namenodes are independent of one another (except that they
both get reports from all the/common DNs in the cluster). It is
natural to see one roll its edit logs based on its own rate of
metadata growth, as compared to the other. Their edits, image, etc.
everything is independent - they also do not know about/talk to each
other.
Given the above, I'd say (1) is true and (2) is invalid, and (3) can
be found in PDFs attached to
https://issues.apache.org/jira/browse/HDFS-1052.
On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one namenode,however
> the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Could you send us the log messages (with timestamps) which you think
is behind the behavior you see? The new edit log format is different
than the one in 1.x, and uses several smaller edit log files. This new
format is described in PDFs attached at
https://issues.apache.org/jira/browse/HDFS-1073, and the logs may mean
that the NN just opened a new file and closed the older transactions
file.
On Mon, Feb 25, 2013 at 10:16 AM, YouPeng Yang
<yy...@gmail.com> wrote:
> Hi Azuryy,Harsh
>
> It is strange that the NameNode rolling edits log rolls edits logs all
> the times,even i do nothing on the NameNode vi client.
>
> Any futher help will be appreciated.
>
>
> regard.
>
>
>
>
>
>
> 2013/2/25 YouPeng Yang <yy...@gmail.com>
>>
>> Hi Azuryy
>>
>> Yes.that is what is.
>> Thank you for your reply. I am making effort to make clear about
>> federation and HA.
>>
>> Regards.
>>
>>
>>
>>
>> 2013/2/25 Azuryy Yu <az...@gmail.com>
>>>
>>> I think you mixed federation with HA. am I right?
>>>
>>> If another name node hasn't changes, then It doesn't do any edit log
>>> rolling. federated NNs don't keep concurrency( I think you want say keep
>>> sync-able?)
>>>
>>>
>>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang
>>> <yy...@gmail.com> wrote:
>>>>
>>>> Hi All
>>>>
>>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>>> I find that it is Rolling Edit Log continuously on one
>>>> namenode,however the other one changes nothing.
>>>>
>>>> My question :
>>>> 1. Is it the right situation.
>>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>>> why they get differences.
>>>> 3. is there any advanced docs about HDFS Federation.
>>>>
>>>> thanks.
>>>>
>>>> Regards.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Could you send us the log messages (with timestamps) which you think
is behind the behavior you see? The new edit log format is different
than the one in 1.x, and uses several smaller edit log files. This new
format is described in PDFs attached at
https://issues.apache.org/jira/browse/HDFS-1073, and the logs may mean
that the NN just opened a new file and closed the older transactions
file.
On Mon, Feb 25, 2013 at 10:16 AM, YouPeng Yang
<yy...@gmail.com> wrote:
> Hi Azuryy,Harsh
>
> It is strange that the NameNode rolling edits log rolls edits logs all
> the times,even i do nothing on the NameNode vi client.
>
> Any futher help will be appreciated.
>
>
> regard.
>
>
>
>
>
>
> 2013/2/25 YouPeng Yang <yy...@gmail.com>
>>
>> Hi Azuryy
>>
>> Yes.that is what is.
>> Thank you for your reply. I am making effort to make clear about
>> federation and HA.
>>
>> Regards.
>>
>>
>>
>>
>> 2013/2/25 Azuryy Yu <az...@gmail.com>
>>>
>>> I think you mixed federation with HA. am I right?
>>>
>>> If another name node hasn't changes, then It doesn't do any edit log
>>> rolling. federated NNs don't keep concurrency( I think you want say keep
>>> sync-able?)
>>>
>>>
>>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang
>>> <yy...@gmail.com> wrote:
>>>>
>>>> Hi All
>>>>
>>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>>> I find that it is Rolling Edit Log continuously on one
>>>> namenode,however the other one changes nothing.
>>>>
>>>> My question :
>>>> 1. Is it the right situation.
>>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>>> why they get differences.
>>>> 3. is there any advanced docs about HDFS Federation.
>>>>
>>>> thanks.
>>>>
>>>> Regards.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Could you send us the log messages (with timestamps) which you think
is behind the behavior you see? The new edit log format is different
than the one in 1.x, and uses several smaller edit log files. This new
format is described in PDFs attached at
https://issues.apache.org/jira/browse/HDFS-1073, and the logs may mean
that the NN just opened a new file and closed the older transactions
file.
On Mon, Feb 25, 2013 at 10:16 AM, YouPeng Yang
<yy...@gmail.com> wrote:
> Hi Azuryy,Harsh
>
> It is strange that the NameNode rolling edits log rolls edits logs all
> the times,even i do nothing on the NameNode vi client.
>
> Any futher help will be appreciated.
>
>
> regard.
>
>
>
>
>
>
> 2013/2/25 YouPeng Yang <yy...@gmail.com>
>>
>> Hi Azuryy
>>
>> Yes.that is what is.
>> Thank you for your reply. I am making effort to make clear about
>> federation and HA.
>>
>> Regards.
>>
>>
>>
>>
>> 2013/2/25 Azuryy Yu <az...@gmail.com>
>>>
>>> I think you mixed federation with HA. am I right?
>>>
>>> If another name node hasn't changes, then It doesn't do any edit log
>>> rolling. federated NNs don't keep concurrency( I think you want say keep
>>> sync-able?)
>>>
>>>
>>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang
>>> <yy...@gmail.com> wrote:
>>>>
>>>> Hi All
>>>>
>>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>>> I find that it is Rolling Edit Log continuously on one
>>>> namenode,however the other one changes nothing.
>>>>
>>>> My question :
>>>> 1. Is it the right situation.
>>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>>> why they get differences.
>>>> 3. is there any advanced docs about HDFS Federation.
>>>>
>>>> thanks.
>>>>
>>>> Regards.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Could you send us the log messages (with timestamps) which you think
is behind the behavior you see? The new edit log format is different
than the one in 1.x, and uses several smaller edit log files. This new
format is described in PDFs attached at
https://issues.apache.org/jira/browse/HDFS-1073, and the logs may mean
that the NN just opened a new file and closed the older transactions
file.
On Mon, Feb 25, 2013 at 10:16 AM, YouPeng Yang
<yy...@gmail.com> wrote:
> Hi Azuryy,Harsh
>
> It is strange that the NameNode rolling edits log rolls edits logs all
> the times,even i do nothing on the NameNode vi client.
>
> Any futher help will be appreciated.
>
>
> regard.
>
>
>
>
>
>
> 2013/2/25 YouPeng Yang <yy...@gmail.com>
>>
>> Hi Azuryy
>>
>> Yes.that is what is.
>> Thank you for your reply. I am making effort to make clear about
>> federation and HA.
>>
>> Regards.
>>
>>
>>
>>
>> 2013/2/25 Azuryy Yu <az...@gmail.com>
>>>
>>> I think you mixed federation with HA. am I right?
>>>
>>> If another name node hasn't changes, then It doesn't do any edit log
>>> rolling. federated NNs don't keep concurrency( I think you want say keep
>>> sync-able?)
>>>
>>>
>>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang
>>> <yy...@gmail.com> wrote:
>>>>
>>>> Hi All
>>>>
>>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>>> I find that it is Rolling Edit Log continuously on one
>>>> namenode,however the other one changes nothing.
>>>>
>>>> My question :
>>>> 1. Is it the right situation.
>>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>>> why they get differences.
>>>> 3. is there any advanced docs about HDFS Federation.
>>>>
>>>> thanks.
>>>>
>>>> Regards.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy,Harsh
It is strange that the NameNode rolling edits log rolls edits logs all
the times,even i do nothing on the NameNode vi client.
Any futher help will be appreciated.
regard.
2013/2/25 YouPeng Yang <yy...@gmail.com>
> Hi Azuryy
>
> Yes.that is what is.
> Thank you for your reply. I am making effort to make clear about
> federation and HA.
>
> Regards.
>
>
>
>
> 2013/2/25 Azuryy Yu <az...@gmail.com>
>
>> I think you mixed federation with HA. am I right?
>>
>> If another name node hasn't changes, then It doesn't do any edit log
>> rolling. federated NNs don't keep concurrency( I think you want say keep
>> sync-able?)
>>
>>
>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yypvsxf19870706@gmail.com
>> > wrote:
>>
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one
>>> namenode,however the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>>
>>>
>>>
>>>
>>
>>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy,Harsh
It is strange that the NameNode rolling edits log rolls edits logs all
the times,even i do nothing on the NameNode vi client.
Any futher help will be appreciated.
regard.
2013/2/25 YouPeng Yang <yy...@gmail.com>
> Hi Azuryy
>
> Yes.that is what is.
> Thank you for your reply. I am making effort to make clear about
> federation and HA.
>
> Regards.
>
>
>
>
> 2013/2/25 Azuryy Yu <az...@gmail.com>
>
>> I think you mixed federation with HA. am I right?
>>
>> If another name node hasn't changes, then It doesn't do any edit log
>> rolling. federated NNs don't keep concurrency( I think you want say keep
>> sync-able?)
>>
>>
>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yypvsxf19870706@gmail.com
>> > wrote:
>>
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one
>>> namenode,however the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>>
>>>
>>>
>>>
>>
>>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy,Harsh
It is strange that the NameNode rolling edits log rolls edits logs all
the times,even i do nothing on the NameNode vi client.
Any futher help will be appreciated.
regard.
2013/2/25 YouPeng Yang <yy...@gmail.com>
> Hi Azuryy
>
> Yes.that is what is.
> Thank you for your reply. I am making effort to make clear about
> federation and HA.
>
> Regards.
>
>
>
>
> 2013/2/25 Azuryy Yu <az...@gmail.com>
>
>> I think you mixed federation with HA. am I right?
>>
>> If another name node hasn't changes, then It doesn't do any edit log
>> rolling. federated NNs don't keep concurrency( I think you want say keep
>> sync-able?)
>>
>>
>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yypvsxf19870706@gmail.com
>> > wrote:
>>
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one
>>> namenode,however the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>>
>>>
>>>
>>>
>>
>>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy,Harsh
It is strange that the NameNode rolling edits log rolls edits logs all
the times,even i do nothing on the NameNode vi client.
Any futher help will be appreciated.
regard.
2013/2/25 YouPeng Yang <yy...@gmail.com>
> Hi Azuryy
>
> Yes.that is what is.
> Thank you for your reply. I am making effort to make clear about
> federation and HA.
>
> Regards.
>
>
>
>
> 2013/2/25 Azuryy Yu <az...@gmail.com>
>
>> I think you mixed federation with HA. am I right?
>>
>> If another name node hasn't changes, then It doesn't do any edit log
>> rolling. federated NNs don't keep concurrency( I think you want say keep
>> sync-able?)
>>
>>
>> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yypvsxf19870706@gmail.com
>> > wrote:
>>
>>> Hi All
>>>
>>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one
>>> namenode,however the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>>
>>>
>>>
>>>
>>
>>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy
Yes.that is what is.
Thank you for your reply. I am making effort to make clear about
federation and HA.
Regards.
2013/2/25 Azuryy Yu <az...@gmail.com>
> I think you mixed federation with HA. am I right?
>
> If another name node hasn't changes, then It doesn't do any edit log
> rolling. federated NNs don't keep concurrency( I think you want say keep
> sync-able?)
>
>
> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one
>> namenode,however the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>>
>>
>>
>>
>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy
Yes.that is what is.
Thank you for your reply. I am making effort to make clear about
federation and HA.
Regards.
2013/2/25 Azuryy Yu <az...@gmail.com>
> I think you mixed federation with HA. am I right?
>
> If another name node hasn't changes, then It doesn't do any edit log
> rolling. federated NNs don't keep concurrency( I think you want say keep
> sync-able?)
>
>
> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one
>> namenode,however the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>>
>>
>>
>>
>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy
Yes.that is what is.
Thank you for your reply. I am making effort to make clear about
federation and HA.
Regards.
2013/2/25 Azuryy Yu <az...@gmail.com>
> I think you mixed federation with HA. am I right?
>
> If another name node hasn't changes, then It doesn't do any edit log
> rolling. federated NNs don't keep concurrency( I think you want say keep
> sync-able?)
>
>
> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one
>> namenode,however the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>>
>>
>>
>>
>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Azuryy
Yes.that is what is.
Thank you for your reply. I am making effort to make clear about
federation and HA.
Regards.
2013/2/25 Azuryy Yu <az...@gmail.com>
> I think you mixed federation with HA. am I right?
>
> If another name node hasn't changes, then It doesn't do any edit log
> rolling. federated NNs don't keep concurrency( I think you want say keep
> sync-able?)
>
>
> On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
>
>> Hi All
>>
>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one
>> namenode,however the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>>
>>
>>
>>
>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Azuryy Yu <az...@gmail.com>.
I think you mixed federation with HA. am I right?
If another name node hasn't changes, then It doesn't do any edit log
rolling. federated NNs don't keep concurrency( I think you want say keep
sync-able?)
On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one
> namenode,however the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Azuryy Yu <az...@gmail.com>.
I think you mixed federation with HA. am I right?
If another name node hasn't changes, then It doesn't do any edit log
rolling. federated NNs don't keep concurrency( I think you want say keep
sync-able?)
On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one
> namenode,however the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Federated namenodes are independent of one another (except that they
both get reports from all the/common DNs in the cluster). It is
natural to see one roll its edit logs based on its own rate of
metadata growth, as compared to the other. Their edits, image, etc.
everything is independent - they also do not know about/talk to each
other.
Given the above, I'd say (1) is true and (2) is invalid, and (3) can
be found in PDFs attached to
https://issues.apache.org/jira/browse/HDFS-1052.
On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one namenode,however
> the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Harsh J <ha...@cloudera.com>.
Hi,
Federated namenodes are independent of one another (except that they
both get reports from all the/common DNs in the cluster). It is
natural to see one roll its edit logs based on its own rate of
metadata growth, as compared to the other. Their edits, image, etc.
everything is independent - they also do not know about/talk to each
other.
Given the above, I'd say (1) is true and (2) is invalid, and (3) can
be found in PDFs attached to
https://issues.apache.org/jira/browse/HDFS-1052.
On Sun, Feb 24, 2013 at 8:39 PM, YouPeng Yang <yy...@gmail.com> wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one namenode,however
> the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
--
Harsh J
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Azuryy Yu <az...@gmail.com>.
I think you mixed federation with HA. am I right?
If another name node hasn't changes, then It doesn't do any edit log
rolling. federated NNs don't keep concurrency( I think you want say keep
sync-able?)
On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one
> namenode,however the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
>
Re: One NameNode keeps Rolling Edit Log on HDFS Federation
Posted by Azuryy Yu <az...@gmail.com>.
I think you mixed federation with HA. am I right?
If another name node hasn't changes, then It doesn't do any edit log
rolling. federated NNs don't keep concurrency( I think you want say keep
sync-able?)
On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang <yy...@gmail.com>wrote:
> Hi All
>
> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> I find that it is Rolling Edit Log continuously on one
> namenode,however the other one changes nothing.
>
> My question :
> 1. Is it the right situation.
> 2. I have thought that the two namenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
>