You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Azuryy Yu <az...@gmail.com> on 2013/04/08 07:23:29 UTC
A question of QJM with HDFS federation
Hi dears,
I deployed Hadoopv2 with HA enabled using QJM, so my question is:
1) if we also configured HDFS federation, such as:
NN1 is active, NN2 is standby
NN3 is active, NN4 is standby
they are configured as HDFS federation, then,
Can these four NNs using the same Journal nodes and ZKs?
if your answer is yes, does that enough just using different
dfs.journalnode.edits.dir?
such as:
NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
Thanks.
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Hi Harsh,
If they are two separate cluster, instead of federated, haing the same
cluster ID, but using different name service ID, so can they use the same
journal nodes and ZK nodes?
and if they are also separated cluster, with different clusterID, but using
the same name service ID, so can they use the same journal nodes and ZK
nodes?
Thanks.
On Mon, Apr 8, 2013 at 2:57 PM, Azuryy Yu <az...@gmail.com> wrote:
> Thank you very much, Harsh.
>
> not yet question now.
>
> --Send from my Sony mobile.
> On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> Hi Azurry,
>>
>> QJM:
>>
>> Yes, multiple nameservices can share a single QJM set. The QJM
>> configuration allows for a journal ID prefix path which you should
>> configure to be the nameservice ID. You do not need to change disk
>> paths/etc. at all.
>>
>> For example,
>>
>> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS1
>>
>> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS2
>>
>> Which will separate both of them logically and still make them use the
>> same QJM set of nodes.
>>
>> ZKFC:
>>
>> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
>> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
>> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
>> the nameservice ID as its parent znode name on the ZK instance, and
>> won't collide with another NSs' ZKFCs.
>>
>> Do post back if there are still some more doubts.
>>
>> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi dears,
>> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>> >
>> > 1) if we also configured HDFS federation, such as:
>> >
>> > NN1 is active, NN2 is standby
>> > NN3 is active, NN4 is standby
>> >
>> > they are configured as HDFS federation, then,
>> > Can these four NNs using the same Journal nodes and ZKs?
>> >
>> > if your answer is yes, does that enough just using different
>> > dfs.journalnode.edits.dir?
>> >
>> > such as:
>> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
>> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>> >
>> >
>> > Thanks.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Hi Harsh,
If they are two separate cluster, instead of federated, haing the same
cluster ID, but using different name service ID, so can they use the same
journal nodes and ZK nodes?
and if they are also separated cluster, with different clusterID, but using
the same name service ID, so can they use the same journal nodes and ZK
nodes?
Thanks.
On Mon, Apr 8, 2013 at 2:57 PM, Azuryy Yu <az...@gmail.com> wrote:
> Thank you very much, Harsh.
>
> not yet question now.
>
> --Send from my Sony mobile.
> On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> Hi Azurry,
>>
>> QJM:
>>
>> Yes, multiple nameservices can share a single QJM set. The QJM
>> configuration allows for a journal ID prefix path which you should
>> configure to be the nameservice ID. You do not need to change disk
>> paths/etc. at all.
>>
>> For example,
>>
>> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS1
>>
>> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS2
>>
>> Which will separate both of them logically and still make them use the
>> same QJM set of nodes.
>>
>> ZKFC:
>>
>> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
>> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
>> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
>> the nameservice ID as its parent znode name on the ZK instance, and
>> won't collide with another NSs' ZKFCs.
>>
>> Do post back if there are still some more doubts.
>>
>> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi dears,
>> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>> >
>> > 1) if we also configured HDFS federation, such as:
>> >
>> > NN1 is active, NN2 is standby
>> > NN3 is active, NN4 is standby
>> >
>> > they are configured as HDFS federation, then,
>> > Can these four NNs using the same Journal nodes and ZKs?
>> >
>> > if your answer is yes, does that enough just using different
>> > dfs.journalnode.edits.dir?
>> >
>> > such as:
>> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
>> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>> >
>> >
>> > Thanks.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Hi Harsh,
If they are two separate cluster, instead of federated, haing the same
cluster ID, but using different name service ID, so can they use the same
journal nodes and ZK nodes?
and if they are also separated cluster, with different clusterID, but using
the same name service ID, so can they use the same journal nodes and ZK
nodes?
Thanks.
On Mon, Apr 8, 2013 at 2:57 PM, Azuryy Yu <az...@gmail.com> wrote:
> Thank you very much, Harsh.
>
> not yet question now.
>
> --Send from my Sony mobile.
> On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> Hi Azurry,
>>
>> QJM:
>>
>> Yes, multiple nameservices can share a single QJM set. The QJM
>> configuration allows for a journal ID prefix path which you should
>> configure to be the nameservice ID. You do not need to change disk
>> paths/etc. at all.
>>
>> For example,
>>
>> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS1
>>
>> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS2
>>
>> Which will separate both of them logically and still make them use the
>> same QJM set of nodes.
>>
>> ZKFC:
>>
>> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
>> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
>> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
>> the nameservice ID as its parent znode name on the ZK instance, and
>> won't collide with another NSs' ZKFCs.
>>
>> Do post back if there are still some more doubts.
>>
>> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi dears,
>> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>> >
>> > 1) if we also configured HDFS federation, such as:
>> >
>> > NN1 is active, NN2 is standby
>> > NN3 is active, NN4 is standby
>> >
>> > they are configured as HDFS federation, then,
>> > Can these four NNs using the same Journal nodes and ZKs?
>> >
>> > if your answer is yes, does that enough just using different
>> > dfs.journalnode.edits.dir?
>> >
>> > such as:
>> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
>> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>> >
>> >
>> > Thanks.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Hi Harsh,
If they are two separate cluster, instead of federated, haing the same
cluster ID, but using different name service ID, so can they use the same
journal nodes and ZK nodes?
and if they are also separated cluster, with different clusterID, but using
the same name service ID, so can they use the same journal nodes and ZK
nodes?
Thanks.
On Mon, Apr 8, 2013 at 2:57 PM, Azuryy Yu <az...@gmail.com> wrote:
> Thank you very much, Harsh.
>
> not yet question now.
>
> --Send from my Sony mobile.
> On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> Hi Azurry,
>>
>> QJM:
>>
>> Yes, multiple nameservices can share a single QJM set. The QJM
>> configuration allows for a journal ID prefix path which you should
>> configure to be the nameservice ID. You do not need to change disk
>> paths/etc. at all.
>>
>> For example,
>>
>> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS1
>>
>> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
>> qjournal://node1.example.com:8485;node2.example.com:8485;
>> node3.example.com:8485/NS2
>>
>> Which will separate both of them logically and still make them use the
>> same QJM set of nodes.
>>
>> ZKFC:
>>
>> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
>> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
>> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
>> the nameservice ID as its parent znode name on the ZK instance, and
>> won't collide with another NSs' ZKFCs.
>>
>> Do post back if there are still some more doubts.
>>
>> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi dears,
>> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>> >
>> > 1) if we also configured HDFS federation, such as:
>> >
>> > NN1 is active, NN2 is standby
>> > NN3 is active, NN4 is standby
>> >
>> > they are configured as HDFS federation, then,
>> > Can these four NNs using the same Journal nodes and ZKs?
>> >
>> > if your answer is yes, does that enough just using different
>> > dfs.journalnode.edits.dir?
>> >
>> > such as:
>> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
>> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>> >
>> >
>> > Thanks.
>>
>>
>>
>> --
>> Harsh J
>>
>
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Thank you very much, Harsh.
not yet question now.
--Send from my Sony mobile.
On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
> Hi Azurry,
>
> QJM:
>
> Yes, multiple nameservices can share a single QJM set. The QJM
> configuration allows for a journal ID prefix path which you should
> configure to be the nameservice ID. You do not need to change disk
> paths/etc. at all.
>
> For example,
>
> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS1
>
> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS2
>
> Which will separate both of them logically and still make them use the
> same QJM set of nodes.
>
> ZKFC:
>
> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
> the nameservice ID as its parent znode name on the ZK instance, and
> won't collide with another NSs' ZKFCs.
>
> Do post back if there are still some more doubts.
>
> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi dears,
> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
> >
> > 1) if we also configured HDFS federation, such as:
> >
> > NN1 is active, NN2 is standby
> > NN3 is active, NN4 is standby
> >
> > they are configured as HDFS federation, then,
> > Can these four NNs using the same Journal nodes and ZKs?
> >
> > if your answer is yes, does that enough just using different
> > dfs.journalnode.edits.dir?
> >
> > such as:
> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
> >
> >
> > Thanks.
>
>
>
> --
> Harsh J
>
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Thank you very much, Harsh.
not yet question now.
--Send from my Sony mobile.
On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
> Hi Azurry,
>
> QJM:
>
> Yes, multiple nameservices can share a single QJM set. The QJM
> configuration allows for a journal ID prefix path which you should
> configure to be the nameservice ID. You do not need to change disk
> paths/etc. at all.
>
> For example,
>
> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS1
>
> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS2
>
> Which will separate both of them logically and still make them use the
> same QJM set of nodes.
>
> ZKFC:
>
> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
> the nameservice ID as its parent znode name on the ZK instance, and
> won't collide with another NSs' ZKFCs.
>
> Do post back if there are still some more doubts.
>
> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi dears,
> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
> >
> > 1) if we also configured HDFS federation, such as:
> >
> > NN1 is active, NN2 is standby
> > NN3 is active, NN4 is standby
> >
> > they are configured as HDFS federation, then,
> > Can these four NNs using the same Journal nodes and ZKs?
> >
> > if your answer is yes, does that enough just using different
> > dfs.journalnode.edits.dir?
> >
> > such as:
> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
> >
> >
> > Thanks.
>
>
>
> --
> Harsh J
>
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Thank you very much, Harsh.
not yet question now.
--Send from my Sony mobile.
On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
> Hi Azurry,
>
> QJM:
>
> Yes, multiple nameservices can share a single QJM set. The QJM
> configuration allows for a journal ID prefix path which you should
> configure to be the nameservice ID. You do not need to change disk
> paths/etc. at all.
>
> For example,
>
> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS1
>
> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS2
>
> Which will separate both of them logically and still make them use the
> same QJM set of nodes.
>
> ZKFC:
>
> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
> the nameservice ID as its parent znode name on the ZK instance, and
> won't collide with another NSs' ZKFCs.
>
> Do post back if there are still some more doubts.
>
> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi dears,
> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
> >
> > 1) if we also configured HDFS federation, such as:
> >
> > NN1 is active, NN2 is standby
> > NN3 is active, NN4 is standby
> >
> > they are configured as HDFS federation, then,
> > Can these four NNs using the same Journal nodes and ZKs?
> >
> > if your answer is yes, does that enough just using different
> > dfs.journalnode.edits.dir?
> >
> > such as:
> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
> >
> >
> > Thanks.
>
>
>
> --
> Harsh J
>
Re: A question of QJM with HDFS federation
Posted by Azuryy Yu <az...@gmail.com>.
Thank you very much, Harsh.
not yet question now.
--Send from my Sony mobile.
On Apr 8, 2013 2:51 PM, "Harsh J" <ha...@cloudera.com> wrote:
> Hi Azurry,
>
> QJM:
>
> Yes, multiple nameservices can share a single QJM set. The QJM
> configuration allows for a journal ID prefix path which you should
> configure to be the nameservice ID. You do not need to change disk
> paths/etc. at all.
>
> For example,
>
> NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS1
>
> NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
> qjournal://node1.example.com:8485;node2.example.com:8485;
> node3.example.com:8485/NS2
>
> Which will separate both of them logically and still make them use the
> same QJM set of nodes.
>
> ZKFC:
>
> Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
> NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
> the same ha.zookeeper.quorum value since the ZKFC automatically reuses
> the nameservice ID as its parent znode name on the ZK instance, and
> won't collide with another NSs' ZKFCs.
>
> Do post back if there are still some more doubts.
>
> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi dears,
> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
> >
> > 1) if we also configured HDFS federation, such as:
> >
> > NN1 is active, NN2 is standby
> > NN3 is active, NN4 is standby
> >
> > they are configured as HDFS federation, then,
> > Can these four NNs using the same Journal nodes and ZKs?
> >
> > if your answer is yes, does that enough just using different
> > dfs.journalnode.edits.dir?
> >
> > such as:
> > NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> > NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
> >
> >
> > Thanks.
>
>
>
> --
> Harsh J
>
Re: A question of QJM with HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Hi Azurry,
QJM:
Yes, multiple nameservices can share a single QJM set. The QJM
configuration allows for a journal ID prefix path which you should
configure to be the nameservice ID. You do not need to change disk
paths/etc. at all.
For example,
NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS1
NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS2
Which will separate both of them logically and still make them use the
same QJM set of nodes.
ZKFC:
Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
the same ha.zookeeper.quorum value since the ZKFC automatically reuses
the nameservice ID as its parent znode name on the ZK instance, and
won't collide with another NSs' ZKFCs.
Do post back if there are still some more doubts.
On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi dears,
> I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>
> 1) if we also configured HDFS federation, such as:
>
> NN1 is active, NN2 is standby
> NN3 is active, NN4 is standby
>
> they are configured as HDFS federation, then,
> Can these four NNs using the same Journal nodes and ZKs?
>
> if your answer is yes, does that enough just using different
> dfs.journalnode.edits.dir?
>
> such as:
> NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>
>
> Thanks.
--
Harsh J
Re: A question of QJM with HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Hi Azurry,
QJM:
Yes, multiple nameservices can share a single QJM set. The QJM
configuration allows for a journal ID prefix path which you should
configure to be the nameservice ID. You do not need to change disk
paths/etc. at all.
For example,
NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS1
NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS2
Which will separate both of them logically and still make them use the
same QJM set of nodes.
ZKFC:
Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
the same ha.zookeeper.quorum value since the ZKFC automatically reuses
the nameservice ID as its parent znode name on the ZK instance, and
won't collide with another NSs' ZKFCs.
Do post back if there are still some more doubts.
On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi dears,
> I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>
> 1) if we also configured HDFS federation, such as:
>
> NN1 is active, NN2 is standby
> NN3 is active, NN4 is standby
>
> they are configured as HDFS federation, then,
> Can these four NNs using the same Journal nodes and ZKs?
>
> if your answer is yes, does that enough just using different
> dfs.journalnode.edits.dir?
>
> such as:
> NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>
>
> Thanks.
--
Harsh J
Re: A question of QJM with HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Hi Azurry,
QJM:
Yes, multiple nameservices can share a single QJM set. The QJM
configuration allows for a journal ID prefix path which you should
configure to be the nameservice ID. You do not need to change disk
paths/etc. at all.
For example,
NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS1
NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS2
Which will separate both of them logically and still make them use the
same QJM set of nodes.
ZKFC:
Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
the same ha.zookeeper.quorum value since the ZKFC automatically reuses
the nameservice ID as its parent znode name on the ZK instance, and
won't collide with another NSs' ZKFCs.
Do post back if there are still some more doubts.
On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi dears,
> I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>
> 1) if we also configured HDFS federation, such as:
>
> NN1 is active, NN2 is standby
> NN3 is active, NN4 is standby
>
> they are configured as HDFS federation, then,
> Can these four NNs using the same Journal nodes and ZKs?
>
> if your answer is yes, does that enough just using different
> dfs.journalnode.edits.dir?
>
> such as:
> NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>
>
> Thanks.
--
Harsh J
Re: A question of QJM with HDFS federation
Posted by Harsh J <ha...@cloudera.com>.
Hi Azurry,
QJM:
Yes, multiple nameservices can share a single QJM set. The QJM
configuration allows for a journal ID prefix path which you should
configure to be the nameservice ID. You do not need to change disk
paths/etc. at all.
For example,
NS1 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS1
NS2 NNs can have the dfs.namenode.shared.edits.dir configured as:
qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485/NS2
Which will separate both of them logically and still make them use the
same QJM set of nodes.
ZKFC:
Each NN needs its own HDFS ZKFC daemon, but all ZKFCs across multiple
NSes can share a single ZK cluster. All ZKFC's core-site.xml can have
the same ha.zookeeper.quorum value since the ZKFC automatically reuses
the nameservice ID as its parent znode name on the ZK instance, and
won't collide with another NSs' ZKFCs.
Do post back if there are still some more doubts.
On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu <az...@gmail.com> wrote:
> Hi dears,
> I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>
> 1) if we also configured HDFS federation, such as:
>
> NN1 is active, NN2 is standby
> NN3 is active, NN4 is standby
>
> they are configured as HDFS federation, then,
> Can these four NNs using the same Journal nodes and ZKs?
>
> if your answer is yes, does that enough just using different
> dfs.journalnode.edits.dir?
>
> such as:
> NN1 and NN2 configured dfs.journalnode.edits.dir as /data1,
> NN3 and NN4 configured dfs.journalnode.edits.dir as /data2
>
>
> Thanks.
--
Harsh J