You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by lei liu <li...@gmail.com> on 2013/07/17 10:03:45 UTC
QJM for federation
I have two namespaces, example below:
<property>
<name>dfs.nameservices</name>
<value>ns1,ns2</value>
</property>
Can I config the "dfs.namenode.shared.edits.dir" to below content?
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
</property>
Thanks,
LiuLei
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
Yeah ideally you needn't configure a local edits directory,
unless you want extra safety. There was an issue where it still gets
configured and logs local edits though but am not sure if this was
fixed. If you are not able to turn off local edit logging and if it
still always happens, do file a HDFS jira.
On Wed, Jul 17, 2013 at 2:19 PM, lei liu <li...@gmail.com> wrote:
>
> I have another question for QJM.
>
> If I use QJM for HA, do I need to save edit log on the local filesystem?
>
> I think the QJM is high availability for edit log, so I don't need to
> config the "dfs.namenode.edits.dir".
>
>
> Thanks,
>
> LiuLei
>
>
> 2013/7/17 lei liu <li...@gmail.com>
>>
>> Thanks Harsh.
>>
>>
>>
>>
>> 2013/7/17 Harsh J <ha...@cloudera.com>
>>>
>>> This has been asked previously. Use suffixes to solve your issue. See
>>> http://search-hadoop.com/m/Fingkg6Dk91
>>>
>>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>>> > I have two namespaces, example below:
>>> >
>>> > <property>
>>> > <name>dfs.nameservices</name>
>>> > <value>ns1,ns2</value>
>>> > </property>
>>> >
>>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>>> > <property>
>>> > <name>dfs.namenode.shared.edits.dir</name>
>>> >
>>> >
>>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>>> >
>>> > </property>
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > LiuLei
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
--
Harsh J
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
Yeah ideally you needn't configure a local edits directory,
unless you want extra safety. There was an issue where it still gets
configured and logs local edits though but am not sure if this was
fixed. If you are not able to turn off local edit logging and if it
still always happens, do file a HDFS jira.
On Wed, Jul 17, 2013 at 2:19 PM, lei liu <li...@gmail.com> wrote:
>
> I have another question for QJM.
>
> If I use QJM for HA, do I need to save edit log on the local filesystem?
>
> I think the QJM is high availability for edit log, so I don't need to
> config the "dfs.namenode.edits.dir".
>
>
> Thanks,
>
> LiuLei
>
>
> 2013/7/17 lei liu <li...@gmail.com>
>>
>> Thanks Harsh.
>>
>>
>>
>>
>> 2013/7/17 Harsh J <ha...@cloudera.com>
>>>
>>> This has been asked previously. Use suffixes to solve your issue. See
>>> http://search-hadoop.com/m/Fingkg6Dk91
>>>
>>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>>> > I have two namespaces, example below:
>>> >
>>> > <property>
>>> > <name>dfs.nameservices</name>
>>> > <value>ns1,ns2</value>
>>> > </property>
>>> >
>>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>>> > <property>
>>> > <name>dfs.namenode.shared.edits.dir</name>
>>> >
>>> >
>>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>>> >
>>> > </property>
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > LiuLei
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
--
Harsh J
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
Yeah ideally you needn't configure a local edits directory,
unless you want extra safety. There was an issue where it still gets
configured and logs local edits though but am not sure if this was
fixed. If you are not able to turn off local edit logging and if it
still always happens, do file a HDFS jira.
On Wed, Jul 17, 2013 at 2:19 PM, lei liu <li...@gmail.com> wrote:
>
> I have another question for QJM.
>
> If I use QJM for HA, do I need to save edit log on the local filesystem?
>
> I think the QJM is high availability for edit log, so I don't need to
> config the "dfs.namenode.edits.dir".
>
>
> Thanks,
>
> LiuLei
>
>
> 2013/7/17 lei liu <li...@gmail.com>
>>
>> Thanks Harsh.
>>
>>
>>
>>
>> 2013/7/17 Harsh J <ha...@cloudera.com>
>>>
>>> This has been asked previously. Use suffixes to solve your issue. See
>>> http://search-hadoop.com/m/Fingkg6Dk91
>>>
>>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>>> > I have two namespaces, example below:
>>> >
>>> > <property>
>>> > <name>dfs.nameservices</name>
>>> > <value>ns1,ns2</value>
>>> > </property>
>>> >
>>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>>> > <property>
>>> > <name>dfs.namenode.shared.edits.dir</name>
>>> >
>>> >
>>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>>> >
>>> > </property>
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > LiuLei
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
--
Harsh J
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
Yeah ideally you needn't configure a local edits directory,
unless you want extra safety. There was an issue where it still gets
configured and logs local edits though but am not sure if this was
fixed. If you are not able to turn off local edit logging and if it
still always happens, do file a HDFS jira.
On Wed, Jul 17, 2013 at 2:19 PM, lei liu <li...@gmail.com> wrote:
>
> I have another question for QJM.
>
> If I use QJM for HA, do I need to save edit log on the local filesystem?
>
> I think the QJM is high availability for edit log, so I don't need to
> config the "dfs.namenode.edits.dir".
>
>
> Thanks,
>
> LiuLei
>
>
> 2013/7/17 lei liu <li...@gmail.com>
>>
>> Thanks Harsh.
>>
>>
>>
>>
>> 2013/7/17 Harsh J <ha...@cloudera.com>
>>>
>>> This has been asked previously. Use suffixes to solve your issue. See
>>> http://search-hadoop.com/m/Fingkg6Dk91
>>>
>>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>>> > I have two namespaces, example below:
>>> >
>>> > <property>
>>> > <name>dfs.nameservices</name>
>>> > <value>ns1,ns2</value>
>>> > </property>
>>> >
>>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>>> > <property>
>>> > <name>dfs.namenode.shared.edits.dir</name>
>>> >
>>> >
>>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>>> >
>>> > </property>
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > LiuLei
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>>
>
--
Harsh J
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
I have another question for QJM.
If I use QJM for HA, do I need to save edit log on the local filesystem?
I think the QJM is high availability for edit log, so I don't need to
config the "dfs.namenode.edits.dir".
Thanks,
LiuLei
2013/7/17 lei liu <li...@gmail.com>
> Thanks Harsh.
>
>
>
>
> 2013/7/17 Harsh J <ha...@cloudera.com>
>
>> This has been asked previously. Use suffixes to solve your issue. See
>> http://search-hadoop.com/m/Fingkg6Dk91
>>
>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>> > I have two namespaces, example below:
>> >
>> > <property>
>> > <name>dfs.nameservices</name>
>> > <value>ns1,ns2</value>
>> > </property>
>> >
>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>> > <property>
>> > <name>dfs.namenode.shared.edits.dir</name>
>> >
>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
>> 10.232.98.63:20022/nn1,nn2</value>
>> >
>> > </property>
>> >
>> >
>> > Thanks,
>> >
>> > LiuLei
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
I have another question for QJM.
If I use QJM for HA, do I need to save edit log on the local filesystem?
I think the QJM is high availability for edit log, so I don't need to
config the "dfs.namenode.edits.dir".
Thanks,
LiuLei
2013/7/17 lei liu <li...@gmail.com>
> Thanks Harsh.
>
>
>
>
> 2013/7/17 Harsh J <ha...@cloudera.com>
>
>> This has been asked previously. Use suffixes to solve your issue. See
>> http://search-hadoop.com/m/Fingkg6Dk91
>>
>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>> > I have two namespaces, example below:
>> >
>> > <property>
>> > <name>dfs.nameservices</name>
>> > <value>ns1,ns2</value>
>> > </property>
>> >
>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>> > <property>
>> > <name>dfs.namenode.shared.edits.dir</name>
>> >
>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
>> 10.232.98.63:20022/nn1,nn2</value>
>> >
>> > </property>
>> >
>> >
>> > Thanks,
>> >
>> > LiuLei
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
I have another question for QJM.
If I use QJM for HA, do I need to save edit log on the local filesystem?
I think the QJM is high availability for edit log, so I don't need to
config the "dfs.namenode.edits.dir".
Thanks,
LiuLei
2013/7/17 lei liu <li...@gmail.com>
> Thanks Harsh.
>
>
>
>
> 2013/7/17 Harsh J <ha...@cloudera.com>
>
>> This has been asked previously. Use suffixes to solve your issue. See
>> http://search-hadoop.com/m/Fingkg6Dk91
>>
>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>> > I have two namespaces, example below:
>> >
>> > <property>
>> > <name>dfs.nameservices</name>
>> > <value>ns1,ns2</value>
>> > </property>
>> >
>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>> > <property>
>> > <name>dfs.namenode.shared.edits.dir</name>
>> >
>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
>> 10.232.98.63:20022/nn1,nn2</value>
>> >
>> > </property>
>> >
>> >
>> > Thanks,
>> >
>> > LiuLei
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
I have another question for QJM.
If I use QJM for HA, do I need to save edit log on the local filesystem?
I think the QJM is high availability for edit log, so I don't need to
config the "dfs.namenode.edits.dir".
Thanks,
LiuLei
2013/7/17 lei liu <li...@gmail.com>
> Thanks Harsh.
>
>
>
>
> 2013/7/17 Harsh J <ha...@cloudera.com>
>
>> This has been asked previously. Use suffixes to solve your issue. See
>> http://search-hadoop.com/m/Fingkg6Dk91
>>
>> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
>> > I have two namespaces, example below:
>> >
>> > <property>
>> > <name>dfs.nameservices</name>
>> > <value>ns1,ns2</value>
>> > </property>
>> >
>> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
>> > <property>
>> > <name>dfs.namenode.shared.edits.dir</name>
>> >
>> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
>> 10.232.98.63:20022/nn1,nn2</value>
>> >
>> > </property>
>> >
>> >
>> > Thanks,
>> >
>> > LiuLei
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
Thanks Harsh.
2013/7/17 Harsh J <ha...@cloudera.com>
> This has been asked previously. Use suffixes to solve your issue. See
> http://search-hadoop.com/m/Fingkg6Dk91
>
> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> > I have two namespaces, example below:
> >
> > <property>
> > <name>dfs.nameservices</name>
> > <value>ns1,ns2</value>
> > </property>
> >
> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
> > <property>
> > <name>dfs.namenode.shared.edits.dir</name>
> >
> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
> 10.232.98.63:20022/nn1,nn2</value>
> >
> > </property>
> >
> >
> > Thanks,
> >
> > LiuLei
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
Thanks Harsh.
2013/7/17 Harsh J <ha...@cloudera.com>
> This has been asked previously. Use suffixes to solve your issue. See
> http://search-hadoop.com/m/Fingkg6Dk91
>
> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> > I have two namespaces, example below:
> >
> > <property>
> > <name>dfs.nameservices</name>
> > <value>ns1,ns2</value>
> > </property>
> >
> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
> > <property>
> > <name>dfs.namenode.shared.edits.dir</name>
> >
> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
> 10.232.98.63:20022/nn1,nn2</value>
> >
> > </property>
> >
> >
> > Thanks,
> >
> > LiuLei
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
Thanks Harsh.
2013/7/17 Harsh J <ha...@cloudera.com>
> This has been asked previously. Use suffixes to solve your issue. See
> http://search-hadoop.com/m/Fingkg6Dk91
>
> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> > I have two namespaces, example below:
> >
> > <property>
> > <name>dfs.nameservices</name>
> > <value>ns1,ns2</value>
> > </property>
> >
> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
> > <property>
> > <name>dfs.namenode.shared.edits.dir</name>
> >
> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
> 10.232.98.63:20022/nn1,nn2</value>
> >
> > </property>
> >
> >
> > Thanks,
> >
> > LiuLei
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: QJM for federation
Posted by lei liu <li...@gmail.com>.
Thanks Harsh.
2013/7/17 Harsh J <ha...@cloudera.com>
> This has been asked previously. Use suffixes to solve your issue. See
> http://search-hadoop.com/m/Fingkg6Dk91
>
> On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> > I have two namespaces, example below:
> >
> > <property>
> > <name>dfs.nameservices</name>
> > <value>ns1,ns2</value>
> > </property>
> >
> > Can I config the "dfs.namenode.shared.edits.dir" to below content?
> > <property>
> > <name>dfs.namenode.shared.edits.dir</name>
> >
> > <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;
> 10.232.98.63:20022/nn1,nn2</value>
> >
> > </property>
> >
> >
> > Thanks,
> >
> > LiuLei
> >
> >
> >
> >
> >
>
>
>
> --
> Harsh J
>
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
This has been asked previously. Use suffixes to solve your issue. See
http://search-hadoop.com/m/Fingkg6Dk91
On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> I have two namespaces, example below:
>
> <property>
> <name>dfs.nameservices</name>
> <value>ns1,ns2</value>
> </property>
>
> Can I config the "dfs.namenode.shared.edits.dir" to below content?
> <property>
> <name>dfs.namenode.shared.edits.dir</name>
>
> <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>
> </property>
>
>
> Thanks,
>
> LiuLei
>
>
>
>
>
--
Harsh J
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
This has been asked previously. Use suffixes to solve your issue. See
http://search-hadoop.com/m/Fingkg6Dk91
On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> I have two namespaces, example below:
>
> <property>
> <name>dfs.nameservices</name>
> <value>ns1,ns2</value>
> </property>
>
> Can I config the "dfs.namenode.shared.edits.dir" to below content?
> <property>
> <name>dfs.namenode.shared.edits.dir</name>
>
> <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>
> </property>
>
>
> Thanks,
>
> LiuLei
>
>
>
>
>
--
Harsh J
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
This has been asked previously. Use suffixes to solve your issue. See
http://search-hadoop.com/m/Fingkg6Dk91
On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> I have two namespaces, example below:
>
> <property>
> <name>dfs.nameservices</name>
> <value>ns1,ns2</value>
> </property>
>
> Can I config the "dfs.namenode.shared.edits.dir" to below content?
> <property>
> <name>dfs.namenode.shared.edits.dir</name>
>
> <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>
> </property>
>
>
> Thanks,
>
> LiuLei
>
>
>
>
>
--
Harsh J
Re: QJM for federation
Posted by Harsh J <ha...@cloudera.com>.
This has been asked previously. Use suffixes to solve your issue. See
http://search-hadoop.com/m/Fingkg6Dk91
On Wed, Jul 17, 2013 at 1:33 PM, lei liu <li...@gmail.com> wrote:
> I have two namespaces, example below:
>
> <property>
> <name>dfs.nameservices</name>
> <value>ns1,ns2</value>
> </property>
>
> Can I config the "dfs.namenode.shared.edits.dir" to below content?
> <property>
> <name>dfs.namenode.shared.edits.dir</name>
>
> <value>qjournal://10.232.98.61:20022;10.232.98.62:20022;10.232.98.63:20022/nn1,nn2</value>
>
> </property>
>
>
> Thanks,
>
> LiuLei
>
>
>
>
>
--
Harsh J