You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Henning Kropp <hk...@microlution.de> on 2016/04/01 19:38:50 UTC
Support for Multiple Nameservices
Hi,
we noticed that throughout Ambari hdfs_site['dfs.nameservices'] is
treated as a string denoting just one nameservice. It is possible to
configure multiple namservices for example to support seamless distcp
between two HA clusters. In general this makes it possible to use
multiple nameservices wiht hdfs.
This makes dfs.nameservices a comma separated string holding mulitple
namservices. When doing this with the current Ambari release this leads
to multiple problems. One is that Ambari marks the restart of HDFS as
failed, even though the restart was succesful. The reason for that is
that hdfs_resources.py is not working this a comma separated list of
nameservices AMBARI-15506.
We created an umbrella JIRA to track the other issues AMBARI-15507.
Problems with Blueprint install, because the DN's were registered with
the other cluster AMBARI-15509. Web alerting for NNs does not work
AMBARI-15508.
There might be other places where dfs.namservices is treated just a
string? How can web alerting be refactored to work with multiple
nameservices?
Also I would appreciate any feedback about the function to resolve the
current nameservice for the current cluster.
For AMBARI-15506 I defined the following nameservice resolution:
1. split names by ','
2. for all services check if string is also contained in
dfs.namenode.shared.edits.dir . Typically this are
jounalnode1,journalnode2,journalnode3:port/servicename. Here it would
probably be better to verify the name with fs.defaultFS, but this is
part of core-site not hdfs-site, which would add a separate dependency.
For namenode_ha_utils.py for me this seemed like an issue, because
refactoring all the python scripts to also include the core-site seemed
much more involved.
3. A default fallback the first string in the list is used as the
nameservice.
Thanks and regards,
Henning
Re: Support for Multiple Nameservices
Posted by Henning Kropp <hk...@microlution.de>.
Hi Sumit,
sounds great!
The use of dfs.namenode.shared.edits.dir is a "hack" it contains the
current nameservice as part of the path for the journalnodes. From the
docs:
https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
/*dfs.namenode.shared.edits.dir*/ - the URI which identifies the group
of JNs where the NameNodes will write/read edits /
This is where one configures the addresses of the JournalNodes which
provide the shared edits storage, written to by the Active nameNode and
read by the Standby NameNode to stay up-to-date with all the file system
changes the Active NameNode makes. Though you must specify several
JournalNode addresses, //*you should only configure one of these
URIs.*//The URI should be of the form:
//qjournal://*host1:port1*;*host2:port2*;*host3:port3*/*journalId*//.
The Journal ID is a unique identifier for this nameservice, which allows
a single set of JournalNodes to provide storage for multiple federated
namesystems. Though not a requirement, it’s a good idea to reuse the
nameservice ID for the journal identifier.//
F//or example, if the JournalNodes for this cluster were running on the
machines “node1.example.com”, “node2.example.com”, and
“node3.example.com” //_*and the nameservice ID were “mycluster”*_//, you
would use the following as the value for this setting (the default port
for the JournalNode is 8485)://
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://node1.example.com:8485;node2.example.com:8485;node3.example.com:8485///*mycluster*//</value>
</property>
/
In my opinion the best way to get the current nameservice would be
_*fs.defaultFS*_.**In an HA cluster this contains the nameservice with
protocol.
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
Both are only suitable if you are able to do a string /contains/,
/is-in/, or /regex/ to retrieve the service id.
The problem I was facing with "namenode_ha_utils.py" is that only
hdfs-site got injected. So I needed a property in hdfs-site which
contains the current service id. I picked dfs.namenode.shared.edits.dir
in this case.
Thinking about this now any comparision should also not be
case-sensitive, or is the service id case-sensitive?
Here is my implementation of "get_nameservice(hdfs_site)"
name_service = hdfs_site['dfs.nameservices']
if name_service:
for ns in name_service.split(","):
if ns in hdfs_site['dfs.namenode.shared.edits.dir']: # better would be core_site['fs.defaultFS'] but it's not available
return ns
return name_service.split(",")[0] # default to return the first nameservice
return name_service
Regards,
Henning
Am 01/04/16 um 22:13 schrieb Sumit Mohanty:
> A fall back logic will be good, perhaps required. Any documentation around dfs.namenode.shared.edits.dir that I can refer to in terms of its content and reliability of it?
>
> My original thoughts were:
>
> * For fresh NN/HA enablements and fresh blueprint deployments, it will work fine as Amabri can add and rely on the existence of "dfs.internal.nameservice"
> * For existing clusters, during Ambari upgrade it can auto copy value from dfs.nameservice to dfs.internal.nameservice if the value not a comma separated list
> ** If the value is a comma separated list then we already a problem - so those deployments need to fix it manually after upgrade as Ambari can now understand dfs.internal.nameservice
> * We may still need a fall back logic
> ** For fallback, I was thinking of falling back to the current logic (mostly, as I did not know about dfs.namenode.shared.edits.dir)
> ** Fallback should not be needed if we can fix the configs automatically during Ambari Upgrade
>
> -Sumit
> ________________________________________
> From: Henning Kropp <hk...@microlution.de>
> Sent: Friday, April 01, 2016 12:49 PM
> To: dev@ambari.apache.org
> Subject: Re: Support for Multiple Nameservices
>
> Hi Sumit,
>
> very interesting. I will certainly explore the possibility of using
> dfs.inernal.nameservices (leaving federation for another day :)
>
> You think a fallback to dfs.nameservices (the current behaivour), if
> internal nameservices is not set, is needed?
>
> Regards,
> Henning
>
>
> Am 01/04/16 um 20:30 schrieb Sumit Mohanty:
>> Hi Henning.
>>
>> Its a very good problem that you brought up.
>>
>> https://community.hortonworks.com/questions/8989/how-to-use-name-service-id-between-to-clusters.html talks about "dfs.internal.nameservices" and if Ambari populates this and uses it for all its internal scripts/alerts then that may be a solution.
>>
>> This, as you notice also says "nameservices" (plural) which points to HDFS federation. That is probably a problem for a different day.
>> In any case, in absence of HDFS federation, we can have Ambari refer to "dfs.internal.nameservices" for its own use. The requirement will be to have NN HA wizard and Blueprint deployment populate this property.
>>
>> AMBARI-15615 modified the NN HA wizard to populate the property. But as far as I can see no other changes have happened.
>>
>> I was not aware of hdfs_site['dfs.namenode.shared.edits.dir'] as a possible solution but talking to few folks from HDFS it seems that populating/using "dfs.internal.nameservices" will be more desirable.
>>
>> Do you want to explore the possibility of using "dfs.internal.nameservices"?
>>
>> thanks
>> Sumit
>> ________________________________________
>> From: Henning Kropp <hk...@microlution.de>
>> Sent: Friday, April 01, 2016 10:38 AM
>> To: dev@ambari.apache.org
>> Subject: Support for Multiple Nameservices
>>
>> Hi,
>>
>> we noticed that throughout Ambari hdfs_site['dfs.nameservices'] is
>> treated as a string denoting just one nameservice. It is possible to
>> configure multiple namservices for example to support seamless distcp
>> between two HA clusters. In general this makes it possible to use
>> multiple nameservices wiht hdfs.
>>
>> This makes dfs.nameservices a comma separated string holding mulitple
>> namservices. When doing this with the current Ambari release this leads
>> to multiple problems. One is that Ambari marks the restart of HDFS as
>> failed, even though the restart was succesful. The reason for that is
>> that hdfs_resources.py is not working this a comma separated list of
>> nameservices AMBARI-15506.
>>
>> We created an umbrella JIRA to track the other issues AMBARI-15507.
>> Problems with Blueprint install, because the DN's were registered with
>> the other cluster AMBARI-15509. Web alerting for NNs does not work
>> AMBARI-15508.
>>
>> There might be other places where dfs.namservices is treated just a
>> string? How can web alerting be refactored to work with multiple
>> nameservices?
>>
>> Also I would appreciate any feedback about the function to resolve the
>> current nameservice for the current cluster.
>>
>> For AMBARI-15506 I defined the following nameservice resolution:
>> 1. split names by ','
>> 2. for all services check if string is also contained in
>> dfs.namenode.shared.edits.dir . Typically this are
>> jounalnode1,journalnode2,journalnode3:port/servicename. Here it would
>> probably be better to verify the name with fs.defaultFS, but this is
>> part of core-site not hdfs-site, which would add a separate dependency.
>> For namenode_ha_utils.py for me this seemed like an issue, because
>> refactoring all the python scripts to also include the core-site seemed
>> much more involved.
>> 3. A default fallback the first string in the list is used as the
>> nameservice.
>>
>> Thanks and regards,
>> Henning
>
Re: Support for Multiple Nameservices
Posted by Sumit Mohanty <sm...@hortonworks.com>.
A fall back logic will be good, perhaps required. Any documentation around dfs.namenode.shared.edits.dir that I can refer to in terms of its content and reliability of it?
My original thoughts were:
* For fresh NN/HA enablements and fresh blueprint deployments, it will work fine as Amabri can add and rely on the existence of "dfs.internal.nameservice"
* For existing clusters, during Ambari upgrade it can auto copy value from dfs.nameservice to dfs.internal.nameservice if the value not a comma separated list
** If the value is a comma separated list then we already a problem - so those deployments need to fix it manually after upgrade as Ambari can now understand dfs.internal.nameservice
* We may still need a fall back logic
** For fallback, I was thinking of falling back to the current logic (mostly, as I did not know about dfs.namenode.shared.edits.dir)
** Fallback should not be needed if we can fix the configs automatically during Ambari Upgrade
-Sumit
________________________________________
From: Henning Kropp <hk...@microlution.de>
Sent: Friday, April 01, 2016 12:49 PM
To: dev@ambari.apache.org
Subject: Re: Support for Multiple Nameservices
Hi Sumit,
very interesting. I will certainly explore the possibility of using
dfs.inernal.nameservices (leaving federation for another day :)
You think a fallback to dfs.nameservices (the current behaivour), if
internal nameservices is not set, is needed?
Regards,
Henning
Am 01/04/16 um 20:30 schrieb Sumit Mohanty:
> Hi Henning.
>
> Its a very good problem that you brought up.
>
> https://community.hortonworks.com/questions/8989/how-to-use-name-service-id-between-to-clusters.html talks about "dfs.internal.nameservices" and if Ambari populates this and uses it for all its internal scripts/alerts then that may be a solution.
>
> This, as you notice also says "nameservices" (plural) which points to HDFS federation. That is probably a problem for a different day.
> In any case, in absence of HDFS federation, we can have Ambari refer to "dfs.internal.nameservices" for its own use. The requirement will be to have NN HA wizard and Blueprint deployment populate this property.
>
> AMBARI-15615 modified the NN HA wizard to populate the property. But as far as I can see no other changes have happened.
>
> I was not aware of hdfs_site['dfs.namenode.shared.edits.dir'] as a possible solution but talking to few folks from HDFS it seems that populating/using "dfs.internal.nameservices" will be more desirable.
>
> Do you want to explore the possibility of using "dfs.internal.nameservices"?
>
> thanks
> Sumit
> ________________________________________
> From: Henning Kropp <hk...@microlution.de>
> Sent: Friday, April 01, 2016 10:38 AM
> To: dev@ambari.apache.org
> Subject: Support for Multiple Nameservices
>
> Hi,
>
> we noticed that throughout Ambari hdfs_site['dfs.nameservices'] is
> treated as a string denoting just one nameservice. It is possible to
> configure multiple namservices for example to support seamless distcp
> between two HA clusters. In general this makes it possible to use
> multiple nameservices wiht hdfs.
>
> This makes dfs.nameservices a comma separated string holding mulitple
> namservices. When doing this with the current Ambari release this leads
> to multiple problems. One is that Ambari marks the restart of HDFS as
> failed, even though the restart was succesful. The reason for that is
> that hdfs_resources.py is not working this a comma separated list of
> nameservices AMBARI-15506.
>
> We created an umbrella JIRA to track the other issues AMBARI-15507.
> Problems with Blueprint install, because the DN's were registered with
> the other cluster AMBARI-15509. Web alerting for NNs does not work
> AMBARI-15508.
>
> There might be other places where dfs.namservices is treated just a
> string? How can web alerting be refactored to work with multiple
> nameservices?
>
> Also I would appreciate any feedback about the function to resolve the
> current nameservice for the current cluster.
>
> For AMBARI-15506 I defined the following nameservice resolution:
> 1. split names by ','
> 2. for all services check if string is also contained in
> dfs.namenode.shared.edits.dir . Typically this are
> jounalnode1,journalnode2,journalnode3:port/servicename. Here it would
> probably be better to verify the name with fs.defaultFS, but this is
> part of core-site not hdfs-site, which would add a separate dependency.
> For namenode_ha_utils.py for me this seemed like an issue, because
> refactoring all the python scripts to also include the core-site seemed
> much more involved.
> 3. A default fallback the first string in the list is used as the
> nameservice.
>
> Thanks and regards,
> Henning
Re: Support for Multiple Nameservices
Posted by Henning Kropp <hk...@microlution.de>.
Hi Sumit,
very interesting. I will certainly explore the possibility of using
dfs.inernal.nameservices (leaving federation for another day :)
You think a fallback to dfs.nameservices (the current behaivour), if
internal nameservices is not set, is needed?
Regards,
Henning
Am 01/04/16 um 20:30 schrieb Sumit Mohanty:
> Hi Henning.
>
> Its a very good problem that you brought up.
>
> https://community.hortonworks.com/questions/8989/how-to-use-name-service-id-between-to-clusters.html talks about "dfs.internal.nameservices" and if Ambari populates this and uses it for all its internal scripts/alerts then that may be a solution.
>
> This, as you notice also says "nameservices" (plural) which points to HDFS federation. That is probably a problem for a different day.
> In any case, in absence of HDFS federation, we can have Ambari refer to "dfs.internal.nameservices" for its own use. The requirement will be to have NN HA wizard and Blueprint deployment populate this property.
>
> AMBARI-15615 modified the NN HA wizard to populate the property. But as far as I can see no other changes have happened.
>
> I was not aware of hdfs_site['dfs.namenode.shared.edits.dir'] as a possible solution but talking to few folks from HDFS it seems that populating/using "dfs.internal.nameservices" will be more desirable.
>
> Do you want to explore the possibility of using "dfs.internal.nameservices"?
>
> thanks
> Sumit
> ________________________________________
> From: Henning Kropp <hk...@microlution.de>
> Sent: Friday, April 01, 2016 10:38 AM
> To: dev@ambari.apache.org
> Subject: Support for Multiple Nameservices
>
> Hi,
>
> we noticed that throughout Ambari hdfs_site['dfs.nameservices'] is
> treated as a string denoting just one nameservice. It is possible to
> configure multiple namservices for example to support seamless distcp
> between two HA clusters. In general this makes it possible to use
> multiple nameservices wiht hdfs.
>
> This makes dfs.nameservices a comma separated string holding mulitple
> namservices. When doing this with the current Ambari release this leads
> to multiple problems. One is that Ambari marks the restart of HDFS as
> failed, even though the restart was succesful. The reason for that is
> that hdfs_resources.py is not working this a comma separated list of
> nameservices AMBARI-15506.
>
> We created an umbrella JIRA to track the other issues AMBARI-15507.
> Problems with Blueprint install, because the DN's were registered with
> the other cluster AMBARI-15509. Web alerting for NNs does not work
> AMBARI-15508.
>
> There might be other places where dfs.namservices is treated just a
> string? How can web alerting be refactored to work with multiple
> nameservices?
>
> Also I would appreciate any feedback about the function to resolve the
> current nameservice for the current cluster.
>
> For AMBARI-15506 I defined the following nameservice resolution:
> 1. split names by ','
> 2. for all services check if string is also contained in
> dfs.namenode.shared.edits.dir . Typically this are
> jounalnode1,journalnode2,journalnode3:port/servicename. Here it would
> probably be better to verify the name with fs.defaultFS, but this is
> part of core-site not hdfs-site, which would add a separate dependency.
> For namenode_ha_utils.py for me this seemed like an issue, because
> refactoring all the python scripts to also include the core-site seemed
> much more involved.
> 3. A default fallback the first string in the list is used as the
> nameservice.
>
> Thanks and regards,
> Henning
Re: Support for Multiple Nameservices
Posted by Sumit Mohanty <sm...@hortonworks.com>.
Hi Henning.
Its a very good problem that you brought up.
https://community.hortonworks.com/questions/8989/how-to-use-name-service-id-between-to-clusters.html talks about "dfs.internal.nameservices" and if Ambari populates this and uses it for all its internal scripts/alerts then that may be a solution.
This, as you notice also says "nameservices" (plural) which points to HDFS federation. That is probably a problem for a different day.
In any case, in absence of HDFS federation, we can have Ambari refer to "dfs.internal.nameservices" for its own use. The requirement will be to have NN HA wizard and Blueprint deployment populate this property.
AMBARI-15615 modified the NN HA wizard to populate the property. But as far as I can see no other changes have happened.
I was not aware of hdfs_site['dfs.namenode.shared.edits.dir'] as a possible solution but talking to few folks from HDFS it seems that populating/using "dfs.internal.nameservices" will be more desirable.
Do you want to explore the possibility of using "dfs.internal.nameservices"?
thanks
Sumit
________________________________________
From: Henning Kropp <hk...@microlution.de>
Sent: Friday, April 01, 2016 10:38 AM
To: dev@ambari.apache.org
Subject: Support for Multiple Nameservices
Hi,
we noticed that throughout Ambari hdfs_site['dfs.nameservices'] is
treated as a string denoting just one nameservice. It is possible to
configure multiple namservices for example to support seamless distcp
between two HA clusters. In general this makes it possible to use
multiple nameservices wiht hdfs.
This makes dfs.nameservices a comma separated string holding mulitple
namservices. When doing this with the current Ambari release this leads
to multiple problems. One is that Ambari marks the restart of HDFS as
failed, even though the restart was succesful. The reason for that is
that hdfs_resources.py is not working this a comma separated list of
nameservices AMBARI-15506.
We created an umbrella JIRA to track the other issues AMBARI-15507.
Problems with Blueprint install, because the DN's were registered with
the other cluster AMBARI-15509. Web alerting for NNs does not work
AMBARI-15508.
There might be other places where dfs.namservices is treated just a
string? How can web alerting be refactored to work with multiple
nameservices?
Also I would appreciate any feedback about the function to resolve the
current nameservice for the current cluster.
For AMBARI-15506 I defined the following nameservice resolution:
1. split names by ','
2. for all services check if string is also contained in
dfs.namenode.shared.edits.dir . Typically this are
jounalnode1,journalnode2,journalnode3:port/servicename. Here it would
probably be better to verify the name with fs.defaultFS, but this is
part of core-site not hdfs-site, which would add a separate dependency.
For namenode_ha_utils.py for me this seemed like an issue, because
refactoring all the python scripts to also include the core-site seemed
much more involved.
3. A default fallback the first string in the list is used as the
nameservice.
Thanks and regards,
Henning