You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jitendra Nath Pandey (Created) (JIRA)" <ji...@apache.org> on 2012/01/25 03:42:44 UTC
[jira] [Created] (HDFS-2839) Nameservice id in file uri could cause
issues
Nameservice id in file uri could cause issues
---------------------------------------------
Key: HDFS-2839
URL: https://issues.apache.org/jira/browse/HDFS-2839
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HA branch (HDFS-1623)
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey
There might be two issues with having a nameservice id in fully qualified paths of the files.
1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) Nameservice id in file uri could
cause issues
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201691#comment-13201691 ]
Aaron T. Myers commented on HDFS-2839:
--------------------------------------
Jitendra, do Todd's responses address your concerns? Unless you have a specific bug in mind with using the nameservice ID in the URI, I'm inclined to close this JIRA.
> Nameservice id in file uri could cause issues
> ---------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> There might be two issues with having a nameservice id in fully qualified paths of the files.
> 1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
> 2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) Nameservice id in file uri could
cause issues
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202172#comment-13202172 ]
Aaron T. Myers commented on HDFS-2839:
--------------------------------------
bq. But #2 is still open and needs discussion. I think it is an important issue because some applications do store the file paths. I agree it is not an highest priority thing right now, but I would recommend keeping the jira open so that it stays in the radar.
Isn't this identical to if a federated cluster is running, but some remote client on another cluster doesn't have viewfs configured appropriately? It seems that in any system where there's a client doing some resolution using a logical URI, that this necessitates some client-side configuration. I don't see any way around it.
> Nameservice id in file uri could cause issues
> ---------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> There might be two issues with having a nameservice id in fully qualified paths of the files.
> 1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
> 2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2839) HA: Remove need for client
configuration of nameservice ID
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2839:
---------------------------------
Description:
The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
was:
There might be two issues with having a nameservice id in fully qualified paths of the files.
1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
Summary: HA: Remove need for client configuration of nameservice ID (was: Nameservice id in file uri could cause issues)
OK, that's fine. I've updated the JIRA summary and description to reflect this.
> HA: Remove need for client configuration of nameservice ID
> ----------------------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) Nameservice id in file uri could
cause issues
Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202692#comment-13202692 ]
Jitendra Nath Pandey commented on HDFS-2839:
--------------------------------------------
There could be ways around it for example we can be creative with the format of nameservice id, something that encodes the hostnames of primary and standby in the nameservice id. I am not saying it is the best idea, but I am reluctant to dismiss this issue as too hard to even think about. It is similar to viewfs issue, but viewfs use cases do require the viewfs authorities to be agnostic to underlying nameservices. For HA this requirement can be relaxed.
> Nameservice id in file uri could cause issues
> ---------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> There might be two issues with having a nameservice id in fully qualified paths of the files.
> 1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
> 2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2839) Nameservice id in file uri could cause
issues
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2839:
---------------------------------
Component/s: name-node
hdfs client
ha
Target Version/s: HA branch (HDFS-1623)
> Nameservice id in file uri could cause issues
> ---------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> There might be two issues with having a nameservice id in fully qualified paths of the files.
> 1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
> 2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2839) HA: Remove need for client
configuration of nameservice ID
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2839:
---------------------------------
Target Version/s: 0.24.0 (was: HA branch (HDFS-1623))
Affects Version/s: (was: HA branch (HDFS-1623))
0.24.0
Converting to top-level issue with commit of HDFS-1623.
> HA: Remove need for client configuration of nameservice ID
> ----------------------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: 0.24.0
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) HA: Remove need for client
configuration of nameservice ID
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202821#comment-13202821 ]
Todd Lipcon commented on HDFS-2839:
-----------------------------------
One possibility we'd discussed last year is to use DNS "SRV" records, and have a different ConfiguredClientProxyProvider implementation that grabs the list of NNs from the SRV instead of the local conf.
Of course the question then is how the unconfigured client knows to even look for the SRV record -- unless this becomes a default fallback in response to a "host not found" -- ie if hdfs://foo.example.com is used, but there is no A or CNAME record, then assume HA and fall back to SRV?
> HA: Remove need for client configuration of nameservice ID
> ----------------------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) HA: Remove need for client
configuration of nameservice ID
Posted by "Daryn Sharp (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240500#comment-13240500 ]
Daryn Sharp commented on HDFS-2839:
-----------------------------------
Comments. They'll reference and span both solution A & B to hopefully provide clarity.
* NN has a logical name that is authority in URI (hdfs://logicalName/path)
* (Requirement B.5) The non-HA NN’s DNS name is the logical name of NN.
I think this may be your intent, so would you please clarify/confirm? The logical name (URI's authority) for the NN should always be, for both HA and non-HA, a DNS name with a primary mapping to the main NN. The logical name may be a CNAME (see below) in the case of HA. This will allow both HA and non-HA aware clients to access the cluster. The non-HA aware clients just won't failover.
* The LogicalName is the DNS name that maps to a single IP which failsover
* DNS-Resolver - DNS-Name to IP or IPs (single IP in case of IP-Failover)
These two statements seem contradictory about whether the the logical name (URI authority) has a dns mapping to 1 or many hosts. I'm assuming that the "single IP" approach would rely on config settings for failover support?
Here's how I would envision a dns configuration to support both HA and non-HA:
* A record for logical name: nn.domain -> IP
** Non-HA client works as it does now.
** HA client has a single resolution so there's no failover.
* CNAME for logical name: nn.domain -> nn1.domain & nn2.domain
** The non-HA client works as it does now. The CNAME is resolved to the primary address (nn1.domain). No code change required.
** The HA client can specifically query for all resolutions to build the failover list.
In fact, the HA aware client could be "smart" and instantiate the HA RPC proxy only if the logical name has multiple resolutions. The HA aware client resolves the logical name with {{getAllByName}}, instead of {{getByName}}, to find the multiple mappings for HA.
Regarding cross cluster access:
* Cross cluster access must be supported
* Cross cluster – The logical to IP mapping must be available across clusters
* ConfigFile-resolver - the mapping in the config file - this config file will need to be be available in all clusters, for all clusters to allow cross cluster access.
I'm uneasy about propagating the current model where clients require a lot of config info about remote clusters. It becomes a maintenance burden to keep them in sync, more so when some users have their configs. Favoring the dns/resolver approach should minimize the need to sync all cluster configs for HA.
> HA: Remove need for client configuration of nameservice ID
> ----------------------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: ha, hdfs client, name-node
> Affects Versions: 0.24.0
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) Nameservice id in file uri could
cause issues
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192842#comment-13192842 ]
Todd Lipcon commented on HDFS-2839:
-----------------------------------
For #1 above, you could configure the "logical URI" of the HA-ified cluster to match exactly the hostname of the non-HA cluster.
For #2, I don't see any way around it -- you need some way of finding the hosts... unless you are proposing a VIP based client failover approach here?
> Nameservice id in file uri could cause issues
> ---------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> There might be two issues with having a nameservice id in fully qualified paths of the files.
> 1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
> 2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2839) HA: Remove need for client
configuration of nameservice ID
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2839:
---------------------------------
Issue Type: New Feature (was: Sub-task)
Parent: (was: HDFS-1623)
> HA: Remove need for client configuration of nameservice ID
> ----------------------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: ha, hdfs client, name-node
> Affects Versions: 0.24.0
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) Nameservice id in file uri could
cause issues
Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202142#comment-13202142 ]
Jitendra Nath Pandey commented on HDFS-2839:
--------------------------------------------
For #1 the workaround should work. But #2 is still open and needs discussion.
I think it is an important issue because some applications do store the file paths. I agree it is not an highest priority thing right now, but I would recommend keeping the jira open so that it stays in the radar.
> Nameservice id in file uri could cause issues
> ---------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, hdfs client, name-node
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> There might be two issues with having a nameservice id in fully qualified paths of the files.
> 1) Some applications might be storing the fully qualified paths. They will have hostnames from pre-ha installations, and they might break after the failover.
> 2) The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) HA: Remove need for client
configuration of nameservice ID
Posted by "Sanjay Radia (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227729#comment-13227729 ]
Sanjay Radia commented on HDFS-2839:
------------------------------------
Lets look at the bigger picture of the various connections to the NN, the requirements and possible solutions.
*Requirements*
# A DfsClient must be redirected when failover occurs
# Http client must be redirected when failover occurs
# Delegation tokens
** delegation tokens should work for the new active, even if issued by previous active
** renewal - renewer must be able to renew at the new active transparently
# Cross cluster access must be supported
# URI compatibility when transitioning nonHA-to-HA and HA-to-nonHA: A URI to NN that contains a name (not ip address) should continue to work.
# Solution must work in
** Experimental seeting where DNS changes may not be permitted and also
** Production setting where the customer wants to use DNS to locate NNs.
*Solutions*
*Solution A: IP Failover*
* Ip Failover satisfies all requirements as long as clients (DFSClient, Token renewer etc) reconnect on failure
*Solution B:*
* NN has a logical name that is authority in URI (hdfs://logicalName/path)
* Logical name is mapped to Active and Standby’s IP (issues discussed below)
* *Addressing the requirements in Solution B*
*# DNS Client uses the multiple IPs
*# HTTP - HTTP Proxy or DNS Round robin with standby redirecting
*# Delegation token
*** Delegation token secret is shared with standby
*** Renewer – logical name is CanonicalService name and a proxy provider object maps to the IPs.
*# Cross cluster – The logical to IP mapping must be available across clusters
*# The non-HA NN’s DNS name is the logical name of NN.
*# Supply 2 resolvers – config file resolver and DNS resolver. Later we can add ZK resolver.
*What is the Logical Name?*
Since the logical name is being exposed it should be the volumeName. Note this is distinct from NameServiceId since the NameServiceId was supposed to allow a name server to store multiple Volumes.
* Proposal: Add VolumeName to config
** Currently a 1-1 mapping between Volume Name and NameServiceId since a NameService can host only one volume.
** Why do this now? the logical name is being exposed via the URI - we should select the right terminology now
** How are Volume names mapped to IPs? (See below)
*Is the notion of Logical name used when using the IP failover Solution*
Yes – the LogicalName is the DNS name that maps to a single IP which failsover
*Where is the LogicalName -to-IPs Mapping Stored?*
* DNS-Resolver - DNS-Name to IP or IPs (single IP in case of IP-Failover)
* ConfigFile-resolver - the mapping in the config file - this config file will need to be be available in all clusters, for all clusters to allow cross cluster access.
This configuration file will be in addition to the client side mount tables. May want to consider having a single conf file that resolves mount points as well as logical names?
* Zookeeper-resolver (Volume management at some point should move to Zookeeper )
Scalability of zookeeper may need to be addressed.
> HA: Remove need for client configuration of nameservice ID
> ----------------------------------------------------------
>
> Key: HDFS-2839
> URL: https://issues.apache.org/jira/browse/HDFS-2839
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: ha, hdfs client, name-node
> Affects Versions: 0.24.0
> Reporter: Jitendra Nath Pandey
> Assignee: Jitendra Nath Pandey
>
> The fully qualified path from an ha cluster, won't be usable from a different cluster that doesn't know about that particular namespace id.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira