[jira] [Commented] (HADOOP-16068) ABFS Auth and DT plugins to be bound to specific URI of the FS

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16762878#comment-16762878 ] 

Steve Loughran commented on HADOOP-16068:
-----------------------------------------

one aspect of this patch is that it returns the FS URI in a call to AzureBlobFileSystem.getCanonicalServiceName() when DTs are turned on.

I'd always believed that you wouldn't get anything here, but actually the base class seems to return "52.239.143.194:0", which is the IPAddr of the store; with the new code the FS service name would become something like abfs://abfs-testcontainer-57aa1ca6-761b-4798-92ca-d05a9105f303@abfsaccount.dfs.core.windows.net  i..e unique for each FS instance. 

It is going to mark a change in behaviour. We'll need to talk to the implementors of any existing use of the DT plugin to see what 

Today the canonical service name of *all* ABFS filesystems will be that of the IPAddr of the {{abfsaccount.dfs.core.windows.net}} value, which turns out to be (for me). something in amsterdam:

{code}
dfs.ams06prdstr12a.store.core.windows.net.
Name:	dfs.ams06prdstr12a.store.core.windows.net
Address: 52.239.143.194
{code}

Which means that
# all abfs stores created with the same a/c share the same canonical name
# and on job submit, only one DT will be picked up.
# if the auth plugin for the FS uses its canonical service name for DT retrieval, then every FS will pick up the relevant credentials, even if only one FS instance has generated a DT
# and so if you call spark-submit for a job, and your cluster ifs is abfs://, then the retrieved DT can be used for every abfs:// container, even without you listing them in {{spark.yarn.access.hadoopFileSystems}}

What I'm going to do to retain this is have the bound token provider return that canonical service name, or null, with the base/unbound value being null, so falling back to the base class. With a test for this behaviour.


> ABFS Auth and DT plugins to be bound to specific URI of the FS
> --------------------------------------------------------------
>
>                 Key: HADOOP-16068
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16068
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/azure
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-16068-001.patch, HADOOP-16068-002.patch
>
>
> followup from HADOOP-15692: pass in the URI & conf of the owner FS to bind the plugins to the specific FS instance. Without that you can't have per FS auth
> +add a stub DT plugin for testing, verify that DTs are collected.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org