You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Sergey Shabalov <se...@metaintegration.com> on 2021/11/02 21:16:20 UTC
How to access Azure blob storage using Hadoop SDK 3.3.1 and Azure MSI authentication
I try to use a simple code
1. conf.set("fs.defaultFS","wasbs://sshcont01@sshblobhierarchyoff.blob.core.windows.net");
2. conf.set("fs.azure.ssl.channel.mode","Default_JSSE");
3. conf.set("fs.azure.account.oauth.provider.type","org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider");
4. conf.set("fs.azure.account.auth.type","OAuth");
5. conf.set("fs.azure.account.oauth2.msi.tenant","My Tenant ID");
6. conf.set("fs.azure.account.oauth2.client.id","My Client ID");
7. FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
but I have got:
org.apache.hadoop.fs.azure.AzureException:
org.apache.hadoop.fs.azure.AzureException: No credentials found for account
sshblobhierarchyoff.blob.core.windows.net in the configuration, and its
container sshcont01 is not accessible using anonymous credentials. Please
check if the container exists first. If it is not publicly available, you
have to provide account credentials.
Error [18:01:18] at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1098)
Error [18:01:18] at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:547)
Error [18:01:18] at
org.apache.hadoop.fs.azure.NativeAzureFilSystem.initialize(NativeAzureFileSystem.java:1379)
Error [18:01:18] at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
Error [18:01:18] at
org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
Error [18:01:18] at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
Error [18:01:18] at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
Error [18:01:18] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
Error [18:01:18] at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
But if I set "access level: "container" for this container it passes well.
We suppose the MSI should provide access to the container without making it
public.
BTW: MSI access to Azure Data Lake works properly
Thanks,
Sergiy
Re: How to access Azure blob storage using Hadoop SDK 3.3.1 and Azure
MSI authentication
Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Tue, 2 Nov 2021 at 21:16, Sergey Shabalov <
sergey.shabalov@metaintegration.com> wrote:
> I try to use a simple code
>
>
> 1. conf.set("fs.defaultFS","wasbs://
> sshcont01@sshblobhierarchyoff.blob.core.windows.net");
> 2. conf.set("fs.azure.ssl.channel.mode","Default_JSSE");
> 3.
> conf.set("fs.azure.account.oauth.provider.type","org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider");
>
that's an abfs class
> 4. conf.set("fs.azure.account.auth.type","OAuth");
> 5. conf.set("fs.azure.account.oauth2.msi.tenant","My Tenant
> ID");
> 6. conf.set("fs.azure.account.oauth2.client.id","My Client
> ID");
> 7. FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
>
> i think you;re trying to use abfs config options with a wasb url.
the abfs connector can connect to wasb stores -it just doesn't give you
"real" directories
>
> but I have got:
>
> org.apache.hadoop.fs.azure.AzureException:
> org.apache.hadoop.fs.azure.AzureException: No credentials found for account
> sshblobhierarchyoff.blob.core.windows.net in the configuration, and its
> container sshcont01 is not accessible using anonymous credentials. Please
> check if the container exists first. If it is not publicly available, you
> have to provide account credentials.
> Error [18:01:18] at
>
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1098)
> Error [18:01:18] at
>
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:547)
> Error [18:01:18] at
>
> org.apache.hadoop.fs.azure.NativeAzureFilSystem.initialize(NativeAzureFileSystem.java:1379)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
>
> But if I set "access level: "container" for this container it passes well.
> We suppose the MSI should provide access to the container without making it
> public.
>
> BTW: MSI access to Azure Data Lake works properly
>
>
> Thanks,
>
> Sergiy
>