[jira] [Created] (HUDI-5103) Does not work with Azure Data lake Gen2

["Dheeraj Panangat (Jira)" <ji...@apache.org>]

Dheeraj Panangat created HUDI-5103:
--------------------------------------

             Summary: Does not work with Azure Data lake Gen2
                 Key: HUDI-5103
                 URL: https://issues.apache.org/jira/browse/HUDI-5103
             Project: Apache Hudi
          Issue Type: Bug
            Reporter: Dheeraj Panangat


Unable to use Hudi with Flink for Azure Data Lake 
It tries to look for 
{code:java}
Caused by: Configuration property <datalakeaccount>.dfs.core.windows.net not found.
    at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174) {code}

Following properties are specified when running the Flink code : 
{code:java}
"fs.azure.account.auth.type": "OAuth",
       "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
       "fs.azure.account.oauth2.client.id": "<appId>",
       "fs.azure.account.oauth2.client.secret": "<clientSecret>",
       "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant>/oauth2/token",
       "fs.azure.createRemoteFileSystemDuringInitialization": "true" {code}
as per document : [Microsoft Azure Spark to Data Lake Gen2|https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-use-databricks-spark]

 

 

In AWS it works because it takes the credentials from the environment, but in Azure it needs to get it from config, which does not reach till the point where the FileSystem is initialized.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)