You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 06:01:23 UTC

[I] Does not work with Azure Data lake Gen2 [hudi]

hudi-bot opened a new issue, #15523:
URL: https://github.com/apache/hudi/issues/15523

   Unable to use Hudi with Flink for Azure Data Lake 
   It tries to look for 
   {code:java}
   Caused by: Configuration property <datalakeaccount>.dfs.core.windows.net not found.
       at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
       at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
       at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174) {code}
   
   Following properties are specified when running the Flink code : 
   {code:java}
   "fs.azure.account.auth.type": "OAuth",
          "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
          "fs.azure.account.oauth2.client.id": "<appId>",
          "fs.azure.account.oauth2.client.secret": "<clientSecret>",
          "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant>/oauth2/token",
          "fs.azure.createRemoteFileSystemDuringInitialization": "true" {code}
   as per document : [Microsoft Azure Spark to Data Lake Gen2|https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-use-databricks-spark]
   
    
   
    
   
   In AWS it works because it takes the credentials from the environment, but in Azure it needs to get it from config, which does not reach till the point where the FileSystem is initialized.
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-5103
   - Type: Bug
   
   
   ---
   
   
   ## Comments
   
   16/Nov/22 05:11;dheerajpanangat;There are couple of ways to resolve this : 
   1. Add core-site.xml to resources :
   {code:java}
   <?xml version="1.0"?>
   <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
   
   <configuration>
       <property>
           <name>fs.hdfs.impl</name>
           <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
       </property>
       <property>
           <name>fs.file.impl</name>
           <value>org.apache.hadoop.fs.LocalFileSystem</value>
       </property>
       <property>
           <name>fs.azure.account.auth.type</name>
           <value>OAuth</value>
       </property>
       <property>
           <name>fs.azure.account.oauth.provider.type</name>
           <value>org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider</value>
       </property>
       <property>
           <name>fs.azure.account.oauth2.client.id</name>
           <value>****</value>
       </property>
       <property>
           <name>fs.azure.account.oauth2.client.secret</name>
           <value>**********</value>
       </property>
       <property>
           <name>fs.azure.account.oauth2.client.endpoint</name>
           <value>https://login.microsoftonline.com/***********/oauth2/token</value>
       </property>
       <property>
           <name>fs.azure.createRemoteFileSystemDuringInitialization</name>
           <value>true</value>
       </property>
   </configuration> {code}
   
   2. Pass this in the Flink Table config by appending 'hadoop.' prefix
   {code:java}
   "hadoop.fs.azure.account.auth.type": "OAuth",
          "hadoop.fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
          "hadoop.fs.azure.account.oauth2.client.id": "<appId>",
          "hadoop.fs.azure.account.oauth2.client.secret": "<clientSecret>",
          "hadoop.fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<tenant>/oauth2/token",
          "hadoop.fs.azure.createRemoteFileSystemDuringInitialization": "true"  {code}
   Thanks;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Does not work with Azure Data lake Gen2 [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #15523:
URL: https://github.com/apache/hudi/issues/15523#issuecomment-5019047200

   This issue was reviewed as part of the JIRA-migrated backlog triage.
   
   Findings: this is a configuration/usage issue, not a Hudi code bug, and the linked PR #7097 was closed without merging. Flink does not forward `fs.azure.*` credentials to the underlying Hadoop FileSystem unless they are supplied as Hadoop config. 
   
   Documented workarounds: 
   (a) add a `core-site.xml` containing the `fs.azure.*` properties to the classpath, or 
   (b) pass the same keys via Flink config prefixed with `hadoop.` (e.g. `hadoop.fs.azure.account.oauth2.[client.id](http://client.id/)`). The report is >3.5 years old and requires no code fix.
   
   Closing as a usage issue / insufficient to reproduce. Please reopen against a current release if a genuine code defect remains after applying the config above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Does not work with Azure Data lake Gen2 [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15523:
URL: https://github.com/apache/hudi/issues/15523#issuecomment-3630120289

   ### Linked PR(s)
   
   - https://github.com/apache/hudi/pull/7097


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Does not work with Azure Data lake Gen2 [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #15523: Does not work with Azure Data lake Gen2
URL: https://github.com/apache/hudi/issues/15523


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org