You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Dheeraj Panangat (Jira)" <ji...@apache.org> on 2023/01/18 14:45:00 UTC

[jira] [Created] (FLINK-30745) Check-pointing with Azure Data Lake Storage

Dheeraj Panangat created FLINK-30745:
----------------------------------------

             Summary: Check-pointing with Azure Data Lake Storage
                 Key: FLINK-30745
                 URL: https://issues.apache.org/jira/browse/FLINK-30745
             Project: Flink
          Issue Type: Bug
    Affects Versions: 1.14.6, 1.15.2
            Reporter: Dheeraj Panangat


Hi,

While checkpointing to Azure Blob Storage using Flink, we get the following error :
Caused by: Configuration property <accoutnname>.dfs.core.windows.net not found.
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)

We have given the configurations in core-site.xml too for following
fs.hdfs.impl
fs.abfs.impl -> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem
fs.file.impl
fs.azure.account.auth.type
fs.azure.account.oauth.provider.type
fs.azure.account.oauth2.client.id
fs.azure.account.oauth2.client.secret
fs.azure.account.oauth2.client.endpoint
fs.azure.createRemoteFileSystemDuringInitialization -> true

On debugging found that flink reads from core-default-shaded.xml, but even if the properties are specified there, the default configs are not loaded and we get a different exception as :
Caused by: Unable to load key provider class.
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getTokenProvider(AbfsConfiguration.java:540)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1136)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
	at org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)