Unable to create connection to Azure Data Lake Gen2 with abfs: "Configuration property {storage_account}.dfs.core.windows.net not found"

[Preston Price <na...@gmail.com>]

Some details about my runtime/environment:
Java 11
Flink version 1.14.0
Running locally in IntelliJ

The error message that I am getting is: Configuration property
{storage_account}.dfs.core.windows.net not found.
Reading through all the docs hasn't yielded much help.

In the Flink docs here
<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/filesystems/azure/#credentials-configuration>,
it's claimed that we can set the credentials for ABFS by specifying the
value in flink-conf.yaml, so this is what I am trying. However, in the code
path expressed in the stack trace, I don't believe the configuration loaded
from flink-conf.yaml is ever consulted. Here are the relevant parts of the
stack trace:
Caused by:
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.contracts.exceptions.ConfigurationPropertyNotFoundException:
Configuration property {storage_account}.dfs.core.windows.net not found.
at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
at
org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
at
org.apache.flink.fs.azurefs.AbstractAzureFSFactory.create(AbstractAzureFSFactory.java:79)
~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
~[flink-core-1.14.0.jar:1.14.0]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
~[flink-core-1.14.0.jar:1.14.0]

When we first call org.apache.flink.core.fs.FileSystem.get the
FileSystemFactories are initialized with a new/empty configuration
<https://github.com/apache/flink/blob/release-1.14.0/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L497>,
not the configuration loaded from flink-conf.yaml Therefore, later on when
we're calling AbstractAzureFSFactory.create
<https://github.com/apache/flink/blob/release-1.14.0/flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AbstractAzureFSFactory.java#L74>
we
have an empty config, so the call
to HadoopConfigLoader.getOrLoadHadoopConfig() then
HadoopConfigLoader.loadHadoopConfigFromFlink
<http://HadoopConfigLoader.loadHadoopConfigFromFlinkhttps://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L95>
can't
merge in our config from flink-conf.yaml

So if the Configuration loaded from flink-conf.yaml isn't supplied to the
AbstractAzureFSFactory, how do we configure Flink to connect to Azure Data
Lake?

Thanks!

Re: Unable to create connection to Azure Data Lake Gen2 with abfs: "Configuration property {storage_account}.dfs.core.windows.net not found"

[Arvid Heise <ar...@apache.org>]

Hi Preston,

Before FileSystems are accessed, they are first initialized with the
configuration coming from the flink-conf. [1]
It seems that your local execution in IDE is bypassing that, so I'd call
that manually. Which entry point are you using? We should probably fix it,
such that it's also calling initialize.
Also make sure your config options are prefixed with (fs.)azure. because
only they are relayed.

[1]
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L319-L319

On Wed, Oct 20, 2021 at 12:30 AM Preston Price <na...@gmail.com> wrote:

> Some details about my runtime/environment:
> Java 11
> Flink version 1.14.0
> Running locally in IntelliJ
>
> The error message that I am getting is: Configuration property
> {storage_account}.dfs.core.windows.net not found.
> Reading through all the docs hasn't yielded much help.
>
> In the Flink docs here
> <https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/filesystems/azure/#credentials-configuration>,
> it's claimed that we can set the credentials for ABFS by specifying the
> value in flink-conf.yaml, so this is what I am trying. However, in the
> code path expressed in the stack trace, I don't believe the configuration
> loaded from flink-conf.yaml is ever consulted. Here are the relevant
> parts of the stack trace:
> Caused by:
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.contracts.exceptions.ConfigurationPropertyNotFoundException:
> Configuration property {storage_account}.dfs.core.windows.net not found.
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AbfsConfiguration.getStorageAccountKey(AbfsConfiguration.java:372)
> ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.initializeClient(AzureBlobFileSystemStore.java:1133)
> ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:174)
> ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
> at
> org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:110)
> ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
> at
> org.apache.flink.fs.azurefs.AbstractAzureFSFactory.create(AbstractAzureFSFactory.java:79)
> ~[flink-azure-fs-hadoop-1.14.0.jar:1.14.0]
> at
> org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:508)
> ~[flink-core-1.14.0.jar:1.14.0]
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
> ~[flink-core-1.14.0.jar:1.14.0]
>
> When we first call org.apache.flink.core.fs.FileSystem.get the
> FileSystemFactories are initialized with a new/empty configuration
> <https://github.com/apache/flink/blob/release-1.14.0/flink-core/src/main/java/org/apache/flink/core/fs/FileSystem.java#L497>,
> not the configuration loaded from flink-conf.yaml Therefore, later on
> when we're calling AbstractAzureFSFactory.create
> <https://github.com/apache/flink/blob/release-1.14.0/flink-filesystems/flink-azure-fs-hadoop/src/main/java/org/apache/flink/fs/azurefs/AbstractAzureFSFactory.java#L74> we
> have an empty config, so the call
> to HadoopConfigLoader.getOrLoadHadoopConfig() then
> HadoopConfigLoader.loadHadoopConfigFromFlink
> <http://HadoopConfigLoader.loadHadoopConfigFromFlinkhttps://github.com/apache/flink/blob/99c2a415e9eeefafacf70762b6f54070f7911ceb/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopConfigLoader.java#L95> can't
> merge in our config from flink-conf.yaml
>
> So if the Configuration loaded from flink-conf.yaml isn't supplied to the
> AbstractAzureFSFactory, how do we configure Flink to connect to Azure Data
> Lake?
>
> Thanks!
>