You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "samuelkhtu (via GitHub)" <gi...@apache.org> on 2025/07/14 08:41:49 UTC
[I] `account_host` missing in `get_fs()` in adls.py for AzureBlobFileSystem Body [airflow]
samuelkhtu opened a new issue, #53333:
URL: https://github.com/apache/airflow/issues/53333
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.10.5
### What happened?
In `airflow/providers/microsoft/azure/fs/adls.py`, the `get_fs()` function constructs a dictionary of `options` by pulling connection information from Airflow's connection system and then passes these options to `AzureBlobFileSystem(**options)`.
By default, the function constructs `account_url` using `parse_blob_account_url(conn.host, conn.login)`, which assumes the Azure Blob endpoint will use the standard `core.windows.net` domain. While this works for default endpoints, it does not support scenarios where users want to override the domain — for example, when using a private endpoint like `.core.mydomain.io`.
The root issue is:
* `account_host` (the correct field expected by `adlfs.AzureBlobFileSystem`) is not included in the list of parsed fields.
* Even if the user provides `account_host` in the connection extras, `get_fs()` ignores it and always constructs `account_url` using the hardcoded domain logic.
* `AzureBlobFileSystem` does not support `account_url` as a constructor parameter, so the custom domain is never applied — silently falling back to the default.
As a result, there is **no way for users to override the account URL** via Airflow connection configuration, even though `adlfs.AzureBlobFileSystem` supports this through its `account_host` parameter.
This blocks use cases such as:
* Custom domains
* Private endpoints
* Sovereign or air-gapped cloud regions
This limitation exists even though the underlying library (`adlfs`) already supports the necessary parameter (`account_host`).
### What you think should happen instead?
### What you think should happen instead?
The `get_fs()` function should support passing a user-defined `account_host` value from the Airflow connection extras directly to the `AzureBlobFileSystem` constructor.
Specifically:
* Add `"account_host"` to the list of fields extracted from `extras`.
* If `account_host` is provided, it should be passed directly to `AzureBlobFileSystem` as a supported parameter.
* Currently, `get_fs()` sets `account_url` using `parse_blob_account_url(...)`, but `account_url` is **not** a valid parameter for `AzureBlobFileSystem`. It can be removed or renamed to `account_host`.
However, to maintain backward compatibility:
* We could retain the existing `account_url` logic as a fallback.
* But prefer `account_host` when it is explicitly defined in the extras.
This would allow users to configure non-standard Azure Blob endpoints — such as custom domains or private links — via the standard Airflow connection mechanism, while maintaining compatibility with existing deployments.
### How to reproduce
1. Create an Airflow `Connection` object with a custom domain in the `extra` field:
```python
from airflow.models import Connection
from airflow.providers.microsoft.azure.fs.adls import get_fs
conn = Connection(
conn_id="testconn",
conn_type="wasb",
login="testaccountname",
password="p",
host="testaccountID",
extra={
"account_name": "n",
"tenant_id": "t",
"account_host": "https://testaccountname.blob.core.customdomain.io",
},
)
# Insert or mock this connection in Airflow metadata
```
2. Call `get_fs()` with this connection ID:
```python
fs = get_fs("testconn")
```
3. Observe that:
* Despite `account_host` being set to a custom domain in extras, `get_fs()` ignores it.
* The `options` passed to `adlfs.AzureBlobFileSystem` do **not** include `account_host`.
* Instead, `get_fs()` builds and passes an `account_url` derived from the default domain based on `host` and `login`.
* As a result, the custom domain override does not take effect.
This confirms the current limitation that `account_host` cannot be used to override the default Azure Blob endpoint via Airflow’s connection system.
### Operating System
MacOS 15.5
### Versions of Apache Airflow Providers
apache-airflow-providers-microsoft-azure==12.5.0
### Deployment
Astronomer
### Deployment details
k8s
### Anything else?
* This is a backward-compatible enhancement since adding support for `account_host` does not remove or change existing parameters.
* Supporting `account_host` enables Airflow to better integrate with Azure environments using private endpoints, custom domains, or sovereign clouds.
* The underlying `adlfs.AzureBlobFileSystem` already supports `account_host`, so this change leverages existing functionality.
* Implementing this will improve user experience and reduce the need for workarounds or custom patches.
* I want to submit a PR but would appreciate suggestions on the best approach.
* My current thinking is to simply add `"account_host"` to the existing `fields` list in `get_fs()` so that this block picks it up automatically:
```python
fields = [
"account_name",
"account_key",
"sas_token",
"tenant_id",
"managed_identity_client_id",
"workload_identity_client_id",
"workload_identity_tenant_id",
"anon",
"account_host", # <- add here
]
for field in fields:
value = get_field(conn_id=conn_id, conn_type=conn_type, extras=extras, field_name=field)
if value is not None:
if value == "":
options.pop(field, "")
else:
options[field] = value
```
* Would this be the preferred way, or are there alternative approaches to consider?
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `account_host` missing in `get_fs()` in adls.py for AzureBlobFileSystem Body [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #53333:
URL: https://github.com/apache/airflow/issues/53333#issuecomment-3070629859
@samuelkhtu, would you like this assigned to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `account_host` missing in `get_fs()` in adls.py for AzureBlobFileSystem Body [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #53333: `account_host` missing in `get_fs()` in adls.py for AzureBlobFileSystem Body
URL: https://github.com/apache/airflow/issues/53333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `account_host` missing in `get_fs()` in adls.py for AzureBlobFileSystem Body [airflow]
Posted by "samuelkhtu (via GitHub)" <gi...@apache.org>.
samuelkhtu commented on issue #53333:
URL: https://github.com/apache/airflow/issues/53333#issuecomment-3071533964
Thank you @jroachgolf84 comment, that's a great question!
The `account_host` is expected to be a **full hostname**, such as:
```
testaccountname.blob.core.customdomain.io
```
This is **not** just the default domain (`blob.core.windows.net`), and it’s also **not** the full URL with `https://` — the `AzureBlobFileSystem` constructor internally prepends `https://` when building the `account_url`.
---
### 🔍 Example Use Case
Here’s a real-world example of how this is configured in Airflow:
```json
{
"conn_type": "wasb",
"description": "Connection for Azure XCom Backend",
"login": "<Azure client id>",
"password": "<Azure client secret>",
"host": "https://myblobstorageaccount.blob.core.mynet.io",
"extra": {
"tenant_id": "<Azure Tenant ID>",
"account_name": "testaccount"
// Note: the custom domain name (myblobstorageaccount) is different from the actual Azure Storage account resource name.
}
}
```
In this case, the `get_fs()` function returns:
```python
return AzureBlobFileSystem(**options)
```
Where `options` includes:
```json
{
"account_url": "https://myblobstorageaccount.blob.core.mynet.io",
"client_id": "<Azure client ID>",
"client_secret": "<Azure client secret>",
"account_name": "testaccount",
"tenant_id": "<Azure Tenant ID>"
}
```
---
### ⚠️ Why `account_host` Matters
Inside the `AzureBlobFileSystem` constructor, the logic for setting `account_url` is:
```python
if hasattr(self, "account_host"):
self.account_url = f"https://{self.account_host}"
else:
self.account_url = f"https://{self.account_name}.blob.core.windows.net"
```
If `account_host` is **not passed**, the fallback logic uses the default domain — which breaks the connection for users relying on custom endpoints.
This is exactly what happens if `account_host` is not extracted from the Airflow connection extras and passed into `options`.
---
### 🤔 Why not just use `host`?
In Airflow, the `host` field is part of the base connection object, but it’s not automatically mapped to `account_host`. Instead, provider-specific parameters like `account_host` must be explicitly extracted from the `extra` dictionary.
This separation allows Airflow to maintain a consistent interface across providers while still supporting advanced configurations like custom Azure domains.
---
### ✅ Summary
- `account_host` should be a full hostname like `myblobstorageaccount.blob.core.mynet.io`.
- It must be explicitly passed to `AzureBlobFileSystem` to override the default domain.
- Without it, the fallback logic incorrectly assumes `blob.core.windows.net`, which fails for custom/private endpoints.
- The current implementation in Airflow needs to be updated to include `account_host` in the list of extracted fields.
Let me know if you have any suggestion!
Yes, please assign the PR to me. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `account_host` missing in `get_fs()` in adls.py for AzureBlobFileSystem Body [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #53333:
URL: https://github.com/apache/airflow/issues/53333#issuecomment-3070674182
What is the typical form that an `account_host` would take? Would it be something like this?
`blob.core.windows.net`
Or would it be the entire URL, more like this?
`https://testaccountname.blob.core.customdomain.io`
If the latter, what would happy if a `host` and `login` were provided? Why not just use the `host` itself, rather than setting an `account_host`?
This is the logic that's being implemented right now:
```python
def parse_blob_account_url(host: str | None, login: str | None) -> str:
account_url = host if host else f"https://{login}.blob.core.windows.net/"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `account_host` missing in `get_fs()` in adls.py for AzureBlobFileSystem Body [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #53333:
URL: https://github.com/apache/airflow/issues/53333#issuecomment-3068434957
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org