You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "baha-bouali (via GitHub)" <gi...@apache.org> on 2026/08/04 08:56:13 UTC
[PR] Azure batch 15x migration [airflow]
baha-bouali opened a new pull request, #71071:
URL: https://github.com/apache/airflow/pull/71071
closes: #66466
related: #67600
`azure-batch` 15.x is a ground-up rewrite of Azure's Python SDK (track 1 → track 2); `BatchServiceClient` and the old model classes are gone. #66452 capped `azure-batch<15.0.0` as a stopgap; #66466 tracks the real migration to lift it.
### The migration (#67600, by @arieleli01212)
This PR carries forward @arieleli01212's work from [#67600](https://github.com/apache/airflow/pull/67600) as-is: `BatchServiceClient` → `BatchClient`, shared-key auth moved to `AzureNamedKeyCredential`, the renamed model classes (`PoolAddParameter` → `BatchPoolCreateOptions`, `CloudTask` → `BatchTask`, etc.), the move to the new flat client API, dropping `CloudServiceConfiguration`/`os_family`/`os_version`, and the matching test rewrites.
### What this PR adds on top
Two commits on top of his 16, kept separate:
**1. Rebase onto current `main`.** One real conflict, in `docs/changelog.rst` (an unrelated `14.0.0` entry had landed since). Resolved by keeping it and placing this migration's pending note above it.
**2. Fix for the identity-auth path**, addressing [@aaron-y-chen's open review comment](https://github.com/apache/airflow/pull/67600#pullrequestreview-4679686593)(https://github.com/apache/airflow/pull/67600#pullrequestreview-4679686593) on the `# type: ignore[arg-type]` next to the `BatchClient` credential. The identity-based branch of `get_conn()` built an `AzureIdentityCredentialAdapter`, which subclasses msrest's track-1 `BasicTokenAuthentication` and doesn't implement `get_token()` — what azure-core's `TokenCredential` protocol requires, and what `BatchClient` needs to authenticate. `AzureBaseHook.get_token()` in this provider already documents this and raises `AttributeError` for it. So this path passed every mocked test but would fail against a real account. Switched to `get_sync_default_azure_credential()`, already used by `AzureBaseHook` and `asb.py` for this exact purpose, which returns a real `DefaultAzureCredential` that does implement `get_token()`. The `type: igno
re` is removed; updated the matching test; added a changelog line.
### Testing
`pytest providers/microsoft/azure/tests/unit/microsoft/azure/hooks/test_batch.py` passes.
[@eladkal asked](https://github.com/apache/airflow/pull/67600#issuecomment-4850934389) whether this had been tested against a live account. Since the bug above only shows up once the SDK actually tries to authenticate, I validated it two ways:
**1. Credential check (no Azure account needed):**
```python
hasattr(AzureIdentityCredentialAdapter(None, resource_id="https://batch.core.windows.net/.default"), "get_token") # False
hasattr(get_sync_default_azure_credential(), "get_token") # True
```
**2. Live pool lifecycle against a real Azure Batch account** (Azure for Students, Sweden Central), identity-based auth via Azure CLI login, no shared key:
```python
from airflow.providers.microsoft.azure.hooks.batch import AzureBatchHook
hook = AzureBatchHook(azure_batch_conn_id="test_live_azure_batch")
client = hook.get_conn()
pool = hook.configure_pool(
pool_id="claude-smoketest-pool",
vm_publisher="canonical",
vm_offer="0001-com-ubuntu-server-jammy",
vm_sku="22_04-lts",
vm_node_agent_sku_id="batch.node.ubuntu 22.04",
vm_size="Standard_A1_v2",
target_dedicated_nodes=0,
)
client.create_pool(pool)
pools = list(client.list_pools())
client.begin_delete_pool("claude-smoketest-pool").result()
```
Real HTTP round trips, no mocking:
```
DefaultAzureCredential acquired a token from AzureCliCredential
POST /pools → 201, pool created
GET /pools → 200, pool visible
DELETE /pools/claude-smoketest-pool → 202, then 404 on follow-up : confirmed deleted
```
No `AttributeError`: the exact failure mode of the pre-fix code. Test resources were deleted after validation.
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Sonnet 5), reviewed and tested by the submitter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Azure batch 15x migration [airflow]
Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #71071:
URL: https://github.com/apache/airflow/pull/71071#discussion_r3892636514
##########
providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/batch.py:
##########
@@ -253,7 +241,6 @@ def _check_inputs(self) -> Any:
def execute(self, context: Context) -> None:
self._check_inputs()
- self.hook.connection.config.retry_policy = self.batch_max_retries
Review Comment:
The `batch_max_retries` is not used anymore. I create a followup PR for this.
https://github.com/apache/airflow/pull/72317
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Azure batch 15x migration [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on PR #71071:
URL: https://github.com/apache/airflow/pull/71071#issuecomment-5291649536
@potiuk CI is fully green now and all checks passing. Ready for another look whenever you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Azure batch 15x migration [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on PR #71071:
URL: https://github.com/apache/airflow/pull/71071#issuecomment-5326811688
cc @eladkal. Could you please take another look. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Azure batch 15x migration [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #71071:
URL: https://github.com/apache/airflow/pull/71071
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Azure batch 15x migration [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #71071:
URL: https://github.com/apache/airflow/pull/71071#discussion_r3798471743
##########
providers/keycloak/docs/changelog.rst:
##########
Review Comment:
This is not relevant for the change you are making
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Azure batch 15x migration [airflow]
Posted by "baha-bouali (via GitHub)" <gi...@apache.org>.
baha-bouali commented on code in PR #71071:
URL: https://github.com/apache/airflow/pull/71071#discussion_r3800052015
##########
providers/keycloak/docs/changelog.rst:
##########
Review Comment:
My bad, that was a stray file from an earlier local git mistake, unrelated to this change. Removed in the latest push. Thanks for catching it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Azure batch 15x migration [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #71071:
URL: https://github.com/apache/airflow/pull/71071#discussion_r3803901310
##########
generated/known_airflow_exceptions.txt:
##########
@@ -351,7 +351,7 @@ providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_facto
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_lake.py::1
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/msgraph.py::3
providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/wasb.py::2
-providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/batch.py::9
+providers/microsoft/azure/src/airflow/providers/microsoft/azure/operators/batch.py::8
Review Comment:
I think we need to get to a point where this entry is removed. can you look into this in a followup PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org