You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Vamsi-klu (via GitHub)" <gi...@apache.org> on 2026/06/14 02:52:08 UTC
[PR] Add proxy support to Databricks connections [airflow]
Vamsi-klu opened a new pull request, #68527:
URL: https://github.com/apache/airflow/pull/68527
Add explicit proxy support for Databricks connections.
## What changed
This adds a documented `proxies` connection extra for the Databricks provider. The extra is intentionally allowlisted and only accepts `http` and `https` keys, for example:
```json
{
"proxies": {
"http": "http://proxy.example.com:8080",
"https": "http://proxy.example.com:8443"
}
}
```
The configured proxy is applied to:
- Databricks REST API calls
- Databricks OAuth token exchanges
- Azure Identity token acquisition for AAD service principal and default Azure credential authentication
- Sync and async Databricks hook paths
## Why
Users running Airflow behind a corporate proxy could not authenticate to Azure Databricks because token acquisition to Azure/Databricks endpoints did not receive proxy configuration.
This fixes #32576 while keeping connection extras explicit and safe. The implementation does not pass arbitrary connection extras through to client libraries; it validates and forwards only the reviewed `proxies` extra.
## Testing
I added coverage for:
- Valid `proxies` parsing from Databricks connection extras
- Rejection of invalid proxy shapes and unsupported keys
- Sync Databricks REST API calls using proxies
- Async Databricks REST API calls using the matching `aiohttp` proxy
- AAD service principal token paths passing proxies to Azure Identity
- Databricks service-principal OAuth token exchange using proxies
- Sync and async token/API combinations
Commands run:
- `uv run ruff format providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py`
- `uv run ruff check --fix providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py`
- `uv run --project providers/databricks pytest providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py -q`
Result: `228 passed`
closes: #32576
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes — Codex (GPT-5)
Generated-by: Codex (GPT-5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
Drafted-by: Codex (GPT-5) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410064879
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
Review Comment:
If not a dictionary, what type would this take?
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Have you validated that these args can be passed to `DefaultAzureCredential`?
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
+ if not isinstance(extra_dejson, dict):
+ return None
+
+ proxies = extra_dejson.get("proxies")
+ if proxies is None:
+ return None
+ if not isinstance(proxies, dict):
+ raise DatabricksProxyConfigurationError("Connection extra 'proxies' must be a JSON object.")
+
+ invalid_keys = set(proxies) - {"http", "https"}
+ if invalid_keys:
+ invalid_keys_str = ", ".join(sorted(invalid_keys))
+ raise DatabricksProxyConfigurationError(
+ f"Connection extra 'proxies' only supports 'http' and 'https' keys. Got: {invalid_keys_str}."
+ )
+
+ for proxy_scheme, proxy_url in proxies.items():
Review Comment:
Would it be an issue if both `http` and `https` were in `proxies`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3479479518
##########
providers/databricks/docs/connections/databricks.rst:
##########
@@ -81,6 +81,36 @@ Extra (optional)
* ``token``: Specify PAT to use. Consider to switch to specification of PAT in the Password field as it's more secure.
+ The following optional parameter can be used when Airflow workers need to access Databricks or Azure
+ token endpoints through an HTTP proxy:
+
+ * ``proxies``: JSON object with optional ``http`` and ``https`` keys, using the same shape as the
+ ``requests`` and Azure SDK ``proxies`` argument. Only these two keys are accepted. The configured proxy
+ is applied to Databricks REST API calls, Databricks OAuth token exchanges, and Azure Identity token
+ acquisition for AAD and default Azure credential authentication.
+
+ .. code-block:: json
+
+ {
+ "proxies": {
+ "http": "http://proxy.example.com:8080",
+ "https": "http://proxy.example.com:8443"
+ }
+ }
+
+ **Note:** The ``proxies`` extra is only needed for paths that do not already pick up the standard
+ ``HTTP_PROXY`` / ``HTTPS_PROXY`` / ``NO_PROXY`` environment variables. Synchronous REST API and token
+ requests use ``requests``, and Azure AAD / default-credential token acquisition uses the Azure Identity
+ SDK; both honor those environment variables by default. The asynchronous (deferrable operator and
+ triggerer) REST API and token paths use ``aiohttp`` with a session that does **not** trust the
+ environment, so proxy environment variables are ignored there and the ``proxies`` extra is required to
+ proxy them. Use the extra when you need a proxy on the asynchronous paths, when you want to force a
+ specific proxy regardless of the worker environment, or when only some endpoints should be proxied.
Review Comment:
Thanks elad, I will do this in couple of days. Will let you know once I do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "moomindani (via GitHub)" <gi...@apache.org>.
moomindani commented on PR #68527:
URL: https://github.com/apache/airflow/pull/68527#issuecomment-4805401041
Thanks for the correction — you're right that the Azure Identity SDK paths honor the env vars; it's the async `aiohttp` paths that don't. The doc note, the `ManagedIdentityCredential` comments, and the regression test all look good to me.
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @moomindani before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #68527:
URL: https://github.com/apache/airflow/pull/68527#issuecomment-4806182138
@moomindani any futher comments or are you approving this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #68527:
URL: https://github.com/apache/airflow/pull/68527
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3479555996
##########
providers/databricks/docs/connections/databricks.rst:
##########
@@ -81,6 +81,36 @@ Extra (optional)
* ``token``: Specify PAT to use. Consider to switch to specification of PAT in the Password field as it's more secure.
+ The following optional parameter can be used when Airflow workers need to access Databricks or Azure
+ token endpoints through an HTTP proxy:
+
+ * ``proxies``: JSON object with optional ``http`` and ``https`` keys, using the same shape as the
+ ``requests`` and Azure SDK ``proxies`` argument. Only these two keys are accepted. The configured proxy
+ is applied to Databricks REST API calls, Databricks OAuth token exchanges, and Azure Identity token
+ acquisition for AAD and default Azure credential authentication.
+
+ .. code-block:: json
+
+ {
+ "proxies": {
+ "http": "http://proxy.example.com:8080",
+ "https": "http://proxy.example.com:8443"
+ }
+ }
+
+ **Note:** The ``proxies`` extra is only needed for paths that do not already pick up the standard
+ ``HTTP_PROXY`` / ``HTTPS_PROXY`` / ``NO_PROXY`` environment variables. Synchronous REST API and token
+ requests use ``requests``, and Azure AAD / default-credential token acquisition uses the Azure Identity
+ SDK; both honor those environment variables by default. The asynchronous (deferrable operator and
+ triggerer) REST API and token paths use ``aiohttp`` with a session that does **not** trust the
+ environment, so proxy environment variables are ignored there and the ``proxies`` extra is required to
+ proxy them. Use the extra when you need a proxy on the asynchronous paths, when you want to force a
+ specific proxy regardless of the worker environment, or when only some endpoints should be proxied.
Review Comment:
Done. I rephrased both occurrences of `proxied` in `providers/databricks/docs/connections/databricks.rst` and verified the file no longer contains that word. I also ran the focused docs checks for the updated file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410088766
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
Review Comment:
Thanks — I made proxy parsing fail-fast and deterministic by validating `extra_dejson` type, `proxies` shape, and values with explicit `DatabricksProxyConfigurationError` messages that include concrete runtime type details for malformed inputs
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Thanks — I updated both sync and async default Azure credential paths to use helper-based class/kwargs construction, and added tests that assert proxy kwargs are passed when configured and filtered safely when unsupported constructor params are encountered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410088766
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
Review Comment:
Thanks — I made proxy parsing fail-fast and deterministic by validating `extra_dejson` type, `proxies` shape, and values with explicit `DatabricksProxyConfigurationError` messages that include concrete runtime type details for malformed inputs
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Thanks — I updated both sync and async default Azure credential paths to use helper-based class/kwargs construction, and added tests that assert proxy kwargs are passed when configured and filtered safely when unsupported constructor params are encountered
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
+ if not isinstance(extra_dejson, dict):
+ return None
+
+ proxies = extra_dejson.get("proxies")
+ if proxies is None:
+ return None
+ if not isinstance(proxies, dict):
+ raise DatabricksProxyConfigurationError("Connection extra 'proxies' must be a JSON object.")
+
+ invalid_keys = set(proxies) - {"http", "https"}
+ if invalid_keys:
+ invalid_keys_str = ", ".join(sorted(invalid_keys))
+ raise DatabricksProxyConfigurationError(
+ f"Connection extra 'proxies' only supports 'http' and 'https' keys. Got: {invalid_keys_str}."
+ )
+
+ for proxy_scheme, proxy_url in proxies.items():
Review Comment:
Thanks — I updated regression coverage so `http` and `https` are both supported: sync requests forward full proxy mapping, and async requests resolve the scheme-specific proxy via URL scheme before passing it to the request call
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
Review Comment:
Thanks — I made proxy parsing fail-fast and deterministic by validating `extra_dejson` type, `proxies` shape, and values with explicit `DatabricksProxyConfigurationError` messages that include concrete runtime type details for malformed inputs.\n\n---\nDrafted-by: Codex (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Thanks — I updated both sync and async default Azure credential paths to use helper-based class/kwargs construction, and added tests that assert proxy kwargs are passed when configured and filtered safely when unsupported constructor params are encountered.\n\n---\nDrafted-by: Codex (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
+ if not isinstance(extra_dejson, dict):
+ return None
+
+ proxies = extra_dejson.get("proxies")
+ if proxies is None:
+ return None
+ if not isinstance(proxies, dict):
+ raise DatabricksProxyConfigurationError("Connection extra 'proxies' must be a JSON object.")
+
+ invalid_keys = set(proxies) - {"http", "https"}
+ if invalid_keys:
+ invalid_keys_str = ", ".join(sorted(invalid_keys))
+ raise DatabricksProxyConfigurationError(
+ f"Connection extra 'proxies' only supports 'http' and 'https' keys. Got: {invalid_keys_str}."
+ )
+
+ for proxy_scheme, proxy_url in proxies.items():
Review Comment:
Thanks — I updated regression coverage so `http` and `https` are both supported: sync requests forward full proxy mapping, and async requests resolve the scheme-specific proxy via URL scheme before passing it to the request call.\n\n---\nDrafted-by: Codex (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #68527:
URL: https://github.com/apache/airflow/pull/68527#issuecomment-4762380063
cc @moomindani @mwojtyczka for review from Databricks team
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3479479518
##########
providers/databricks/docs/connections/databricks.rst:
##########
@@ -81,6 +81,36 @@ Extra (optional)
* ``token``: Specify PAT to use. Consider to switch to specification of PAT in the Password field as it's more secure.
+ The following optional parameter can be used when Airflow workers need to access Databricks or Azure
+ token endpoints through an HTTP proxy:
+
+ * ``proxies``: JSON object with optional ``http`` and ``https`` keys, using the same shape as the
+ ``requests`` and Azure SDK ``proxies`` argument. Only these two keys are accepted. The configured proxy
+ is applied to Databricks REST API calls, Databricks OAuth token exchanges, and Azure Identity token
+ acquisition for AAD and default Azure credential authentication.
+
+ .. code-block:: json
+
+ {
+ "proxies": {
+ "http": "http://proxy.example.com:8080",
+ "https": "http://proxy.example.com:8443"
+ }
+ }
+
+ **Note:** The ``proxies`` extra is only needed for paths that do not already pick up the standard
+ ``HTTP_PROXY`` / ``HTTPS_PROXY`` / ``NO_PROXY`` environment variables. Synchronous REST API and token
+ requests use ``requests``, and Azure AAD / default-credential token acquisition uses the Azure Identity
+ SDK; both honor those environment variables by default. The asynchronous (deferrable operator and
+ triggerer) REST API and token paths use ``aiohttp`` with a session that does **not** trust the
+ environment, so proxy environment variables are ignored there and the ``proxies`` extra is required to
+ proxy them. Use the extra when you need a proxy on the asynchronous paths, when you want to force a
+ specific proxy regardless of the worker environment, or when only some endpoints should be proxied.
Review Comment:
Thanks elad, I will do this in couple of days. Will let you know once I do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410097199
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
+ if not isinstance(extra_dejson, dict):
+ return None
+
+ proxies = extra_dejson.get("proxies")
+ if proxies is None:
+ return None
+ if not isinstance(proxies, dict):
+ raise DatabricksProxyConfigurationError("Connection extra 'proxies' must be a JSON object.")
+
+ invalid_keys = set(proxies) - {"http", "https"}
+ if invalid_keys:
+ invalid_keys_str = ", ".join(sorted(invalid_keys))
+ raise DatabricksProxyConfigurationError(
+ f"Connection extra 'proxies' only supports 'http' and 'https' keys. Got: {invalid_keys_str}."
+ )
+
+ for proxy_scheme, proxy_url in proxies.items():
Review Comment:
http and https scheme handling is now explicitly covered by regression assertions: sync proxy passthrough remains preserved, and async requests now resolve and use the matching proxy from the URL scheme path while continuing to ignore unrelated keys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410096403
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
I wired default Azure credential flows to use the new helper-based class selection and a shared kwargs builder so both sync/async constructors get compatibility-safe args. I also added/kept tests to assert proxy kwargs are included when provided and filtered correctly when unsupported constructor params are present
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410095684
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
Review Comment:
Thanks, I made the proxy parsing fail-fast and deterministic by validating extra_dejsontype,proxiesshape, and values with explicitDatabricksProxyConfigurationError that now includes concrete runtime type details. This keeps error messages clear when connection extras are malformed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410088766
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
Review Comment:
Thanks — I made proxy parsing fail-fast and deterministic by validating `extra_dejson` type, `proxies` shape, and values with explicit `DatabricksProxyConfigurationError` messages that include concrete runtime type details for malformed inputs.\n\n---\nDrafted-by: Claude Code (Opus 4.7) (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Thanks — I updated both sync and async default Azure credential paths to use helper-based class/kwargs construction, and added tests that assert proxy kwargs are passed when configured and filtered safely when unsupported constructor params are encountered.\n\n---\nDrafted-by: Claude Code (Opus 4.7) (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Thanks — I updated regression coverage so `http` and `https` are both supported: sync requests forward full proxy mapping, and async requests resolve the scheme-specific proxy via URL scheme before passing it to the request call.\n\n---\nDrafted-by: Claude Code (Opus 4.7) (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
+ if not isinstance(extra_dejson, dict):
+ return None
+
+ proxies = extra_dejson.get("proxies")
+ if proxies is None:
+ return None
+ if not isinstance(proxies, dict):
+ raise DatabricksProxyConfigurationError("Connection extra 'proxies' must be a JSON object.")
+
+ invalid_keys = set(proxies) - {"http", "https"}
+ if invalid_keys:
+ invalid_keys_str = ", ".join(sorted(invalid_keys))
+ raise DatabricksProxyConfigurationError(
+ f"Connection extra 'proxies' only supports 'http' and 'https' keys. Got: {invalid_keys_str}."
+ )
+
+ for proxy_scheme, proxy_url in proxies.items():
Review Comment:
Thanks — I updated regression coverage so `http` and `https` are both supported: sync requests forward full proxy mapping, and async requests resolve the scheme-specific proxy via URL scheme before passing it to the request call.\n\n---\nDrafted-by: Claude Code (Opus 4.7) (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Thanks — I updated regression coverage so `http` and `https` are both supported: sync requests forward full proxy mapping, and async requests resolve the scheme-specific proxy via URL scheme before passing it to the request call.\n\n---\nDrafted-by: Claude Code (Opus 4.7) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #68527:
URL: https://github.com/apache/airflow/pull/68527#issuecomment-4762868002
Thanks @jroachgolf84 for the approval, @eladkal Can i get a stamp please? Thanks for the time!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #68527:
URL: https://github.com/apache/airflow/pull/68527#issuecomment-4795242862
Thanks @moomindani for the careful review — both points addressed in the latest commit (58b7c4a).
1) Docs: added a Note under the `proxies` extra explaining the relationship with HTTP_PROXY / HTTPS_PROXY / NO_PROXY. One correction worth surfacing from tracing the code: the paths that do *not* read the proxy env vars are the async aiohttp paths, not the Azure Identity ones. The hook builds aiohttp.ClientSession() without trust_env=True, so the async REST and token-exchange paths ignore HTTP_PROXY/HTTPS_PROXY entirely — the `proxies` extra is the only way to proxy them. The sync requests paths and the Azure Identity SDK paths (azure-core's transport defaults to use_env_settings=True) both honor the env vars by default, and the `proxies` extra takes precedence over them when both are set. The Note documents this per client.
2) Comment: added a comment at both ManagedIdentityCredential call sites (sync _get_aad_token and async _a_get_aad_token) explaining the exclusion — managed identity authenticates against the link-local IMDS endpoint (169.254.169.254), which must be reached directly and is unsupported behind a proxy, so the `proxies` extra is intentionally not forwarded there, unlike the ClientSecretCredential / DefaultAzureCredential paths that reach the public Entra ID endpoint. (The IMDS metadata-service call is likewise left unproxied.)
Also added a regression test asserting ManagedIdentityCredential and the IMDS metadata call are constructed without `proxies`, while the Databricks REST call still receives it — so the asymmetry can't silently regress.
No changelog/newsfragment: providers don't consume newsfragments, and this is a docs + comment clarification with a test, no behavior change. Re-ran the databricks hook tests (285 passed).
---
Drafted-by: Claude Code (Opus 4.8); reviewed by @Vamsi-klu before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3479402821
##########
providers/databricks/docs/connections/databricks.rst:
##########
@@ -81,6 +81,36 @@ Extra (optional)
* ``token``: Specify PAT to use. Consider to switch to specification of PAT in the Password field as it's more secure.
+ The following optional parameter can be used when Airflow workers need to access Databricks or Azure
+ token endpoints through an HTTP proxy:
+
+ * ``proxies``: JSON object with optional ``http`` and ``https`` keys, using the same shape as the
+ ``requests`` and Azure SDK ``proxies`` argument. Only these two keys are accepted. The configured proxy
+ is applied to Databricks REST API calls, Databricks OAuth token exchanges, and Azure Identity token
+ acquisition for AAD and default Azure credential authentication.
+
+ .. code-block:: json
+
+ {
+ "proxies": {
+ "http": "http://proxy.example.com:8080",
+ "https": "http://proxy.example.com:8443"
+ }
+ }
+
+ **Note:** The ``proxies`` extra is only needed for paths that do not already pick up the standard
+ ``HTTP_PROXY`` / ``HTTPS_PROXY`` / ``NO_PROXY`` environment variables. Synchronous REST API and token
+ requests use ``requests``, and Azure AAD / default-credential token acquisition uses the Azure Identity
+ SDK; both honor those environment variables by default. The asynchronous (deferrable operator and
+ triggerer) REST API and token paths use ``aiohttp`` with a session that does **not** trust the
+ environment, so proxy environment variables are ignored there and the ``proxies`` extra is required to
+ proxy them. Use the extra when you need a proxy on the asynchronous paths, when you want to force a
+ specific proxy regardless of the worker environment, or when only some endpoints should be proxied.
Review Comment:
Please change wording. proxied is not recognized by doc build. Rephrase and replace all occurances of the word in the file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3482669330
##########
providers/databricks/docs/connections/databricks.rst:
##########
@@ -81,6 +81,36 @@ Extra (optional)
* ``token``: Specify PAT to use. Consider to switch to specification of PAT in the Password field as it's more secure.
+ The following optional parameter can be used when Airflow workers need to access Databricks or Azure
+ token endpoints through an HTTP proxy:
+
+ * ``proxies``: JSON object with optional ``http`` and ``https`` keys, using the same shape as the
+ ``requests`` and Azure SDK ``proxies`` argument. Only these two keys are accepted. The configured proxy
+ is applied to Databricks REST API calls, Databricks OAuth token exchanges, and Azure Identity token
+ acquisition for AAD and default Azure credential authentication.
+
+ .. code-block:: json
+
+ {
+ "proxies": {
+ "http": "http://proxy.example.com:8080",
+ "https": "http://proxy.example.com:8443"
+ }
+ }
+
+ **Note:** The ``proxies`` extra is only needed for paths that do not already pick up the standard
+ ``HTTP_PROXY`` / ``HTTPS_PROXY`` / ``NO_PROXY`` environment variables. Synchronous REST API and token
+ requests use ``requests``, and Azure AAD / default-credential token acquisition uses the Azure Identity
+ SDK; both honor those environment variables by default. The asynchronous (deferrable operator and
+ triggerer) REST API and token paths use ``aiohttp`` with a session that does **not** trust the
+ environment, so proxy environment variables are ignored there and the ``proxies`` extra is required to
+ proxy them. Use the extra when you need a proxy on the asynchronous paths, when you want to force a
+ specific proxy regardless of the worker environment, or when only some endpoints should be proxied.
Review Comment:
Thanks for pointing this out. I updated the Databricks connection docs to avoid the word `proxied`, since it is not accepted by the docs build/spell check.
I rephrased both occurrences in `providers/databricks/docs/connections/databricks.rst` while keeping the intended meaning the same:
- selected Databricks/Azure endpoints can still use the configured proxy
- the Azure managed-identity path still intentionally bypasses the configured proxy because it talks to the link-local IMDS endpoint
I also verified that `proxied` no longer appears in the file and ran the focused docs checks for the updated file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3479555996
##########
providers/databricks/docs/connections/databricks.rst:
##########
@@ -81,6 +81,36 @@ Extra (optional)
* ``token``: Specify PAT to use. Consider to switch to specification of PAT in the Password field as it's more secure.
+ The following optional parameter can be used when Airflow workers need to access Databricks or Azure
+ token endpoints through an HTTP proxy:
+
+ * ``proxies``: JSON object with optional ``http`` and ``https`` keys, using the same shape as the
+ ``requests`` and Azure SDK ``proxies`` argument. Only these two keys are accepted. The configured proxy
+ is applied to Databricks REST API calls, Databricks OAuth token exchanges, and Azure Identity token
+ acquisition for AAD and default Azure credential authentication.
+
+ .. code-block:: json
+
+ {
+ "proxies": {
+ "http": "http://proxy.example.com:8080",
+ "https": "http://proxy.example.com:8443"
+ }
+ }
+
+ **Note:** The ``proxies`` extra is only needed for paths that do not already pick up the standard
+ ``HTTP_PROXY`` / ``HTTPS_PROXY`` / ``NO_PROXY`` environment variables. Synchronous REST API and token
+ requests use ``requests``, and Azure AAD / default-credential token acquisition uses the Azure Identity
+ SDK; both honor those environment variables by default. The asynchronous (deferrable operator and
+ triggerer) REST API and token paths use ``aiohttp`` with a session that does **not** trust the
+ environment, so proxy environment variables are ignored there and the ``proxies`` extra is required to
+ proxy them. Use the extra when you need a proxy on the asynchronous paths, when you want to force a
+ specific proxy regardless of the worker environment, or when only some endpoints should be proxied.
Review Comment:
Done. I rephrased both occurrences of `proxied` in `providers/databricks/docs/connections/databricks.rst` and verified the file no longer contains that word. I also ran the focused docs checks for the updated file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on PR #68527:
URL: https://github.com/apache/airflow/pull/68527#issuecomment-4762584640
## Databricks proxy support validation for PR #68527
Validated against exact PR head `124d736716ec26bc58bdcd71d57335b6b5998893` on June 21, 2026.
### Test suite
Focused Databricks provider tests passed:
```text
284 passed, 1 warning in 7.85s
```
Command:
```bash
PYTHONPATH=/Users/nalam/Downloads/airflow-68527-vr/airflow-core/src:/Users/nalam/Downloads/airflow-68527-vr/providers/databricks/src \
/Users/nalam/airflow-1/.venv/bin/python -m pytest \
providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py \
providers/databricks/tests/unit/databricks/hooks/test_databricks.py \
providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py -q
```
### Live Databricks validation
Created a one-day personal access token scoped to `clusters`, used it only for this validation, and revoked it after the run. The access token page returned to `No tokens exist.` after cleanup.
The live validation used Airflow's `DatabricksHook` from the PR worktree with connection extra:
```json
{
"proxies": {
"http": "http://127.0.0.1:<port>",
"https": "http://127.0.0.1:<port>"
}
}
```
The validation called `GET /api/2.0/clusters/list` against the Databricks workspace through a local HTTP CONNECT proxy.
Sync path result:
```text
sync_api_result.ok=true
sync_api_result.cluster_count=0
sync_proxy_tunnel_to_workspace_observed=true
CONNECT dbc-c1a3fd5e-8c66.cloud.databricks.com:443 HTTP/1.1
```
Async path result:
```text
async_api_result.ok=true
async_api_result.cluster_count=0
async_proxy_tunnel_to_workspace_observed=true
CONNECT dbc-c1a3fd5e-8c66.cloud.databricks.com:443 HTTP/1.1
```
This covers the PR's sync `requests` forwarding path and async `aiohttp` forwarding path with an authenticated Databricks API call.
### Evidence
Evidence bundle: https://gist.github.com/Vamsi-klu/fd442d24aec03cd0bae7c6bfc0997e22
Validation JSON: https://gist.githubusercontent.com/Vamsi-klu/fd442d24aec03cd0bae7c6bfc0997e22/raw/70dc93e752813464937281658b7edc34d2c942b1/databricks_proxy_live_validation.json
Repro script: https://gist.githubusercontent.com/Vamsi-klu/fd442d24aec03cd0bae7c6bfc0997e22/raw/70dc93e752813464937281658b7edc34d2c942b1/databricks_proxy_live_validation.py
Workspace availability:
Workspace page:
Post-cleanup token state:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410088863
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
+ if not isinstance(extra_dejson, dict):
+ return None
+
+ proxies = extra_dejson.get("proxies")
+ if proxies is None:
+ return None
+ if not isinstance(proxies, dict):
+ raise DatabricksProxyConfigurationError("Connection extra 'proxies' must be a JSON object.")
+
+ invalid_keys = set(proxies) - {"http", "https"}
+ if invalid_keys:
+ invalid_keys_str = ", ".join(sorted(invalid_keys))
+ raise DatabricksProxyConfigurationError(
+ f"Connection extra 'proxies' only supports 'http' and 'https' keys. Got: {invalid_keys_str}."
+ )
+
+ for proxy_scheme, proxy_url in proxies.items():
Review Comment:
Thanks — I updated regression coverage so `http` and `https` are both supported: sync requests forward full proxy mapping, and async requests resolve the scheme-specific proxy via URL scheme before passing it to the request call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add proxy support to Databricks connections [airflow]
Posted by "Vamsi-klu (via GitHub)" <gi...@apache.org>.
Vamsi-klu commented on code in PR #68527:
URL: https://github.com/apache/airflow/pull/68527#discussion_r3410092440
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
Review Comment:
Thanks — I made proxy parsing fail-fast and deterministic by validating `extra_dejson` type, `proxies` shape, and values with explicit `DatabricksProxyConfigurationError` messages that include concrete runtime type details for malformed inputs.\n\n---\nDrafted-by: Codex (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -445,7 +495,9 @@ def _get_aad_token_for_default_az_credential(self, resource: str) -> str:
#
# While there is a WorkloadIdentityCredential class, the below class is advised by Microsoft
# https://learn.microsoft.com/en-us/azure/aks/workload-identity-overview
- token = DefaultAzureCredential().get_token(f"{resource}/.default")
+ token = DefaultAzureCredential(**self._get_azure_credential_kwargs()).get_token(
Review Comment:
Thanks — I updated both sync and async default Azure credential paths to use helper-based class/kwargs construction, and added tests that assert proxy kwargs are passed when configured and filtered safely when unsupported constructor params are encountered.\n\n---\nDrafted-by: Codex (no human review before posting)
##########
providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py:
##########
@@ -233,6 +238,47 @@ def _get_connection_attr(self, attr_name: str) -> str:
raise ValueError(f"`{attr_name}` must be present in Connection")
return attr
+ @cached_property
+ def proxies(self) -> dict[str, str] | None:
+ """Return validated proxy configuration from connection extras."""
+ extra_dejson = self.databricks_conn.extra_dejson
+ if not isinstance(extra_dejson, dict):
+ return None
+
+ proxies = extra_dejson.get("proxies")
+ if proxies is None:
+ return None
+ if not isinstance(proxies, dict):
+ raise DatabricksProxyConfigurationError("Connection extra 'proxies' must be a JSON object.")
+
+ invalid_keys = set(proxies) - {"http", "https"}
+ if invalid_keys:
+ invalid_keys_str = ", ".join(sorted(invalid_keys))
+ raise DatabricksProxyConfigurationError(
+ f"Connection extra 'proxies' only supports 'http' and 'https' keys. Got: {invalid_keys_str}."
+ )
+
+ for proxy_scheme, proxy_url in proxies.items():
Review Comment:
Thanks — I updated regression coverage so `http` and `https` are both supported: sync requests forward full proxy mapping, and async requests resolve the scheme-specific proxy via URL scheme before passing it to the request call.\n\n---\nDrafted-by: Codex (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org