You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/08/27 18:00:54 UTC

[PR] Drop connection-supplied HTTP headers on cross-host redirects [airflow]

potiuk opened a new pull request, #72165:
URL: https://github.com/apache/airflow/pull/72165

   ## Why
   
   `HttpHook.get_conn` applies every unrecognised key from a connection's `extra` field as a persistent session header:
   
   ```python
   session.headers.update(conn_extra_options)   # _configure_session_from_extra
   ```
   
   and the hook's own docstring documents that field as the place to put them — *"headers can also be specified in the Extra field in json format"*.
   
   `requests` removes only the `Authorization` header when a redirect crosses to a different host (`Session.rebuild_auth` → `should_strip_auth`). A credential carried under any **other** header name — `X-API-Key`, `X-Auth-Token` and similar, which is exactly what the documented `extra`-field pattern produces — stays on the session and is replayed verbatim to whatever host the redirect points at. Redirects are followed by default (`session.max_redirects = DEFAULT_REDIRECT_LIMIT`).
   
   ## What
   
   - Add `_ConnectionHeaderSession`, a `requests.Session` subclass that records which header names came from the connection and drops **only those** when `should_strip_auth()` reports the redirect is cross-origin.
   - Use it in `get_conn`, and populate `connection_header_keys` where the extra-field headers are applied.
   - Tests covering cross-host stripping, same-host preservation, caller-supplied headers being left alone, and the keys being recorded through `get_conn`.
   
   Delegating the decision to `requests`' own `should_strip_auth()` means an `https` → `http` downgrade on the same host is covered too, without duplicating that logic.
   
   ## Compatibility
   
   - Headers passed explicitly by the caller (`get_conn(headers=...)`, `default_headers`) are **not** affected — the caller controls the request either way.
   - Same-host redirects are unchanged.
   - Only connection-`extra`-derived headers are dropped, and only on a cross-origin hop.
   
   ## Testing
   
   Verified locally against real `requests`, all five cases passing: cross-host connection header dropped, same-host preserved, caller-supplied header preserved, `Authorization` still stripped by `requests` itself, and same-host `https`→`http` downgrade stripped.
   
   **The provider test suite could not be executed in my environment** — a local editable-install issue unrelated to this change prevented `airflow` from importing. `ruff check` and `ruff format --check` are clean, and `py_compile` passes on both files, but **CI needs to run `providers/http` tests.**
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Drop connection-supplied HTTP headers on cross-host redirects [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #72165:
URL: https://github.com/apache/airflow/pull/72165#discussion_r3887275398


##########
providers/http/tests/unit/http/hooks/test_http.py:
##########
@@ -898,3 +903,61 @@ async def test_build_request_url_from_endpoint_param(self):
             async with aiohttp.ClientSession() as session:
                 await hook.run(session=session, endpoint="test.com:8080/v1/test")
                 assert mocked_function.call_args.args[0] == "http://test.com:8080/v1/test"
+
+
+class TestConnectionHeadersAcrossRedirects:
+    """Connection-supplied headers must not follow a redirect to a different host.
+
+    ``requests`` strips only ``Authorization`` (``Session.rebuild_auth``), so a credential
+    carried in a differently-named header — the documented ``extra``-field pattern — would
+    otherwise be replayed to whatever host the redirect points at.
+    """
+
+    @staticmethod
+    def _redirect(session, old_url, new_url):
+        original = requests.Request("GET", old_url).prepare()
+        response = Response()
+        response.request = original
+        prepared = requests.Request("GET", new_url).prepare()
+        prepared.headers.update(session.headers)
+        session.rebuild_auth(prepared, response)
+        return prepared.headers
+
+    def test_connection_headers_dropped_on_cross_host_redirect(self):
+        session = _ConnectionHeaderSession()
+        session.headers.update({"X-API-Key": "secret", "Accept": "application/json"})
+        session.connection_header_keys.update({"X-API-Key", "Accept"})
+
+        headers = self._redirect(session, "https://original.example.com/a", "https://evil.example.com/b")
+
+        assert "X-API-Key" not in headers
+        assert "Accept" not in headers
+
+    def test_connection_headers_kept_on_same_host_redirect(self):
+        session = _ConnectionHeaderSession()
+        session.headers.update({"X-API-Key": "secret"})
+        session.connection_header_keys.update({"X-API-Key"})
+
+        headers = self._redirect(session, "https://example.com/a", "https://example.com/b")
+
+        assert headers["X-API-Key"] == "secret"
+
+    def test_caller_supplied_headers_are_not_dropped(self):
+        """Only headers that came from the connection are stripped."""
+        session = _ConnectionHeaderSession()
+        session.headers.update({"X-Caller": "kept"})
+
+        headers = self._redirect(session, "https://original.example.com/a", "https://evil.example.com/b")
+
+        assert headers["X-Caller"] == "kept"
+
+    @mock.patch(
+        "airflow.hooks.base.BaseHook.get_connection",

Review Comment:
   Fixed already.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Drop connection-supplied HTTP headers on cross-host redirects [airflow]

Posted by "SameerMesiah97 (via GitHub)" <gi...@apache.org>.
SameerMesiah97 commented on code in PR #72165:
URL: https://github.com/apache/airflow/pull/72165#discussion_r3874976233


##########
providers/http/tests/unit/http/hooks/test_http.py:
##########
@@ -898,3 +903,61 @@ async def test_build_request_url_from_endpoint_param(self):
             async with aiohttp.ClientSession() as session:
                 await hook.run(session=session, endpoint="test.com:8080/v1/test")
                 assert mocked_function.call_args.args[0] == "http://test.com:8080/v1/test"
+
+
+class TestConnectionHeadersAcrossRedirects:
+    """Connection-supplied headers must not follow a redirect to a different host.
+
+    ``requests`` strips only ``Authorization`` (``Session.rebuild_auth``), so a credential
+    carried in a differently-named header — the documented ``extra``-field pattern — would
+    otherwise be replayed to whatever host the redirect points at.
+    """
+
+    @staticmethod
+    def _redirect(session, old_url, new_url):
+        original = requests.Request("GET", old_url).prepare()
+        response = Response()
+        response.request = original
+        prepared = requests.Request("GET", new_url).prepare()
+        prepared.headers.update(session.headers)
+        session.rebuild_auth(prepared, response)
+        return prepared.headers
+
+    def test_connection_headers_dropped_on_cross_host_redirect(self):
+        session = _ConnectionHeaderSession()
+        session.headers.update({"X-API-Key": "secret", "Accept": "application/json"})
+        session.connection_header_keys.update({"X-API-Key", "Accept"})
+
+        headers = self._redirect(session, "https://original.example.com/a", "https://evil.example.com/b")
+
+        assert "X-API-Key" not in headers
+        assert "Accept" not in headers
+
+    def test_connection_headers_kept_on_same_host_redirect(self):
+        session = _ConnectionHeaderSession()
+        session.headers.update({"X-API-Key": "secret"})
+        session.connection_header_keys.update({"X-API-Key"})
+
+        headers = self._redirect(session, "https://example.com/a", "https://example.com/b")
+
+        assert headers["X-API-Key"] == "secret"
+
+    def test_caller_supplied_headers_are_not_dropped(self):
+        """Only headers that came from the connection are stripped."""
+        session = _ConnectionHeaderSession()
+        session.headers.update({"X-Caller": "kept"})
+
+        headers = self._redirect(session, "https://original.example.com/a", "https://evil.example.com/b")
+
+        assert headers["X-Caller"] == "kept"
+
+    @mock.patch(
+        "airflow.hooks.base.BaseHook.get_connection",

Review Comment:
   This patch could be causing one of the failing tests. I would try the below:
   
   `"airflow.providers.http.hooks.http.HttpHook.get_connection"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Drop connection-supplied HTTP headers on cross-host redirects [airflow]

Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #72165:
URL: https://github.com/apache/airflow/pull/72165#discussion_r4307198291


##########
providers/http/src/airflow/providers/http/hooks/http.py:
##########
@@ -169,6 +169,35 @@ def _retryable_error_async(exception: BaseException) -> bool:
     return exception.status >= 500
 
 
+class _ConnectionHeaderSession(Session):

Review Comment:
   **[warning]** The behaviour change for `HttpHook` is not documented in this PR.
   
   The changelog and connection docs were asked for on #70000 (@aaron-y-chen in their review, and @Samin061 when closing it: "point the connection at the final host, or set `allow_redirects=False` in `extra_options` and make the second request yourself"). #72398 now adds a `.. warning::` to `changelog.rst` and a paragraph to `connections/http.rst`, but it describes the async hook ("follow aiohttp's origin rules"), and nothing in this PR covers the sync side. If the two PRs merge in separate provider waves, or this one first, `HttpHook` users get the change with no note.
   
   The note should also say that **every** connection header is dropped, not only credentials: `Accept` is dropped in `test_connection_headers_dropped_on_cross_host_redirect`, and so would be a `Content-Type` from the extra on a 307/308 that resends the body to the other host. Someone who relied on a CDN hop getting the same `Accept` will see that first.
   
   Suggestion: one hook-neutral note (both hooks, the origin rule from `should_strip_auth`, the two ways out quoted above), carried by whichever of #72165 / #72398 merges first, or by both if they go together.



##########
providers/http/tests/unit/http/hooks/test_http.py:
##########
@@ -1385,3 +1386,87 @@ async def test_resolve_srv_targets_async_missing_dependency_raises(self):
         with mock.patch.dict(sys.modules, {"dns": None, "dns.asyncresolver": None, "dns.exception": None}):
             with pytest.raises(HttpSrvLookupException, match="dnspython"):
                 await hook._resolve_srv_targets_async("_http._tcp.example.com")
+
+
+class TestConnectionHeadersAcrossRedirects:
+    """Connection-supplied headers must not follow a redirect to a different host.
+
+    ``requests`` strips only ``Authorization`` (``Session.rebuild_auth``), so a credential
+    carried in a differently-named header — the documented ``extra``-field pattern — would
+    otherwise be replayed to whatever host the redirect points at.
+    """
+
+    @staticmethod
+    def _redirect(session, old_url, new_url):

Review Comment:
   **[nit]** All the redirect tests call `rebuild_auth` directly, so no test goes through a real redirect.
   
   `_redirect` builds a prepared request by hand and calls `session.rebuild_auth`. The tests would still pass if `run()` / `run_and_check()` stopped going through `Session.send` → `resolve_redirects`, or if `requests` changed what it copies to the next hop. #72398 tests the async side end to end with an aiohttp `TestServer`. One test like it for the sync hook would cover the whole path. With `requests_mock`, the real redirect logic of the session still runs:
   
   ```python
   def test_run_drops_connection_header_on_cross_host_redirect(self, requests_mock):
       requests_mock.get("http://test:8080/v1/test", status_code=302, headers={"Location": "https://cdn.example.com/f"})
       requests_mock.get("https://cdn.example.com/f", text="ok")
       with mock.patch(
           "airflow.providers.http.hooks.http.HttpHook.get_connection",
           side_effect=get_airflow_connection_with_extra({"X-API-Key": "secret"}),
       ):
           HttpHook(method="GET").run("v1/test")
   
       assert requests_mock.request_history[0].headers["X-API-Key"] == "secret"
       assert "X-API-Key" not in requests_mock.request_history[-1].headers
   ```
   
   (I ran the equivalent locally against this head: the cross-host hop drops `X-API-Key` and `Accept`, a same-host hop keeps them, a caller header survives, and a port change strips. So this is about keeping it covered, not a bug.)



##########
providers/http/src/airflow/providers/http/hooks/http.py:
##########
@@ -265,10 +294,11 @@ def get_conn(
             session = self._configure_session_from_extra(session, connection, extra_options)
 
         session = self._configure_session_from_mount_adapters(session)
-        if self.default_headers:
-            session.headers.update(self.default_headers)
-        if headers:
-            session.headers.update(headers)
+        for override in (self.default_headers, headers):
+            if override:
+                session.headers.update(override)
+                if isinstance(session, _ConnectionHeaderSession):

Review Comment:
   **[nit]** The hook keeps the session's headers and `connection_header_keys` in sync from outside, behind two `isinstance` checks.
   
   `get_conn` (here) and `_configure_session_from_extra` (line 394) each ask the session for its type and then update two structures that must stay consistent: `session.headers` and `session.connection_header_keys`. If the session took both updates as one call, e.g. `add_connection_headers(headers)` and `add_caller_headers(headers)` on `_ConnectionHeaderSession`, the lower-casing and the add/remove rule would live in one place next to `rebuild_auth`, which reads them. The hook would then call a method instead of checking the type ([Martin, *OCP*](http://butunclebob.com/ArticleS.UncleBob.PrinciplesOfOod)).
   
   The trade-off: the `isinstance` guard also covers a plain `Session` (the sensor tests' `FakeSession`, or a subclass passing its own session to `_configure_session_from_extra`). With methods, `FakeSession` would need to subclass `_ConnectionHeaderSession`. Fine to leave as is if you prefer that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Drop connection-supplied HTTP headers on cross-host redirects [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #72165:
URL: https://github.com/apache/airflow/pull/72165#issuecomment-6227285723

   @vincbeck @dabla Could you take a look at this when you have a moment? CI is green and it's ready from my side, so it can go into the next providers wave.
   
   ---
   Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org