You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vatsrahul1001 (via GitHub)" <gi...@apache.org> on 2026/04/02 17:13:48 UTC

[PR] Fix ObjectStoragePath NoCredentialsError when using conn_id with remote stores [airflow]

vatsrahul1001 opened a new pull request, #64634:
URL: https://github.com/apache/airflow/pull/64634

   
   ## Problem
   
   After the migration from `CloudPath` to `ProxyUPath` (#60519), `ObjectStoragePath`
   silently breaks for any remote store when a `conn_id` is provided.
   
   ```python
   path = ObjectStoragePath("s3://my-bucket/prefix", conn_id="my_aws_conn")
   path.exists()   # NoCredentialsError
   path.mkdir()    # NoCredentialsError
   path.iterdir()  # NoCredentialsError
   ```
   
   Root cause: ProxyUPath delegates all filesystem operations to an inner self.__wrapped__ UPath instance. That instance is constructed with empty storage_options because conn_id is stripped before super().__init__() (per #62701 to prevent it leaking into fsspec). So self.__wrapped__.fs creates an unauthenticated s3fs/gcsfs — ignoring the conn_id entirely.
   
   This affects every delegated method: exists, mkdir, is_dir, is_file, touch, unlink, rmdir, iterdir, glob, rglob, walk, rename, read_bytes, write_bytes, read_text, write_text, and open.
   
   Reported by @pankajkoti in #3.2.0rc1 testing.
   
   **Fix**
   Inject the Airflow-authenticated filesystem directly into self.__wrapped__._fs_cached immediately after super().__init__().
   
   UPath propagates _fs_cached from parent to child whenever new path segments are appended (p / "sub", parent, iterdir results, glob results, etc.), so two injection points cover every case:
   
   __init__ — for paths constructed directly
   _from_upath — for paths wrapped from a fresh UPath with no cached fs
   This fixes all delegated methods in 22 lines with no per-method overrides.
   
   **Testing**
   
   <img width="1392" height="526" alt="image" src="https://github.com/user-attachments/assets/40e23719-08a1-4a4d-bcd3-6f9a4b6b5a41" />
   
   * closes: https://github.com/apache/airflow/issues/64632
   * related: #ISSUE
   
   
   - [X] Yes (please specify the tool below)
   claude
   
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix ObjectStoragePath NoCredentialsError when using conn_id with remote stores [airflow]

Posted by "vatsrahul1001 (via GitHub)" <gi...@apache.org>.
vatsrahul1001 merged PR #64634:
URL: https://github.com/apache/airflow/pull/64634


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix ObjectStoragePath NoCredentialsError when using conn_id with remote stores [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #64634:
URL: https://github.com/apache/airflow/pull/64634#issuecomment-4180481794

   ### Backport successfully created: v3-2-test
   
   Note: As of [Merging PRs targeted for Airflow 3.X](https://github.com/apache/airflow/blob/main/dev/README_AIRFLOW3_DEV.md#merging-prs-targeted-for-airflow-3x)
   the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
   
   In matter of doubt please ask in [#release-management](https://apache-airflow.slack.com/archives/C03G9H97MM2) Slack channel.
   
   <table>
           <tr>
           <th>Status</th>
           <th>Branch</th>
           <th>Result</th>
       </tr>
       <tr>
           <td>✅</td>
           <td>v3-2-test</td>
           <td><a href="https://github.com/apache/airflow/pull/64646"><img src="https://img.shields.io/badge/PR-64646-blue" alt="PR Link"></a></td>
       </tr>
   </table>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix ObjectStoragePath NoCredentialsError when using conn_id with remote stores [airflow]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #64634:
URL: https://github.com/apache/airflow/pull/64634#discussion_r3029955183


##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -127,6 +138,15 @@ def _from_upath(cls_or_self, upath, /):
         obj = object.__new__(cls)
         obj.__wrapped__ = upath
         obj._conn_id = getattr(cls_or_self, "_conn_id", None) if is_instance else None
+        # If the wrapped UPath has not yet had its fs cached (e.g. when _from_upath is
+        # called as a classmethod with a fresh UPath), inject the authenticated fs now.
+        # Child UPaths produced by __wrapped__ operations (iterdir, glob, etc.) already
+        # inherit _fs_cached from the parent UPath, so the hasattr check is a no-op for them.
+        if obj._conn_id is not None and not hasattr(upath, "_fs_cached"):
+            try:
+                obj.__wrapped__._fs_cached = attach(upath.protocol or "file", obj._conn_id).fs
+            except Exception:
+                pass  # Provider not available or scheme unregistered; will fail naturally when the path is used

Review Comment:
   Similarly, `_from_upath` suppresses all exceptions when attempting to inject the authenticated FS. This can hide real resolution errors and leave the wrapped UPath using a different/unauthenticated filesystem than `ObjectStoragePath.fs`. Prefer using `obj.fs` for injection and only swallowing the specific exceptions that correspond to “optional provider missing / scheme not registered”, or re-raising unexpected ones.
   ```suggestion
                   obj.__wrapped__._fs_cached = obj.fs
               except (ImportError, ModuleNotFoundError, ValueError):
                   pass  # Optional provider unavailable or scheme unregistered; will fail naturally when the path is used
   ```



##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -116,6 +116,17 @@ def __init__(
         # to the underlying fsspec filesystem, which doesn't understand it
         self._conn_id = storage_options.pop("conn_id", None)
         super().__init__(*args, protocol=protocol, **storage_options)
+        # ProxyUPath delegates all operations to self.__wrapped__, which was
+        # constructed with empty storage_options (conn_id stripped above).
+        # Pre-populating __wrapped__._fs_cached with the Airflow-authenticated
+        # filesystem fixes every delegated method (exists, mkdir, iterdir, glob,
+        # walk, rename, read_bytes, write_bytes, …) in one place rather than
+        # requiring individual overrides for each one.
+        if self._conn_id is not None:
+            try:
+                self.__wrapped__._fs_cached = attach(self.protocol or "file", self._conn_id).fs
+            except Exception:
+                pass  # Provider not available or scheme unregistered; will fail naturally when the path is used

Review Comment:
   The broad `except Exception: pass` here will silently skip filesystem injection even for unexpected errors (e.g. invalid conn_id, provider bugs), which can cause delegated methods to fall back to an unauthenticated fsspec FS and fail later with confusing errors. Consider setting `__wrapped__._fs_cached` via `self.fs` (so failures are consistent) and either let exceptions propagate, or only suppress specific “provider not available / scheme unregistered” exceptions (and ideally log at debug when skipping).



##########
task-sdk/tests/task_sdk/io/test_path.py:
##########
@@ -228,6 +228,116 @@ def test_standard_extended_api(self, fake_files, fn, args, fn2, path, expected_a
             method.assert_called_once_with(expected_args, **expected_kwargs)
 
 
+class TestConnIdCredentialResolution:
+    """
+    Regression tests for https://github.com/apache/airflow/issues/64524
+
+    When ObjectStoragePath was migrated from CloudPath to ProxyUPath (3.2.0),
+    methods like exists(), mkdir(), is_dir(), is_file() were delegated to
+    self.__wrapped__ which carries empty storage_options (conn_id is stored
+    separately). This caused NoCredentialsError / 401 errors for remote stores
+    even when a valid conn_id was provided.
+    """
+
+    @pytest.fixture(autouse=True)
+    def restore_cache(self):
+        cache = _STORE_CACHE.copy()
+        yield
+        _STORE_CACHE.clear()
+        _STORE_CACHE.update(cache)
+
+    @pytest.fixture
+    def fake_fs_with_conn(self):
+        fs = _FakeRemoteFileSystem(conn_id="my_conn")
+        attach(protocol="ffs2", conn_id="my_conn", fs=fs)
+        try:
+            yield fs
+        finally:
+            _FakeRemoteFileSystem.store.clear()
+            _FakeRemoteFileSystem.pseudo_dirs[:] = [""]
+
+    def test_exists_uses_authenticated_fs(self, fake_fs_with_conn):
+        """exists() must use self.fs (Airflow-attached) not __wrapped__.fs (unauthenticated)."""
+        p = ObjectStoragePath("ffs2://my_conn@bucket/some_file.txt", conn_id="my_conn")
+        # Verify the correct fs instance was injected, not merely any _FakeRemoteFileSystem
+        assert p.__wrapped__._fs_cached is fake_fs_with_conn
+        fake_fs_with_conn.touch("bucket/some_file.txt")
+
+        assert p.exists() is True
+        assert (
+            ObjectStoragePath("ffs2://my_conn@bucket/no_such_file.txt", conn_id="my_conn").exists() is False
+        )
+
+    def test_mkdir_uses_authenticated_fs(self, fake_fs_with_conn):
+        """mkdir() must use self.fs (Airflow-attached) not __wrapped__.fs (unauthenticated)."""
+        p = ObjectStoragePath("ffs2://my_conn@bucket/new_dir/", conn_id="my_conn")
+        p.mkdir(parents=True, exist_ok=True)
+        assert fake_fs_with_conn.isdir("bucket/new_dir")
+
+    def test_is_dir_uses_authenticated_fs(self, fake_fs_with_conn):
+        """is_dir() must use self.fs (Airflow-attached) not __wrapped__.fs (unauthenticated)."""
+        fake_fs_with_conn.mkdir("bucket/a_dir")
+        p = ObjectStoragePath("ffs2://my_conn@bucket/a_dir", conn_id="my_conn")
+        assert p.is_dir() is True
+
+    def test_is_file_uses_authenticated_fs(self, fake_fs_with_conn):
+        """is_file() must use self.fs (Airflow-attached) not __wrapped__.fs (unauthenticated)."""
+        fake_fs_with_conn.touch("bucket/a_file.txt")
+        p = ObjectStoragePath("ffs2://my_conn@bucket/a_file.txt", conn_id="my_conn")
+        assert p.is_file() is True
+
+    def test_touch_uses_authenticated_fs(self, fake_fs_with_conn):
+        """touch() must use self.fs (Airflow-attached) not __wrapped__.fs (unauthenticated)."""
+        p = ObjectStoragePath("ffs2://my_conn@bucket/touched_file.txt", conn_id="my_conn")
+        p.touch()
+        assert fake_fs_with_conn.exists("bucket/touched_file.txt")
+
+    def test_unlink_uses_authenticated_fs(self, fake_fs_with_conn):
+        """unlink() must use self.fs (Airflow-attached) not __wrapped__.fs (unauthenticated)."""
+        fake_fs_with_conn.touch("bucket/to_delete.txt")
+        p = ObjectStoragePath("ffs2://my_conn@bucket/to_delete.txt", conn_id="my_conn")
+        p.unlink()
+        assert not fake_fs_with_conn.exists("bucket/to_delete.txt")
+
+    def test_rmdir_uses_authenticated_fs(self, fake_fs_with_conn):
+        """rmdir() must use self.fs (Airflow-attached) not __wrapped__.fs (unauthenticated)."""
+        fake_fs_with_conn.mkdir("bucket/empty_dir")
+        p = ObjectStoragePath("ffs2://my_conn@bucket/empty_dir", conn_id="my_conn")
+        # upath's rmdir(recursive=False) calls next(self.iterdir()) without a default,
+        # which raises StopIteration on empty dirs — a upath bug. Use the default (recursive=True).
+        p.rmdir()
+        assert not fake_fs_with_conn.exists("bucket/empty_dir")
+
+    def test_conn_id_in_uri_works_for_exists(self, fake_fs_with_conn):
+        """conn_id embedded in URI (user@host) should also work for exists()."""
+        fake_fs_with_conn.touch("bucket/target.txt")
+        p = ObjectStoragePath("ffs2://my_conn@bucket/target.txt")
+        assert p.conn_id == "my_conn"
+        assert p.exists() is True
+
+    def test_from_upath_injects_fs_when_no_cache(self, fake_fs_with_conn):
+        """_from_upath must inject authenticated fs into a fresh UPath with no _fs_cached."""
+        from upath import UPath
+

Review Comment:
   There’s a new local import (`from upath import UPath`) inside the test method. Since `UPath` is already a hard dependency for this module (and other tests use it indirectly), move the import to the top of the file to keep import behavior consistent and avoid surprises with tooling/static analysis.



##########
task-sdk/tests/task_sdk/io/test_path.py:
##########
@@ -228,6 +228,116 @@ def test_standard_extended_api(self, fake_files, fn, args, fn2, path, expected_a
             method.assert_called_once_with(expected_args, **expected_kwargs)
 
 
+class TestConnIdCredentialResolution:
+    """
+    Regression tests for https://github.com/apache/airflow/issues/64524

Review Comment:
   The regression-test docstring links to issue #64524, but the reported regression in this PR description is #64632. Updating the link will make it easier to track the intended coverage and avoid confusion during future triage.
   ```suggestion
       Regression tests for conn_id-based credential resolution in ObjectStoragePath.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix ObjectStoragePath NoCredentialsError when using conn_id with remote stores [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #64634:
URL: https://github.com/apache/airflow/pull/64634#discussion_r3029381096


##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -116,6 +116,17 @@ def __init__(
         # to the underlying fsspec filesystem, which doesn't understand it
         self._conn_id = storage_options.pop("conn_id", None)
         super().__init__(*args, protocol=protocol, **storage_options)
+        # ProxyUPath delegates all operations to self.__wrapped__, which was
+        # constructed with empty storage_options (conn_id stripped above).
+        # Pre-populating __wrapped__._fs_cached with the Airflow-authenticated
+        # filesystem fixes every delegated method (exists, mkdir, iterdir, glob,
+        # walk, rename, read_bytes, write_bytes, …) in one place rather than
+        # requiring individual overrides for each one.
+        if self._conn_id is not None:
+            try:
+                self.__wrapped__._fs_cached = attach(self.protocol or "file", self._conn_id).fs
+            except (ImportError, ModuleNotFoundError):

Review Comment:
   `except (ImportError, ModuleNotFoundError)` won't catch the actual failure modes. When a scheme has no registered provider, `get_fs()` raises `ValueError`. The provider's `get_fs` method can raise connection-related errors too. Before this PR, construction always succeeded and errors deferred to first use. Now the constructor crashes with an uncaught `ValueError` for unregistered schemes. Broadening to `except Exception` preserves the lazy-failure contract. (`ModuleNotFoundError` is a subclass of `ImportError`, so listing both is also redundant.)



##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -127,6 +138,15 @@ def _from_upath(cls_or_self, upath, /):
         obj = object.__new__(cls)
         obj.__wrapped__ = upath
         obj._conn_id = getattr(cls_or_self, "_conn_id", None) if is_instance else None
+        # If the wrapped UPath has not yet had its fs cached (e.g. when _from_upath is
+        # called as a classmethod with a fresh UPath), inject the authenticated fs now.
+        # Child UPaths produced by __wrapped__ operations (iterdir, glob, etc.) already
+        # inherit _fs_cached from the parent UPath, so the hasattr check is a no-op for them.
+        if obj._conn_id is not None and not hasattr(upath, "_fs_cached"):
+            try:
+                obj.__wrapped__._fs_cached = attach(upath.protocol or "file", obj._conn_id).fs
+            except (ImportError, ModuleNotFoundError):

Review Comment:
   Same issue here -- the except clause should be broadened to match.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix ObjectStoragePath NoCredentialsError when using conn_id with remote stores [airflow]

Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #64634:
URL: https://github.com/apache/airflow/pull/64634#discussion_r3029955304


##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -116,6 +116,17 @@ def __init__(
         # to the underlying fsspec filesystem, which doesn't understand it
         self._conn_id = storage_options.pop("conn_id", None)
         super().__init__(*args, protocol=protocol, **storage_options)
+        # ProxyUPath delegates all operations to self.__wrapped__, which was
+        # constructed with empty storage_options (conn_id stripped above).
+        # Pre-populating __wrapped__._fs_cached with the Airflow-authenticated
+        # filesystem fixes every delegated method (exists, mkdir, iterdir, glob,
+        # walk, rename, read_bytes, write_bytes, …) in one place rather than
+        # requiring individual overrides for each one.
+        if self._conn_id is not None:
+            try:
+                self.__wrapped__._fs_cached = attach(self.protocol or "file", self._conn_id).fs

Review Comment:
   The bare `except Exception: pass` silently eats connection misconfiguration errors. If `attach()` fails because the connection is misconfigured (not just because the provider is uninstalled), the user gets a confusing error later from the unauthenticated fs. A `logger.debug(...)` with `exc_info=True` would give users a breadcrumb without being noisy.



##########
task-sdk/src/airflow/sdk/io/path.py:
##########
@@ -116,6 +116,17 @@ def __init__(
         # to the underlying fsspec filesystem, which doesn't understand it
         self._conn_id = storage_options.pop("conn_id", None)
         super().__init__(*args, protocol=protocol, **storage_options)
+        # ProxyUPath delegates all operations to self.__wrapped__, which was
+        # constructed with empty storage_options (conn_id stripped above).
+        # Pre-populating __wrapped__._fs_cached with the Airflow-authenticated
+        # filesystem fixes every delegated method (exists, mkdir, iterdir, glob,
+        # walk, rename, read_bytes, write_bytes, …) in one place rather than
+        # requiring individual overrides for each one.

Review Comment:
   Minor: the try/except/attach pattern is duplicated between `__init__` and `_from_upath`. A small private helper like `_inject_authenticated_fs(self, wrapped)` would reduce the duplication and make future changes (like adding logging) single-site.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix ObjectStoragePath NoCredentialsError when using conn_id with remote stores [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #64634:
URL: https://github.com/apache/airflow/pull/64634#issuecomment-4180480137

   Hi maintainer, this PR was merged without a milestone set.
   We've automatically set the milestone to **[Airflow 3.2.1](https://github.com/apache/airflow/milestone/137)** based on: backport label targeting v3-2-test
   If this milestone is not correct, please update it to the appropriate milestone.
   
   > This comment was generated by [Milestone Tag Assistant](https://github.com/apache/airflow/blob/main/.github/workflows/milestone-tag-assistant.yml).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org