You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "anishgirianish (via GitHub)" <gi...@apache.org> on 2026/01/18 21:54:14 UTC
[PR] Add SSH/SFTP support for ObjectStoragePath [airflow]
anishgirianish opened a new pull request, #60757:
URL: https://github.com/apache/airflow/pull/60757
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull request.
Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the documentation.
Feel free to ping (in general) for the review if you do not see reaction for a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
## Summary
Adds SSH/SFTP protocol support for Airflow's `ObjectStoragePath`
abstraction, enabling file operations over SFTP using existing Airflow
connections.
- Implements `get_fs()` function using the `sshfs` library
(fsspec-compatible)
- Supports `sftp://` and `ssh://` URI schemes
- Integrates with existing SFTP connection configuration (host, port,
credentials, key files)
## Usage
```python
from airflow.sdk import ObjectStoragePath
# Read/write files over SFTP
path = ObjectStoragePath("sftp://my_sftp_conn@server/path/to/file.txt")
content = path.read_text()
# Copy between backends
path.copy(ObjectStoragePath("s3://aws_conn@bucket/file.txt"))
Changes
- providers/sftp/src/airflow/providers/sftp/fs/sftp.py - Core
implementation
- providers/sftp/tests/unit/sftp/fs/test_sftp.py - Unit tests
- providers/sftp/provider.yaml - Register filesystem
- providers/sftp/pyproject.toml - Add sshfs optional dependency
- providers/sftp/docs/filesystems/sftp.rst - Documentation
Installation
pip install apache-airflow-providers-sftp[sshfs]
closes: #60558
```
---
* Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
* For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
* When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
* For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add SSH/SFTP support for ObjectStoragePath [airflow]
Posted by "anishgirianish (via GitHub)" <gi...@apache.org>.
anishgirianish commented on PR #60757:
URL: https://github.com/apache/airflow/pull/60757#issuecomment-3788026706
Hi @bolkedebruin, hope you're doing well. I would like to request your review on this PR when you have a moment. Please take your time, and I truly appreciate any feedback you might have. Thank you so much - it means a lot! 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add SSH/SFTP support for ObjectStoragePath [airflow]
Posted by "anishgirianish (via GitHub)" <gi...@apache.org>.
anishgirianish commented on PR #60757:
URL: https://github.com/apache/airflow/pull/60757#issuecomment-3775737929
> lgtm. I see lots of tests for connection parsing, and I was going to comment about having an integration test - but I suppose as long as the connection is passed as expected, `sshfs` is going to handle all that.
>
> Were you able to do a full test with an SSH/SFTP system?
Thanks for the review, @fritz-astronomer! Yes, I spun up a local SFTP server with Docker and ran through the
basic operations - reads, writes, listing, deletes all worked fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add SSH/SFTP support for ObjectStoragePath [airflow]
Posted by "bolkedebruin (via GitHub)" <gi...@apache.org>.
bolkedebruin merged PR #60757:
URL: https://github.com/apache/airflow/pull/60757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org