You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2026/09/22 23:21:48 UTC
[I] Concurrent SFTP directory transfers drop the parent hook's connection overrides [airflow]
potiuk opened a new issue, #73585:
URL: https://github.com/apache/airflow/issues/73585
`SFTPHook.store_directory_concurrently()` and `retrieve_directory_concurrently()` build one hook per worker as:
```python
SFTPHook(ssh_conn_id=self.ssh_conn_id, no_host_key_check=self.no_host_key_check)
```
Since #73419 the workers inherit `no_host_key_check`, but every other constructor override on the parent hook is still discarded — `remote_host`, `port`, `username`, `password`, `key_file`, proxy settings, timeouts — so the workers fall back to whatever the connection defines. With an explicit `remote_host`, for example, the directory is listed on that host but the workers transfer against the connection's original host.
**Expected:** worker hooks are built with the parent's effective connection settings.
**Regression test:** patch worker-hook construction, configure a parent `SFTPHook` with a `remote_host` (and port/username) different from the connection, run a concurrent transfer, and assert every worker hook receives the parent's effective values.
Pre-existing; surfaced during review of #73419 and deliberately left out of that PR's scope.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Concurrent SFTP directory transfers drop the parent hook's connection overrides [airflow]
Posted by "abhishekmauryaKsolves (via GitHub)" <gi...@apache.org>.
abhishekmauryaKsolves commented on issue #73585:
URL: https://github.com/apache/airflow/issues/73585#issuecomment-5808277153
I'd like to work on this. Planning to pass the parent hook's resolved attributes (not just constructor args) into the worker hook constructor calls in both store_directory_concurrently() and retrieve_directory_concurrently(), so overrides like remote_host/port/username/password/key_file/proxy/timeouts propagate consistently, same as no_host_key_check does now. Will add the regression test that patches worker-hook construction and asserts the effective values are passed through. Assigning myself — let me know if there's a preferred approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Concurrent SFTP directory transfers drop the parent hook's connection overrides [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #73585: Concurrent SFTP directory transfers drop the parent hook's connection overrides
URL: https://github.com/apache/airflow/issues/73585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org