You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "axelray-dev (via GitHub)" <gi...@apache.org> on 2026/07/13 16:50:28 UTC
[PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
axelray-dev opened a new pull request, #69834:
URL: https://github.com/apache/airflow/pull/69834
### Summary
Fixes inconsistent validation of custom `remote_base_dir` on `SSHRemoteJobOperator` cleanup (#69813).
The operator accepts a custom base directory when constructing job paths, but cleanup validation only allowed the hardcoded defaults (`/tmp/airflow-ssh-jobs` and the Windows temp default). Jobs with a custom base then failed at the end with `ValueError: Invalid job directory ... Expected path under '/tmp/airflow-ssh-jobs'`.
### Changes
- Thread an optional expected base into `_validate_job_dir` and the cleanup command builders, defaulting to the existing platform defaults.
- Pass the operator-configured base (`RemoteJobPaths.base_dir` / `remote_base_dir`) into cleanup so custom bases validate the same way as defaults.
- Keep prefix-plus-separator safety checks so cleanup still rejects paths outside the configured base.
### Verification
- Added unit tests for custom-base cleanup success and rejection of paths outside the custom base.
- Existing default-path cleanup tests remain.
- Targeted provider pytest could not be run in a sparse checkout without the full Airflow workspace; please run provider SSH unit tests in CI.
### Backwards compatibility
- Default behavior is unchanged when `remote_base_dir` is unset.
- Custom bases that already pass init validation now also pass cleanup validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "axelray-dev (via GitHub)" <gi...@apache.org>.
axelray-dev commented on PR #69834:
URL: https://github.com/apache/airflow/pull/69834#issuecomment-4964367767
Thanks for the detailed notes.
I agree we should keep the cleanup validation. Once job_dir comes back from the trigger event, that prefix check is basically the only guard before rm -rf.
I'll update this PR to:
- use `expected_base is None` instead of truthiness
- catch cleanup validation errors and warn/leave instead of failing the task
- add an operator-level custom `remote_base_dir` cleanup test so the execute_complete path is covered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #69834:
URL: https://github.com/apache/airflow/pull/69834#issuecomment-5021134006
@axelray-dev This PR has been converted to **draft** because it does not yet meet our [Pull Request quality criteria](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-quality-criteria).
**Issues found:**
- :x: **Pre-commit / static checks**: Failing: `CI image checks / Static checks`. Run `prek run --from-ref main --stage pre-commit` locally to reproduce and fix.
- :x: **Provider tests**: Failing: `Provider distributions tests / Compat 2.11.1:P3.10:`, `Provider distributions tests / Compat 3.0.6:P3.10:`, `Provider distributions tests / Compat 3.1.8:P3.10:`, `Provider distributions tests / Compat 3.2.2:P3.10:`, `Provider distributions tests / Compat 3.3.0:P3.10:`. Run `breeze testing providers-tests --test-type "Providers[<provider>]"` locally.
- :x: **Unresolved review comments**: There are 4 unresolved review thread(s) on this PR from @kaxil. For each one: either apply the suggestion in a follow-up commit, or reply in-thread with a brief explanation of why the feedback doesn't apply, or resolve the thread if the feedback is no longer relevant. **Once you believe a thread is resolved — whether by pushing a fix or by explaining why the suggestion doesn't apply — it is the author's responsibility to mark it as resolved by clicking the 'Resolve conversation' button at the bottom of the thread.** Reviewers don't auto-close their own threads, so unresolved threads read as 'still waiting on the author' and block the PR from moving forward.
> **Note:** Your branch is **126 commits behind `main`**. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.
**What to do next:**
- Fix each issue listed above.
- Make sure static checks pass locally (`prek run --from-ref main --stage pre-commit`).
- Mark the PR as "Ready for review" when you're done.
Converting a PR to draft is **not** a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the [Airflow Slack](https://s.apache.org/airflow-slack).
---
_Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this [two-stage triage process](https://github.com/apache/airflow/blob/main/contributing-docs/25_maintainer_pr_triage.md#why-the-first-pass-is-automated) so that our maintainers' limited time is spent where it matters most: the conversation with you._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "axelray-dev (via GitHub)" <gi...@apache.org>.
axelray-dev commented on PR #69834:
URL: https://github.com/apache/airflow/pull/69834#issuecomment-5232408973
Closing this in favor of #69885, which already merged the custom remote_base_dir cleanup fix for #69813.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "axelray-dev (via GitHub)" <gi...@apache.org>.
axelray-dev closed pull request #69834: Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation
URL: https://github.com/apache/airflow/pull/69834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #69834:
URL: https://github.com/apache/airflow/pull/69834#issuecomment-4960417660
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our [Contributors' Guide](https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [prek-hooks]( https://github.com/apache/airflow/blob/main/contributing-docs/08_static_code_checks.rst#prerequisites-for-prek-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/airflow-core/docs/howto/custom-operator.rst) Consider adding an example Dag that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/dev/breeze/doc/README.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "AchimGaedkeLynker (via GitHub)" <gi...@apache.org>.
AchimGaedkeLynker commented on code in PR #69834:
URL: https://github.com/apache/airflow/pull/69834#discussion_r3574156958
##########
providers/ssh/src/airflow/providers/ssh/utils/remote_job.py:
##########
@@ -452,27 +459,29 @@ def build_windows_kill_command(pid_file: str) -> str:
return f"powershell.exe -NoProfile -NonInteractive -EncodedCommand {encoded_script}"
-def build_posix_cleanup_command(job_dir: str) -> str:
+def build_posix_cleanup_command(job_dir: str, expected_base: str | None = None) -> str:
"""
Build a POSIX command to clean up the job directory.
:param job_dir: Path to the job directory
+ :param expected_base: Expected base directory for the job
:return: Shell command to remove the directory
:raises ValueError: If job_dir is not under the expected base directory
"""
- _validate_job_dir(job_dir, "posix")
+ _validate_job_dir(job_dir, "posix", expected_base)
Review Comment:
Hmm, interesting. I thought I could trust the return values of the trigger as it comes from the validated operator argument. I probably haven't conquered the full complexity of the deferral mechanism.
The eternal question of "how paranoid should I be" was (rightfully) preventing me from diving into a fix/PR straight away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on code in PR #69834:
URL: https://github.com/apache/airflow/pull/69834#discussion_r3574106477
##########
providers/ssh/src/airflow/providers/ssh/utils/remote_job.py:
##########
@@ -30,18 +30,25 @@
WINDOWS_DEFAULT_BASE_DIR = "$env:TEMP\\airflow-ssh-jobs"
-def _validate_job_dir(job_dir: str, remote_os: Literal["posix", "windows"]) -> None:
+def _validate_job_dir(
+ job_dir: str,
+ remote_os: Literal["posix", "windows"],
+ expected_base: str | None = None,
+) -> None:
"""
Validate that job_dir is under the expected base directory.
:param job_dir: The job directory path to validate
:param remote_os: Operating system type
+ :param expected_base: Expected base directory for the job
:raises ValueError: If job_dir doesn't start with the expected base path
"""
if remote_os == "posix":
- expected_prefix = POSIX_DEFAULT_BASE_DIR + "/"
+ expected_base = expected_base or POSIX_DEFAULT_BASE_DIR
Review Comment:
`expected_base or POSIX_DEFAULT_BASE_DIR` treats a falsy value as unset, but `RemoteJobPaths.__post_init__` only substitutes the default when `base_dir is None`. Since `remote_base_dir` is templated and only validated pre-render, a value that renders empty builds `job_dir` directly under `/` at submission, then cleanup validates it against the hardcoded default here and raises. Checking `expected_base is None` keeps the two consistent.
##########
providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py:
##########
@@ -451,10 +451,11 @@ def _cleanup_remote_job(self, job_dir: str, remote_os: str) -> None:
leave the directory rather than failing the (already finished) task.
"""
self.log.info("Cleaning up remote job directory: %s", job_dir)
+ expected_base = self._paths.base_dir if self._paths else self.remote_base_dir
if remote_os == "posix":
- cleanup_cmd = build_posix_cleanup_command(job_dir)
+ cleanup_cmd = build_posix_cleanup_command(job_dir, expected_base=expected_base)
Review Comment:
If validation fails here, the ValueError escapes `_cleanup_remote_job` before the retry loop and fails a task whose remote job already finished, which is the failure mode from #69813 and contradicts the log-and-leave contract in the docstring above. That's still reachable after this fix: `remote_base_dir` is templated, and templates are re-rendered when the operator resumes after deferral, so a value like `{{ var.value.ssh_jobs_base }}` re-renders against resume-time state while `event["job_dir"]` keeps the submission-time base the trigger was constructed with. If the Variable changes while the job runs, cleanup rejects a valid path and the task fails.
The cheap guard is to catch the ValueError and route it through the same warn-and-leave path used when retries are exhausted. A fuller fix would thread the submission-time base (`self._paths.base_dir`) through the trigger event alongside `job_dir` and validate against that, which also makes the `self._paths` fallback on the line above unnecessary, since cleanup always runs on a fresh instance where `_paths` is `None`. Happy for either to land as a follow-up if you'd rather keep this PR minimal.
##########
providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py:
##########
@@ -451,10 +451,11 @@ def _cleanup_remote_job(self, job_dir: str, remote_os: str) -> None:
leave the directory rather than failing the (already finished) task.
"""
self.log.info("Cleaning up remote job directory: %s", job_dir)
+ expected_base = self._paths.base_dir if self._paths else self.remote_base_dir
Review Comment:
The new tests pin the builders, but this line is the wiring #69813 actually hit: if the operator change were reverted and only the utils change kept, every test in the PR would still pass. `test_execute_complete_with_cleanup` in `operators/test_ssh_remote_job.py` is easy to clone into a variant with `remote_base_dir="/tmp-data/airflow-ssh-jobs"` and a matching event `job_dir`, which would cover the reported scenario end to end.
##########
providers/ssh/src/airflow/providers/ssh/utils/remote_job.py:
##########
@@ -452,27 +459,29 @@ def build_windows_kill_command(pid_file: str) -> str:
return f"powershell.exe -NoProfile -NonInteractive -EncodedCommand {encoded_script}"
-def build_posix_cleanup_command(job_dir: str) -> str:
+def build_posix_cleanup_command(job_dir: str, expected_base: str | None = None) -> str:
"""
Build a POSIX command to clean up the job directory.
:param job_dir: Path to the job directory
+ :param expected_base: Expected base directory for the job
:return: Shell command to remove the directory
:raises ValueError: If job_dir is not under the expected base directory
"""
- _validate_job_dir(job_dir, "posix")
+ _validate_job_dir(job_dir, "posix", expected_base)
Review Comment:
On the suggestion to drop this validation from the cleanup builders instead: I'd keep it. By the time cleanup runs, `job_dir` is whatever came back in the trigger event (`execute_complete` passes `event["job_dir"]` straight through), and it ends up in `rm -rf`. This prefix check is the only thing pinning an event-supplied path under the configured base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "axelray-dev (via GitHub)" <gi...@apache.org>.
axelray-dev commented on code in PR #69834:
URL: https://github.com/apache/airflow/pull/69834#discussion_r3622545124
##########
providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py:
##########
@@ -451,10 +451,11 @@ def _cleanup_remote_job(self, job_dir: str, remote_os: str) -> None:
leave the directory rather than failing the (already finished) task.
"""
self.log.info("Cleaning up remote job directory: %s", job_dir)
+ expected_base = self._paths.base_dir if self._paths else self.remote_base_dir
Review Comment:
Addressed on the current head.
Added `test_execute_complete_with_cleanup_custom_remote_base_dir` so the operator wiring for a custom `remote_base_dir` is covered through `execute_complete`, not only the utils builders.
##########
providers/ssh/src/airflow/providers/ssh/operators/ssh_remote_job.py:
##########
@@ -451,10 +451,11 @@ def _cleanup_remote_job(self, job_dir: str, remote_os: str) -> None:
leave the directory rather than failing the (already finished) task.
"""
self.log.info("Cleaning up remote job directory: %s", job_dir)
+ expected_base = self._paths.base_dir if self._paths else self.remote_base_dir
if remote_os == "posix":
- cleanup_cmd = build_posix_cleanup_command(job_dir)
+ cleanup_cmd = build_posix_cleanup_command(job_dir, expected_base=expected_base)
Review Comment:
Addressed on the current head.
`_cleanup_remote_job` now catches `ValueError` from cleanup validation, logs a warning, and leaves the remote directory in place instead of failing an already-finished task. Covered by `test_cleanup_validation_failure_leaves_directory_without_raising`.
##########
providers/ssh/src/airflow/providers/ssh/utils/remote_job.py:
##########
@@ -452,27 +459,29 @@ def build_windows_kill_command(pid_file: str) -> str:
return f"powershell.exe -NoProfile -NonInteractive -EncodedCommand {encoded_script}"
-def build_posix_cleanup_command(job_dir: str) -> str:
+def build_posix_cleanup_command(job_dir: str, expected_base: str | None = None) -> str:
"""
Build a POSIX command to clean up the job directory.
:param job_dir: Path to the job directory
+ :param expected_base: Expected base directory for the job
:return: Shell command to remove the directory
:raises ValueError: If job_dir is not under the expected base directory
"""
- _validate_job_dir(job_dir, "posix")
+ _validate_job_dir(job_dir, "posix", expected_base)
Review Comment:
Agreed. Kept the cleanup-path validation.
`job_dir` still comes from the trigger event at cleanup time, so the prefix check remains the guard before `rm -rf`.
##########
providers/ssh/src/airflow/providers/ssh/utils/remote_job.py:
##########
@@ -30,18 +30,25 @@
WINDOWS_DEFAULT_BASE_DIR = "$env:TEMP\\airflow-ssh-jobs"
-def _validate_job_dir(job_dir: str, remote_os: Literal["posix", "windows"]) -> None:
+def _validate_job_dir(
+ job_dir: str,
+ remote_os: Literal["posix", "windows"],
+ expected_base: str | None = None,
+) -> None:
"""
Validate that job_dir is under the expected base directory.
:param job_dir: The job directory path to validate
:param remote_os: Operating system type
+ :param expected_base: Expected base directory for the job
:raises ValueError: If job_dir doesn't start with the expected base path
"""
if remote_os == "posix":
- expected_prefix = POSIX_DEFAULT_BASE_DIR + "/"
+ expected_base = expected_base or POSIX_DEFAULT_BASE_DIR
Review Comment:
Addressed on the current head.
Validation now uses `expected_base is None` so empty/falsy values stay consistent with `RemoteJobPaths.__post_init__`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix SSHRemoteJobOperator custom remote_base_dir cleanup validation [airflow]
Posted by "achimgaedke (via GitHub)" <gi...@apache.org>.
achimgaedke commented on PR #69834:
URL: https://github.com/apache/airflow/pull/69834#issuecomment-4962048708
Suggestion: Remove the `_validate_job_dir` from the `build_posix_cleanup_command` - which achieves the same as the PR, but is less code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org