You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "sundeep8967 (via GitHub)" <gi...@apache.org> on 2026/08/30 03:51:34 UTC
[PR] Promote private SparkSubmitHook attributes used by strategy backends to public interface [airflow]
sundeep8967 opened a new pull request, #72285:
URL: https://github.com/apache/airflow/pull/72285
<!--
Thank you for contributing! Please make sure that your PR targets the `main` branch.
-->
Closes: #71058
### Description
Following the refactoring of `SparkSubmitOperator` into deployment backend strategies (`_KubernetesSparkSubmitBackend`, `_YarnSparkSubmitBackend`), several backend methods accessed private attributes and methods on `SparkSubmitHook` directly (`_conf`, `_kubernetes_driver_pod`, `_yarn_application_id`, `_poll_k8s_driver_via_api`).
This PR:
1. Exposes public properties on `SparkSubmitHook`:
- `@property def conf(self)`
- `@property def kubernetes_driver_pod(self)` (with getter and setter)
- `@property def yarn_application_id(self)`
2. Promotes `_poll_k8s_driver_via_api()` to public `poll_k8s_driver_via_api()` (maintaining `_poll_k8s_driver_via_api()` as a backwards-compatible alias).
3. Updates backend strategy classes in `spark_submit.py` to use these clean public properties and methods.
4. Adds unit test coverage in `tests/unit/apache/spark/hooks/test_spark_submit.py`.
Signed-off-by: Sundeep <su...@gmail.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Promote private SparkSubmitHook attributes used by strategy backends to public interface [airflow]
Posted by "sundeep8967 (via GitHub)" <gi...@apache.org>.
sundeep8967 commented on PR #72285:
URL: https://github.com/apache/airflow/pull/72285#issuecomment-5606112388
Hi @aaron-y-chen, thanks for the reminder. No generative AI was used to produce these code changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Promote private SparkSubmitHook attributes used by strategy backends to public interface [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil closed pull request #72285: Promote private SparkSubmitHook attributes used by strategy backends to public interface
URL: https://github.com/apache/airflow/pull/72285
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Promote private SparkSubmitHook attributes used by strategy backends to public interface [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on PR #72285:
URL: https://github.com/apache/airflow/pull/72285#issuecomment-5942558952
Closing this. #71058 was closed as not planned ("an internal implementation detail ... that doesn't justify a GitHub issue for tracking"), and this turns those private `SparkSubmitHook` attributes into permanent public hook surface. The new `or {}` guards are dead code, since `self._conf = conf or {}` already runs in `__init__`, and the operator still reads `self.hook._yarn_application_id`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Promote private SparkSubmitHook attributes used by strategy backends to public interface [airflow]
Posted by "aaron-y-chen (via GitHub)" <gi...@apache.org>.
aaron-y-chen commented on code in PR #72285:
URL: https://github.com/apache/airflow/pull/72285#discussion_r3904790397
##########
providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py:
##########
@@ -1290,6 +1309,10 @@ def _poll_k8s_driver_via_api(self) -> str | None:
finally:
self._run_post_submit_commands()
+ def _poll_k8s_driver_via_api(self) -> str | None:
+ """Deprecated alias for poll_k8s_driver_via_api."""
+ return self.poll_k8s_driver_via_api()
Review Comment:
Why is not
```python
def poll_k8s_driver_via_api(self):
return self._poll_k8s_driver_via_api()
```
##########
providers/apache/spark/src/airflow/providers/apache/spark/operators/spark_submit.py:
##########
@@ -162,15 +163,16 @@ class _YarnSparkSubmitBackend(_SparkSubmitDeploymentBackend):
"""Logic for tracking Spark applications in YARN cluster mode."""
def submit_job(self, context: Context) -> str | None:
- if self.hook._conf.get("spark.yarn.submit.waitAppCompletion", "").strip().lower() == "true":
+ conf = self.hook.conf or {}
Review Comment:
Why do we need this? We already have `or {}` in `_conf`.
https://github.com/apache/airflow/blob/24d8c66ee11e3003801b51511c4308e97a944a97/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org