You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Ferdinanddb (via GitHub)" <gi...@apache.org> on 2025/10/28 10:31:48 UTC
[I] The option "reattach_on_restart" and/or "delete_on_termination" from SparkKubernetesOperator are not working as expected [airflow]
Ferdinanddb opened a new issue, #57416:
URL: https://github.com/apache/airflow/issues/57416
### Apache Airflow version
3.1.1
### If "Other Airflow 2/3 version" selected, which one?
3.X.X
### What happened?
Sometimes, a SparkApplication is marked as completed (when looking at its state with `kubetctl`) but the job on Airflow side failed. Then the retry mechanism kicks in, and since the job is marked as completed then a mechanism is triggered to delete the job I believe, but it fails with the following error:
```
ERROR - Task failed with exception
AttributeError: 'SparkKubernetesOperator' object has no attribute 'launcher'
```
For now, I need to manually delete the SparkApplication marked as `completed`, then clear the Airflow job to retry it.
### What you think should happen instead?
When a SparkApplication is marked as `completed` but the Airflow TaskInstance failed, and the Airflow job has the arguments `reattach_on_restart=True` and `delete_on_termination=True`, then the next retry should find the SparkApplication (this is the case already) and delete it if its state is `completed`, or listen to it if its state is `running` (this is not the case).
### How to reproduce
I believe that the state that I am describing can be reached when a SparkApplication is still running in the background, while the Airflow task that triggered it failed for a reason not linked to the SparkApplication.
### Operating System
docker.io/apache/airflow:3.1.1-python3.12
### Versions of Apache Airflow Providers
apache-airflow[amazon, async, celery, cncf-kubernetes, google, http, postgres, slack, standard, fab, sftp, common-compat, openlineage]==3.1.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The option "reattach_on_restart" and/or "delete_on_termination" from SparkKubernetesOperator are not working as expected [airflow]
Posted by "Ferdinanddb (via GitHub)" <gi...@apache.org>.
Ferdinanddb commented on issue #57416:
URL: https://github.com/apache/airflow/issues/57416#issuecomment-3458933409
This is the behavior I am talking about:
As one can see, the SparkApplication is completed, but was not removed because the worker had an issue (I guess):
<img width="1940" height="256" alt="Image" src="https://github.com/user-attachments/assets/a394b871-a148-4b53-8446-700548f260a8" />
As one can see, the first attempt did not end up well (sometimes this happens but I guess this is normal behavior for CeleryExecutor in the sense that they can sometimes fail):
<img width="2220" height="716" alt="Image" src="https://github.com/user-attachments/assets/0876233e-82b7-4c83-ba4e-278f6c0cc221" />
But I have the feeling that the issue is fixed now, in the sense that the spark application started from scratch and succeeded in the next retry. Before upgrading to 3.1.1 (and all the providers using the constraints file), I thought I had another behavior (i.e. the next runs were blocked and I had to manually remove the sparkapplication objects using `kubectl` so that the job could restart properly). I close this issue, sorry for the noise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The option "reattach_on_restart" and/or "delete_on_termination" from SparkKubernetesOperator are not working as expected [airflow]
Posted by "jroachgolf84 (via GitHub)" <gi...@apache.org>.
jroachgolf84 commented on issue #57416:
URL: https://github.com/apache/airflow/issues/57416#issuecomment-3456282543
@Ferdinanddb, can you provide a code snippet of the DAG that you're trying to execute?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The option "reattach_on_restart" and/or "delete_on_termination" from SparkKubernetesOperator are not working as expected [airflow]
Posted by "Ferdinanddb (via GitHub)" <gi...@apache.org>.
Ferdinanddb commented on issue #57416:
URL: https://github.com/apache/airflow/issues/57416#issuecomment-3456344811
Sure, here is a snippet, and some remarks:
- If the bug happens again -> I will provide some screenshots and copy-paste the logs that I see.
- I strongly think that the behavior that I am trying to explain is similar to the one I described [here](https://github.com/apache/airflow/issues/56291), but I am not using the deferrable mode, just to be clear.
```python
import os
from datetime import datetime, timedelta
from pathlib import Path
from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import SparkKubernetesOperator
from airflow.sdk import DAG, Asset
default_args = {
"owner": "airflow",
"depends_on_past": False,
"email_on_failure": False,
"email_on_retry": False,
"retries": 1,
"start_date": datetime(2016, 1, 4),
"retry_delay": timedelta(minutes=1),
"weight_rule": "upstream",
}
with DAG(
dag_id=Path(__file__).parent.name,
schedule=None,
default_args=default_args,
max_active_runs=1,
max_active_tasks=50,
catchup=False,
on_failure_callback=default_dag_failure_slack_webhook_notification,
) as dag:
full_reload_job = SparkKubernetesOperator(
task_id="full_reload_job",
namespace="spark-operator",
application_file="spark_app/full_reload_job/spark_application_config.yml",
kubernetes_conn_id="kubernetes_default",
random_name_suffix=True,
get_logs=True,
reattach_on_restart=True,
delete_on_termination=True,
do_xcom_push=False,
deferrable=False,
retries=2,
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The option "reattach_on_restart" and/or "delete_on_termination" from SparkKubernetesOperator are not working as expected [airflow]
Posted by "Ferdinanddb (via GitHub)" <gi...@apache.org>.
Ferdinanddb closed issue #57416: The option "reattach_on_restart" and/or "delete_on_termination" from SparkKubernetesOperator are not working as expected
URL: https://github.com/apache/airflow/issues/57416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The option "reattach_on_restart" and/or "delete_on_termination" from SparkKubernetesOperator are not working as expected [airflow]
Posted by "thanhtrung5763 (via GitHub)" <gi...@apache.org>.
thanhtrung5763 commented on issue #57416:
URL: https://github.com/apache/airflow/issues/57416#issuecomment-4124496295
@Ferdinanddb
So to avoid this issue, I need upgrade to 3.1.1, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org