You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "rlebreto (via GitHub)" <gi...@apache.org> on 2025/01/31 18:29:29 UTC
[I] error on _resolve_should_track_driver_status with spark standalone (cluster mode) [airflow]
rlebreto opened a new issue, #46334:
URL: https://github.com/apache/airflow/issues/46334
### Apache Airflow Provider(s)
apache-spark
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-spark 5.0.0
pyspark 3.5.4
### Apache Airflow version
2.10.3
### Operating System
ubuntu server 22.04
### Deployment
Other
### Deployment details
NA
### What happened
Hi
SparkSubmitOperator fails when interacting with a spark standalone server in deploy-mode=cluster.
In such condition, the _resolve_should_track_driver_status function in /usr/local/lib/python3.12/site-packages/airflow/providers/apache/spark/hooks/spark_submit.py returns True...
I have exactly the same pb as the one discussed here :
https://github.com/apache/airflow/discussions/21799
I did find a case opened for this issue.
### What you think should happen instead
I have an error when _resolve_should_track_driver_status returns True.
If I modify the function to return False, testing the task passes but in such case the final spark job status is not confirmed by the spark driver.
### How to reproduce
```
cat sparkoperator_test.py
import airflow
from datetime import datetime, timedelta
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator
default_args = {
'owner': 'xxxxxxxx',
'start_date': datetime(2025, 1, 31),
}
with airflow.DAG('sparkoperator_test',
default_args=default_args,
schedule_interval='@hourly',
description='test of sparkoperator in airflow by Omar and Regis',
) as dag:
spark_conf={
'spark.standalone.submit.waitAppCompletion':'true',
}
spark_compute_pi = SparkSubmitOperator(
task_id='spark-compute-pi',
conn_id='spark_qa1',
java_class='org.apache.spark.examples.SparkPi',
application='/opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar',
name='compute_pi_from_airflow',
application_args=["30"],
total_executor_cores=4,
executor_memory='1g',
executor_cores=1,
verbose='true',
conf=spark_conf,
execution_timeout=timedelta(minutes=10),
retries=0,
)
spark_compute_pi
```
spark version : 3.5.3
1 single node (hosting the master + 1 worker node)
The error happens when testing the task with `airflow tasks test ...`
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] error on _resolve_should_track_driver_status with spark standalone (cluster mode) [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #46334:
URL: https://github.com/apache/airflow/issues/46334#issuecomment-2628041431
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] error on _resolve_should_track_driver_status with spark standalone (cluster mode) [airflow]
Posted by "kgw7401 (via GitHub)" <gi...@apache.org>.
kgw7401 commented on issue #46334:
URL: https://github.com/apache/airflow/issues/46334#issuecomment-2766283759
i want to get assigned this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] error on _resolve_should_track_driver_status with spark standalone (cluster mode) [airflow]
Posted by "meetic-mrobin (via GitHub)" <gi...@apache.org>.
meetic-mrobin commented on issue #46334:
URL: https://github.com/apache/airflow/issues/46334#issuecomment-3183769140
@rlebreto and I found a solution that works, at least for our usecase (standalone, deploy_mode cluster, Spark 3.5).
Our conclusion is that `spark-submit --master XXX --status YYY` is broken (or we didn't understand how it's supposed to operate):
- if you provide a master with the binary port 7077, it fails because it tries to query it in HTTP
- if you provide a master with the HTTP port 6066, it fails because RestClient does not actually understand what HTTP Api returns
So we had to make sure Airflow tracks driver status using Curl strategy. Which means enters this condition:
```
if spark_host.endswith(":6066"):
```
https://github.com/apache/airflow/blob/main/providers/apache/spark/src/airflow/providers/apache/spark/hooks/spark_submit.py#L486
So we have to set port to 6066 in our Spark connector (ie. REST Api port).
In order to have job submission that works with this port, we added the following config flag in our DAG:
```
spark_conf={
'spark.master.rest.enabled':'true',
...
}
spark_compute_pi = SparkSubmitOperator(
conf=spark_conf,
...
)
```
We also had to enable REST API on spark server side (it's disabled by default in recent versions).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] error on _resolve_should_track_driver_status with spark standalone (cluster mode) [airflow]
Posted by "rlebreto (via GitHub)" <gi...@apache.org>.
rlebreto commented on issue #46334:
URL: https://github.com/apache/airflow/issues/46334#issuecomment-2788755890
Hi @kgw7401
Thanks a lot for your help.
To not be blocked, I forced the value returned by **_resolve_should_track_driver_status** from **/usr/local/lib/python3.12/site-packages/airflow/providers/apache/spark/hooks/spark_submit.py** to False
I'm not completely sure of the impact of the problem.
I'm suspecting that airflow will not be able to get back the status from the driver if the connection with it is lost.
I hope that running my sparkSubmit job with verbose='true' + retries=0 will allow me to track an error if the job fails, but I'm not very confident with that :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] error on _resolve_should_track_driver_status with spark standalone (cluster mode) [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #46334:
URL: https://github.com/apache/airflow/issues/46334#issuecomment-3986462064
@kgw7401 We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org