You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hanna-liashchuk (via GitHub)" <gi...@apache.org> on 2023/05/10 11:42:42 UTC
[GitHub] [airflow] hanna-liashchuk opened a new issue, #31183: SparkKubernetesOperator doesn't push to xcom
hanna-liashchuk opened a new issue, #31183:
URL: https://github.com/apache/airflow/issues/31183
### Apache Airflow version
2.6.0
### What happened
After upgrading to 2.6.0 version, pipelines with SparkKubernetesOperator -> SparkKubernetesSensor stopped working correctly.
[this PR](https://github.com/apache/airflow/pull/29977) introduces some enhancement into Spark Kubernetes logic, now SparkKubernetesOperator receives the log from spark pods (which is great), but it doesn't monitor the status of a pod, which means if spark application fails - a task in Airflow finishes successfully.
On the other hand, using previous pipelines (Operator + Sensor) is impossible now, cause SparkKubernetesSensor fails with `jinja2.exceptions.UndefinedError: 'None' has no attribute 'metadata'` as SparkKubernetesOperator is no longer pushing info to xcom.
### What you think should happen instead
Old pipelines should be compatible with Airflow 2.6.0, even though the log would be retrieved in two places - operator and sensor.
OR remove the sensor completely and implement all the functionality in the operator (log + status)
### How to reproduce
Create a DAG with two operators
```
t1 = SparkKubernetesOperator(
kubernetes_conn_id='common/kubernetes_default',
task_id=f"task-submit",
namespace="namespace",
application_file="spark-applications/app.yaml",
do_xcom_push=True,
dag=dag,
)
t2 = SparkKubernetesSensor(
task_id=f"task-sensor",
namespace="namespace",
application_name=f"{{{{ task_instance.xcom_pull(task_ids='task-submit')['metadata']['name'] }}}}",
dag=dag,
attach_log=True,
)
```
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==8.0.0
apache-airflow-providers-apache-spark==4.0.1
apache-airflow-providers-celery==3.1.0
apache-airflow-providers-cncf-kubernetes==6.1.0
apache-airflow-providers-common-sql==1.4.0
apache-airflow-providers-docker==3.6.0
apache-airflow-providers-elasticsearch==4.4.0
apache-airflow-providers-ftp==3.3.1
apache-airflow-providers-google==10.0.0
apache-airflow-providers-grpc==3.1.0
apache-airflow-providers-hashicorp==3.3.1
apache-airflow-providers-http==4.3.0
apache-airflow-providers-imap==3.1.1
apache-airflow-providers-microsoft-azure==6.0.0
apache-airflow-providers-microsoft-mssql==3.3.2
apache-airflow-providers-microsoft-psrp==2.2.0
apache-airflow-providers-microsoft-winrm==3.1.1
apache-airflow-providers-mysql==5.0.0
apache-airflow-providers-odbc==3.2.1
apache-airflow-providers-oracle==3.6.0
apache-airflow-providers-postgres==5.4.0
apache-airflow-providers-redis==3.1.0
apache-airflow-providers-sendgrid==3.1.0
apache-airflow-providers-sftp==4.2.4
apache-airflow-providers-slack==7.2.0
apache-airflow-providers-snowflake==4.0.5
apache-airflow-providers-sqlite==3.3.2
apache-airflow-providers-ssh==3.6.0
apache-airflow-providers-telegram==4.0.0
### Deployment
Other 3rd-party Helm chart
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] adminidever commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "adminidever (via GitHub)" <gi...@apache.org>.
adminidever commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1569135706
with me is associated with the same problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1577135014
> @blcksrx - seems it was result of this change #29977 - is it possible that you take a look ?
The original issue true. since spark operator watches the logs until the job finishes, it didn't makes any sense to return that to sparkSensorOperator.
Also, I see I forgot to cover the case when spark pod fails (it could fix easily)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1576884112
@blcksrx - seems it was result of this change https://github.com/apache/airflow/pull/29977 - is it possible that you take a look ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jamescnowell commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "jamescnowell (via GitHub)" <gi...@apache.org>.
jamescnowell commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1577117254
I've created a PR (#31716) to revert the changes to the SparkOperator as it is:
1. Not backward compatible with the existing Operator
2. Not documented
3. Not clear what exactly the intended behavior is, though I _think_ I understand.
If this new style of Operator is desirable, I would propose adding it as a _new_ Spark operator with _new_ documentation on using it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1581016346
@eladkal you could assign me this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Owen-CH-Leung commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1555908290
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L96
I think the execute() method should return the object created by `hook.create_custom_object()` in order for xcom to work and sensor to poke the operator and get its logs. Also :
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L103-L115
these few lines are trying to populate spark driver log directly into `SparkKubernetesOperator`, but I think the role to fetch spark driver logs should be done by sensor ? sounds duplicated to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1577392774
> With regards to SparkKubernetesSensor, the Operator now remains Running for the duration of the Spark job, which (I think) means some DAG topologies will no longer work as expected. For example
>
> ```
> / ----> some other thing \
> Spark Job --> Final Thing
> \ ----> sensor ----------- /
> ```
>
> Will no longer run "some other thing" while the Spark Job is running, but instead after the job is completed and the DAG will need to be re-arranged.
>
> This will mean either:
>
> 1. People will need to update their DAGs when they notice a runtime change or
> 2. People will not notice the change, but will be running a useless SparkKubernetesSensor
>
> This is why I would propose making this a separate operator, or behind behavioral flags on the existing operator to maintain the existing behavior while documenting an upgrade path.
This example has a logical flaw, since the other thing is dependent to the `SparkJob`, if it wasn't dependent, it would be not related to the `SparkJob` and if it it was, it would be after the `SparkSensor`.
Airflow operators are responsible to doing a job complete. such as moving data between data sources and mean while they keep their state as running and also keeps the logs in the airflow output.
Hence I't doesn't makes any sense just to apply a spark job and ingore it or double the work by using another operator (sparkSensor) to wait on it.
Despite of this fact it an additional argument with default value such as `print_logs=False` could solves these backward compatibility issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hanna-liashchuk commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "hanna-liashchuk (via GitHub)" <gi...@apache.org>.
hanna-liashchuk commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1556090195
Agree, spark driver log in SparkKubernetesOperator is kinda duplicate but still insufficient since it's not displaying the status of the spark driver in the end. Would be great to have it all in one operator, sometimes out Sensor gets SIGTERM for no obvious reason even though spark itself is still running and finishing successfully. but this change is not backward compatible, so ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #31183: SparkKubernetesOperator doesn't push to xcom
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1542054087
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] blcksrx commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1578915006
Yeah, I’ll raise one till the end of the week On Tue, Jun 6, 2023 at 14:44, eladkal ***@***.***> wrote:
@blcksrx can you raise PR to address is?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jamescnowell commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "jamescnowell (via GitHub)" <gi...@apache.org>.
jamescnowell commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1577376322
With regards to SparkKubernetesSensor, the Operator now remains Running for the duration of the Spark job, which (I think) means some DAG topologies will no longer work as expected. For example
```
/ ----> some other thing \
Spark Job --> Final Thing
\ ----> sensor ----------- /
```
Will no longer run "some other thing" while the Spark Job is running, but instead after the job is completed and the DAG will need to be re-arranged.
This will mean either:
1. People will need to update their DAGs when they notice a runtime change or
2. People will not notice the change, but will be running a useless SparkKubernetesSensor
This is why I would propose making this a separate operator, or behind behavioral flags on the existing operator to maintain the existing behavior while documenting an upgrade path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
URL: https://github.com/apache/airflow/issues/31183
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] matheus-rossi commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "matheus-rossi (via GitHub)" <gi...@apache.org>.
matheus-rossi commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1556055467
Just confirming, all my tasks with spark operator on k8s are failing now.
I've noticed the logs on the SparkKubernetesOperator, and since this change the SparkKubernetesSensor now always returns this error:
`'None' has no attribute 'metadata' `
Taking the opportunity, if we could eliminate the sensor, it would be awesome. From time to time, I get errors on production, because the sensor didn't find the pod, even using timeToLive ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hanna-liashchuk commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "hanna-liashchuk (via GitHub)" <gi...@apache.org>.
hanna-liashchuk commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1544348169
yes, seems that missing return in execute() method caused this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pankajastro commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1542737072
maybe related to PR https://github.com/apache/airflow/pull/29977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #31183: SparkKubernetesSensor: 'None' has no attribute 'metadata'
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #31183:
URL: https://github.com/apache/airflow/issues/31183#issuecomment-1578698869
@blcksrx can you raise PR to address is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org