You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Flametaa (via GitHub)" <gi...@apache.org> on 2024/11/07 19:27:44 UTC
[I] Add SPARK_APPLICATION_NAME env variable to driver and executor pods when using SparkKubernetesOperator [airflow]
Flametaa opened a new issue, #43801:
URL: https://github.com/apache/airflow/issues/43801
### Description
I need to add an env variable in my spark pods that exposes the `SPARK_APPLICATION_NAME`, the issue here is that the operator adds a random suffix at the end, so I can't really add it at the level of the scheduler since it doesn't have access to the new name.
### Use case/motivation
I want the SparkKubernetesOperator to add an env variable containing the `SPARK_APPLICATION_NAME`, or make the random suffix fixed across the same dag run (exposing it as a macro?)
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add SPARK_APPLICATION_NAME env variable to driver and executor pods when using SparkKubernetesOperator [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43801:
URL: https://github.com/apache/airflow/issues/43801#issuecomment-3986475367
@Flametaa We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add SPARK_APPLICATION_NAME env variable to driver and executor pods when using SparkKubernetesOperator [airflow]
Posted by "sanchalitorpe-source (via GitHub)" <gi...@apache.org>.
sanchalitorpe-source commented on issue #43801:
URL: https://github.com/apache/airflow/issues/43801#issuecomment-3817471438
Hi @potiuk, I would like to work on adding a `SPARK_APPLICATION_NAME` environment variable to driver and executor pods when using `SparkKubernetesOperator`. I plan to make it deterministic per DAG run to ensure consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add SPARK_APPLICATION_NAME env variable to driver and executor pods when using SparkKubernetesOperator [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #43801:
URL: https://github.com/apache/airflow/issues/43801#issuecomment-2468040254
assigned you @Flametaa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add SPARK_APPLICATION_NAME env variable to driver and executor pods when using SparkKubernetesOperator [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl closed issue #43801: Add SPARK_APPLICATION_NAME env variable to driver and executor pods when using SparkKubernetesOperator
URL: https://github.com/apache/airflow/issues/43801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org