You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/11 11:43:02 UTC

[GitHub] [airflow] morooshka opened a new issue, #25668: SparkKubernetesOperator application file attribute "name" is not mandatory

morooshka opened a new issue, #25668:
URL: https://github.com/apache/airflow/issues/25668

   ### Apache Airflow version
   
   2.3.3
   
   ### What happened
   
   Since commit https://github.com/apache/airflow/commit/3c5bc73579080248b0583d74152f57548aef53a2 the SparkKubernetesOperator application file is expected to have an attribute metadata:name and the operator execution fails with exception `KeyError: 'name'` if it not exists. Please find the example error stack below:
   
   ```
   [2022-07-27, 12:58:07 UTC] {taskinstance.py:1909} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 69, in execute
       response = hook.create_custom_object(
     File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 316, in create_custom_object
       name=body_dict["metadata"]["name"],
   KeyError: 'name'
   ```
   
   ### What you think should happen instead
   
   The operator should start successfully, ignoring the field absence
   The attribute metadata:name in NOT mandatory, and a pair metadata:generateName can be user instead - please find proof here: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#objectmeta-v1-meta, particularly in the following:
   
   ```
   GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided
   ```
   
   
   ### How to reproduce
   
   Start a DAG with SparkKubernetesOperator with an application file like this in the beginning:
   
   ```
   apiVersion: sparkoperator.k8s.io/v1beta2
   kind: SparkApplication
   metadata:
     generateName: spark_app_name
   [...]
   ```
   
   ### Operating System
   
   linux
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow==2.3.3
   apache-airflow-providers-cncf-kubernetes==4.2.0
   
   
   ### Deployment
   
   Virtualenv installation
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   Every time
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #25668: SparkKubernetesOperator application file attribute "name" is not mandatory

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25668:
URL: https://github.com/apache/airflow/issues/25668#issuecomment-1211941822

   Feel free to submit PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #25668: SparkKubernetesOperator application file attribute "name" is not mandatory

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25668: SparkKubernetesOperator application file attribute "name" is not mandatory
URL: https://github.com/apache/airflow/issues/25668


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #25668: SparkKubernetesOperator application file attribute "name" is not mandatory

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25668:
URL: https://github.com/apache/airflow/issues/25668#issuecomment-1211876903

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org