You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "codeprasan (via GitHub)" <gi...@apache.org> on 2023/08/16 16:40:07 UTC
[GitHub] [airflow] codeprasan opened a new issue, #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
codeprasan opened a new issue, #33451:
URL: https://github.com/apache/airflow/issues/33451
### Apache Airflow version
2.6.3
### What happened
While Spark Operator YAML using SparkKubernetesOperator below error is shown. Our Team noticed this error had earlier resolved as part of https://github.com/apache/airflow/issues/27147 however not addressed as part of Airflow2.6.3. Could you please make sure to address this issue in 2.6.3 and upcoming versions as well?
File "/opt/app-root/lib64/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 85, in execute
name = body["metadata"]["name"]
TypeError: string indices must be integers
### What you think should happen instead
The SparkKubernetesOperator should able to call the Spark Operator YAML file without any issues
### How to reproduce
- Set up the connection with kubernetes cluster
- Create a dag using SparkKubernetesOperator
- Provide your 'sparkApplication' as a path to a '.yaml' file
- Execute the dag and Hopefully, the same problem should occur
Below code will help to reproduce this error.
from airflow.providers.cncf.kubernetes.hooks.kubernetes import KubernetesHook, _load_body_to_dict
body = _load_body_to_dict('test.yaml')
print(body)
name = body["metadata"]["name"]
print(name)
### Operating System
FROM registry.access.redhat.com/ubi8/python-38
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==7.1.0
### Deployment
Other Docker-based deployment
### Deployment details
We are using docker file to deploy airflow to RedHat Openshift environment with help of below command
RUN pip3 install --trusted-host=files.pythonhosted.org --trusted-host=pypi.org "apache-airflow[crypto,celery,postgres,hive,jdbc,kubernetes,ssh,oracle,aws]==${AIRFLOW_VERSION}" --constraint constraints-3.8.txt --cert root.pem
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33451:
URL: https://github.com/apache/airflow/issues/33451#issuecomment-1703537722
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
URL: https://github.com/apache/airflow/issues/33451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33451:
URL: https://github.com/apache/airflow/issues/33451#issuecomment-1683383337
BTW. likely #27147 has nothing to do with it - while error is similar, this can be caused by a completely different content returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RNHTTR commented on issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #33451:
URL: https://github.com/apache/airflow/issues/33451#issuecomment-1681461054
What's the result of
```
body = _load_body_to_dict('test.yaml')
print(body)
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33451:
URL: https://github.com/apache/airflow/issues/33451#issuecomment-1712351712
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codeprasan commented on issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "codeprasan (via GitHub)" <gi...@apache.org>.
codeprasan commented on issue #33451:
URL: https://github.com/apache/airflow/issues/33451#issuecomment-1683336372
Here is the result and error,
name = body["metadata"]["name"]
TypeError: string indices must be integers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33451:
URL: https://github.com/apache/airflow/issues/33451#issuecomment-1683382631
What's your `test.yaml` ?
Simply adding print in your installed airflow code and printing whatever is the response dictionary ? I guess this is what your "test.yaml" is about - you took it from the response you received ?
This one:
```
_load_body_to_dict('test.yaml')
```
does not tell us much because we do not see what the `test.yaml` content is. But seeing what you receive in your environment might help in diagnosing the issue.
Not that I have 0 knowledge about yaml + spark but looking at the "reproduction" of yours it seems that your test.yaml is wrong and does not contain "metadata" field, or the field is wrong and does not contain "name" field. But maybe if you get more information about what you actually see as response you will a) understand the root cause b) it might help to improve some diagnostics. and better reaction to (I presume) some error that you see when receiving the response.
Without seeing the dict the there is little chance we can reason about it for anyone who would like to fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #33451: SparkKubernetesOperator: Dag fails when application_file sent as a ".yaml" file
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33451:
URL: https://github.com/apache/airflow/issues/33451#issuecomment-1680935988
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org