You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "xs2tarunkukreja (via GitHub)" <gi...@apache.org> on 2023/10/17 07:37:07 UTC
[I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
xs2tarunkukreja opened a new issue, #34987:
URL: https://github.com/apache/airflow/issues/34987
### Apache Airflow version
2.7.2
### What happened
Currently I am using -
Python version: 3.8.18
Airflow version: 2.7.1v (on K8S)
When we run SparkKubernetesOperator function with YAML file path, it work fine. But when we are running the same after reading the YAML file, it fails.
Example -
def load_template_v1():
....
with open(template_path) as pyspark_template_file:
template = yaml.safe_load(pyspark_template_file)
return template
### What you think should happen instead
It should launch Spark Operator successfully.
### How to reproduce
Instead of passing YAML file, pass the content of file.
### Operating System
K8S
### Versions of Apache Airflow Providers
Python version: 3.8.18
Airflow version: 2.7.1
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
HELM
Python version: 3.8.18
Airflow version: 2.7.1
### Anything else
No
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "xs2tarunkukreja (via GitHub)" <gi...@apache.org>.
xs2tarunkukreja commented on issue #34987:
URL: https://github.com/apache/airflow/issues/34987#issuecomment-1766236733
however [docstring] said that it expected path to the YAML/JSON (as part of Airflow templates) or YAML/JSON string.
So, we can pass YAML string. Initially we are passing YAML file path and in second option, we are passing file content means YAML string.
So, it was not accidental thing. Would you please share the community link, where I should ask this question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "xs2tarunkukreja (via GitHub)" <gi...@apache.org>.
xs2tarunkukreja commented on issue #34987:
URL: https://github.com/apache/airflow/issues/34987#issuecomment-1766135993
It is working fine with older version of airflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #34987: SparkKubernetesOperator is not working when we are passing YAML string
URL: https://github.com/apache/airflow/issues/34987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #34987:
URL: https://github.com/apache/airflow/issues/34987#issuecomment-1766190590
That is not a part of Airflow core, this part of Community provider [`apache-airflow-providers-cncf-kubernetes`](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/changelog.html).
According provided stacktrace, you have a problem with `application_file` attribute, however [docstring](https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_api/airflow/providers/cncf/kubernetes/operators/spark_kubernetes/index.html#airflow.providers.cncf.kubernetes.operators.spark_kubernetes.SparkKubernetesOperator) said that it expected path to the YAML/JSON (as part of Airflow templates) or YAML/JSON string.
That mean previously it accidentally work as undocumented feature, however stop working after some release, I assume after this one https://github.com/apache/airflow/pull/29977.
Feel free to [open a feature request](https://github.com/apache/airflow/issues/new?assignees=&labels=kind%3Afeature%2Cneeds-triage&projects=&template=feature_request.yml), or if you know how to add this functionality feel free to open a PR directly:
- [Contributor's Quick Start](https://github.com/apache/airflow/blob/main/CONTRIBUTORS_QUICK_START.rst#contributors-quick-start)
- [Airflow Breeze](https://github.com/apache/airflow/blob/main/BREEZE.rst)
- [TESTING](https://github.com/apache/airflow/blob/main/TESTING.rst)
- [Static code checks](https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#static-code-checks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "xs2tarunkukreja (via GitHub)" <gi...@apache.org>.
xs2tarunkukreja commented on issue #34987:
URL: https://github.com/apache/airflow/issues/34987#issuecomment-1766133929
I am just creating a spark job. When I pass file name as application_file = "pipeline.yaml" in SparkKubernetesOperator, it works fine. But when I pass the yaml as string e.g. application_file = load_template_v1(), it start to fail.
I am getting following error -
[2023-10-17 10:23:24,375] Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 114, in execute
body = _load_body_to_dict(self.application_file)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 46, in _load_body_to_dict
body_dict = yaml.safe_load(body)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/yaml.py", line 46, in safe_load
return orig(stream, SafeLoader)
File "/home/airflow/.local/lib/python3.8/site-packages/yaml/__init__.py", line 79, in load
loader = Loader(stream)
File "/home/airflow/.local/lib/python3.8/site-packages/yaml/cyaml.py", line 26, in __init__
CParser.__init__(self, stream)
File "yaml/_yaml.pyx", line 288, in yaml._yaml.CParser.__init__
TypeError: a string or stream input is required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #34987:
URL: https://github.com/apache/airflow/issues/34987#issuecomment-1765839165
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #34987:
URL: https://github.com/apache/airflow/issues/34987#issuecomment-1766088399
> template = yaml.safe_load(pyspark_template_file)
This operation transform into the python objects, e.g. it could be dict, lists, and others.
The difficult to say in which attribute you tried to provide this output and is it even documented / expected.
Could you provide a bit more details, e.g. example of your tasks/dags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] SparkKubernetesOperator is not working when we are passing YAML string [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis closed issue #34987: SparkKubernetesOperator is not working when we are passing YAML string
URL: https://github.com/apache/airflow/issues/34987
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org