[I] SparkKubernetesOperator not finding .yaml file same folder as dag [airflow]

["devscheffer (via GitHub)" <gi...@apache.org>]

devscheffer opened a new issue, #40657:
URL: https://github.com/apache/airflow/issues/40657

   ### Apache Airflow version
   
   2.9.2
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   I'm trying to use spark kubernetes operator but I'm getting this error.
   
   File "/home/airflow/.local/lib/python3.10/site-packages/jinja2/loaders.py", line 204, in get_source
       raise TemplateNotFound(template)
   jinja2.exceptions.TemplateNotFound: /opt/airflow/dags/repo/airflow/dags/manual/pocs/sparkoperatortask/sparkapplication.yaml
   
   The manifest is in the same folder as my dag file
   Here is my dag
   
   from datetime import timedelta
   from pathlib import Pathimport pendulum
   from airflow.providers.cncf.kubernetes.operators.spark_kubernetes import (
       SparkKubernetesOperator,
   )
   from airflow import DAG
   
   default_args = {
       "owner": "airflow",
       "email_on_failure": False,
       "email_on_retry": False,
       "retries": 0,
       "retry_delay": timedelta(minutes=1),
   }
   
   base_folder = Path(__file__).parents[0]
   spark_app_manifest = str(base_folder.joinpath("sparkapplication.yaml"))
   
   with DAG(
       dag_id="kubernetes_spark_operator_example",
       default_args=default_args,
       description="An example DAG using Kubernetes Spark Operator",
       schedule=None,
       catchup=False,
       render_template_as_native_obj=True,
       max_active_runs=1,
       max_active_tasks=20,
       start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
       is_paused_upon_creation=False,
   ) as dag:
       spark_task = SparkKubernetesOperator(
           task_id="spark_pi_task",
           namespace="airflow-dags-dai-gigahorse",
           application_file=spark_app_manifest,
           kubernetes_conn_id="kubernetes_default",
           delete_on_termination=False,
       )
   
   ### What you think should happen instead?
   
   the operator should find the file without errors
   
   ### How to reproduce
   
   create a dag with a sparkkubernetesoperator and add a sparkapplication file yaml on the same folder
   
   ### Operating System
   
   ubuntu
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow = "^2.8.1"
   apache-airflow-providers-cncf-kubernetes = "^7.14.0"
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] SparkKubernetesOperator not finding .yaml file same folder as dag [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] closed issue #40657: SparkKubernetesOperator not finding .yaml file same folder as dag
URL: https://github.com/apache/airflow/issues/40657


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] SparkKubernetesOperator not finding .yaml file same folder as dag [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #40657:
URL: https://github.com/apache/airflow/issues/40657#issuecomment-3055958110

   This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] SparkKubernetesOperator not finding .yaml file same folder as dag [airflow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #40657:
URL: https://github.com/apache/airflow/issues/40657#issuecomment-3172421325

   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org