You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "LudvigLundberg (via GitHub)" <gi...@apache.org> on 2024/02/12 11:18:39 UTC

[I] no template rendering for LivyOperator [airflow]

LudvigLundberg opened a new issue, #37357:
URL: https://github.com/apache/airflow/issues/37357

   ### Apache Airflow Provider(s)
   
   apache-livy
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-apache-livy==3.7.1
   
   ### Apache Airflow version
   
   2.8.1
   
   ### Operating System
   
   macOS Sonoma 14.0 (23A344)
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   Template rendering for the livy operator doesn*t work (tested with `args` and `files`), nothing is rendered in the rendered template tab in the task instance view and the un-rendered string is sent to livy, e.g.: 
   
   `Submitting job {"file": "/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi", "args": ["{{ task.task_id }}"], "numExecutors": 1, "conf": {"spark.shuffle.compress": "false"}}`
   
   I verified that template rendering is working in other DAGS by running the included `example_bash_operator` DAG
   
   ### What you think should happen instead
   
   When downgrading the version to `apache-airflow-providers-apache-livy==3.7.0`, rendered template tab is filled with values and the rendered string is sent to livy:
   
   `Submitting job {"file": "/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi", "args": ["pi_java_task"], "numExecutors": 1, "conf": {"spark.shuffle.compress": "false"}}`
   
   ### How to reproduce
   
   Following the instructions from https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html using docker compose and building the following Dockerfile:
   
   ```dockerfile
   FROM apache/airflow:2.8.1
   ADD requirements.txt .
   RUN pip install apache-airflow==2.8.1 -r requirements.txt
   ```
   
   A custom requirements.txt:
   
   ```
   apache-airflow-providers-apache-livy==3.7.1
   ```
   
   Running the following example DAG:
   
   ```python
   from datetime import datetime
   
   from airflow import DAG
   from airflow.providers.apache.livy.operators.livy import LivyOperator
   
   with DAG(
       dag_id='example_livy_operator',
       default_args={'args': [10]},
       schedule_interval='@daily',
       start_date=datetime(2021, 1, 1),
       catchup=False,
   ) as dag:
   
       # [START create_livy]
       livy_java_task = LivyOperator(
           task_id="pi_java_task",
           file='/spark-examples.jar',
           args=["{{ task.task_id }}"],
           num_executors=1,
           conf={
               'spark.shuffle.compress': 'false',
           },
           class_name='org.apache.spark.examples.SparkPi',
       )
   ```
   
   To get template rendering working change the requirements.txt to the following and rebuild/deploy airflow:
   
   ```
   apache-airflow-providers-apache-livy==3.7.0
   ```
   
   ### Anything else
   
   Seems the issue was introduced in https://github.com/apache/airflow/pull/36490 to fix https://github.com/apache/airflow/issues/36484
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] no template rendering for LivyOperator [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #37357:
URL: https://github.com/apache/airflow/issues/37357#issuecomment-1938483864

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] no template rendering for LivyOperator [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #37357: no template rendering for LivyOperator
URL: https://github.com/apache/airflow/issues/37357


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org