[jira] [Commented] (AIRFLOW-573) Jinja2 template failing env argument in BashOperater if string ends in .sh or .bash

["Adamos Loizou (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934824#comment-15934824 ] 

Adamos Loizou commented on AIRFLOW-573:
---------------------------------------

Hello,

I am using the latest Airflow version {{v1.7.1.3}} and I am also seeing a similar behaviour.
_TL;DR_: Airflow passes the content of bash scripts via Jinja and that is not explicitly mentioned or documented.

Details:
My use case is a bit different.
I explicitly specify a *shell script file name* for {{bash_command}}:

{code}
task = BashOperator(
    task_id='fetch_task',
    bash_command='run_task.sh',
    env={
        'SCRIPT_NAME': 'my_custom_fetch',
        'SCRIPT_ARGS': ' '.join([
            '--param1=value1',
            '--param2=value2',
            '--fetch-date={{ ds }}'
        ])
    },
    dag=dag
)
{code}

After debugging Airflow itself, I have concluded that:
* Airflow reads the content of the shell script file that I specify in {{bash_command}} *and then pass it through Jinja*, i.e. it treats its content as a template.
* This happens at {{airflow.models.BaseOperator#resolve_template_files}}
* {{BashOperator}} defines {{template_ext = ('.sh', '.bash',)}} as the endings of a {{bash_command}} that, when found, they should be considered for templating in {{resolve_template_files}}
* As such, the script filename passed should always be considered a *Jinja template*
* After introducing an all-escape Jinja block around my shell script content, it worked fine:
{code}
{% raw %}
#!/usr/bin/env bash

# Script here
# The line below which was using bash's array length syntax was causing a syntax error in Jinja - i.e. the {# #} jinja notation
echo "[${#PYTHON_SCRIPT_ARGS[@]}] Arguments passed"
{% endraw %}
{code}

Conclusion:
I would recommend that the documentation *reflects that bash scripts files will be treated as Jinja templates* and ideally I would also add an option to allow disabling that.

> Jinja2 template failing env argument in BashOperater if string ends in .sh or .bash
> -----------------------------------------------------------------------------------
>
>                 Key: AIRFLOW-573
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-573
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>    Affects Versions: Airflow 2.0
>            Reporter: Michael Shire
>
> A dot/period in a templated string appears to cause jinja template errors in Airflow.
> example:
> I constructed a dictionary to pass as an environment into BashOperator
> i.e.
> envvars = {'use_this_script':'/Users/mshire/x.sh'}
> task4 = BashOperator(
>     bash_command="env",
>     task_id="test_env_variables",
>     env=envvars,
>     dag=dag
>     )
> If I remove the dot/period (i.e. ".") in "x.sh" then it works.  Otherwise I get an error below.  I have tried it directly in python with jinja2 and it works, but not in Airflow.
> > airflow test scratch test_env_variables 2016-10-01
> [2016-10-16 00:39:32,981] {__init__.py:36} INFO - Using executor SequentialExecutor
> [2016-10-16 00:39:33,374] {models.py:154} INFO - Filling up the DagBag from /Users/mshire/airflow/dags
> envvars= {'use_this_script': '/Users/mshire/x.sh'}
> [2016-10-16 00:39:33,479] {models.py:1196} INFO -
> --------------------------------------------------------------------------------
> Starting attempt 1 of 1
> --------------------------------------------------------------------------------
> [2016-10-16 00:39:33,480] {models.py:1219} INFO - Executing <Task(BashOperator): test_env_variables> on 2016-10-01 00:00:00
> [2016-10-16 00:39:33,489] {models.py:1286} ERROR - /Users/mshire/x.sh
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1233, in run
>     self.render_templates()
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1409, in render_templates
>     rendered_content = rt(attr, content, jinja_context)
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2019, in render_template
>     return self.render_template_from_field(attr, content, context, jinja_env)
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, in render_template_from_field
>     for k, v in list(content.items())}
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1995, in <dictcomp>
>     for k, v in list(content.items())}
>   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 2017, in render_template
>     return jinja_env.get_template(content).render(**context)
>   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 812, in get_template
>     return self._load_template(name, self.make_globals(globals))
>   File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 774, in _load_template
>     cache_key = self.loader.get_source(self, name)[1]
>   File "/usr/local/lib/python2.7/site-packages/jinja2/loaders.py", line 187, in get_source
>     raise TemplateNotFound(template)
> TemplateNotFound: /Users/mshire/x.sh
> [2016-10-16 00:39:33,490] {models.py:1306} INFO - Marking task as FAILED.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)