[jira] [Closed] (AIRFLOW-240) TemplateNotFound Error when using PostgresOperator and SQL script

["Greg Reda (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Greg Reda closed AIRFLOW-240.
-----------------------------
    Resolution: Not A Bug

PEBCAK error, specifically with the file paths I'd put into the tasks. They needed to be relative and not full paths. For example:

{code}
copy_s3_to_redshift = PostgresOperator(
    task_id='load_table',
    sql='/copy_to_redshift.sql',
    params=dict(
        AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
        AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
    ),
    postgres_conn_id='postgres_redshift',
    autocommit=False,
    dag=dag
)
{code}

And the templated command needed to change to:

{code}
COPY public.pitches FROM 's3://mybucket/test-data/import/heyward.csv'
CREDENTIALS 'aws_access_key_id={{ params.AWS_ACCESS_KEY_ID }};aws_secret_access_key={{ params.AWS_SECRET_ACCESS_KEY }}'
CSV
NULL as 'null'
IGNOREHEADER as 1;
{code}

> TemplateNotFound Error when using PostgresOperator and SQL script
> -----------------------------------------------------------------
>
>                 Key: AIRFLOW-240
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-240
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>         Environment: Ubuntu 14.04
>            Reporter: Greg Reda
>
> When using the PostgresOperator, I've been unable to get templating to work. It's quite possible I'm doing something wrong, but I'm pretty lost as to what the issue might be. Here's an example to reproduce the TemplateNotFound error I've been getting:
> {code:title=/my_dag/csv_to_redshift.py}
> copy_s3_to_redshift = PostgresOperator(
>     task_id='load_table',
>     sql=this_dag_path + '/copy_to_redshift.sql',
>     params=dict(
>         AWS_ACCESS_KEY_ID=Variable.get('AWS_ACCESS_KEY_ID'),
>         AWS_SECRET_ACCESS_KEY=Variable.get('AWS_SECRET_ACCESS_KEY')
>     ),
>     postgres_conn_id='postgres_redshift',
>     autocommit=False,
>     dag=dag
> )
> {code}
> {code:title=/my_dag/copy_to_redshift.sql}
> COPY public.table_foobar FROM 's3://mybucket/test-data/import/foobar.csv'
> CREDENTIALS 'aws_access_key_id={{ AWS_ACCESS_KEY_ID }};aws_secret_access_key={{ AWS_SECRET_ACCESS_KEY }}'
> CSV
> NULL as 'null'
> IGNOREHEADER as 1;
> {code}
> {code}
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 2038, in resolve_template_files
>     setattr(self, attr, env.loader.get_source(env, content)[0])
>   File "/usr/local/lib/python2.7/dist-packages/jinja2/loaders.py", line 187, in get_source
>     raise TemplateNotFound(template)
> TemplateNotFound: /home/gregreda/airflow/dags/my_dag/copy_to_redshift.sql
> {code}
> Any ideas towards a fix are much appreciated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)