[jira] [Closed] (AIRFLOW-5774) PapermillOperator fails to render parameters

["Mariusz Górski (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mariusz Górski closed AIRFLOW-5774.
-----------------------------------
    Resolution: Invalid

> PapermillOperator fails to render parameters
> --------------------------------------------
>
>                 Key: AIRFLOW-5774
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5774
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: utils
>    Affects Versions: 1.10.4
>            Reporter: Mariusz Górski
>            Priority: Minor
>              Labels: bug
>
> *Problem*
> When running sample PapermillOperator described in docs, it fails with
> {code:java}
> //[2019-10-27 10:48:53,134] {{taskinstance.py:1047}} ERROR - Can't compile non template nodes
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 922, in _run_raw_task
>     result = task_copy.execute(context=context)
>   File "/usr/local/lib/python3.7/site-packages/airflow/operators/papermill_operator.py", line 71, in execute
>     logging.info(inlet.as_dict())
>   File "/usr/local/lib/python3.7/site-packages/airflow/lineage/datasets.py", line 85, in as_dict
>     attributes[key] = env.from_string(value).render(**self.context)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 880, in from_string
>     return cls.from_code(self, self.compile(source), globals, None)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 581, in compile
>     defer_init=defer_init)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 543, in _generate
>     optimized=self.optimized)
>   File "/usr/local/lib/python3.7/site-packages/jinja2/compiler.py", line 78, in generate
>     raise TypeError('Can\'t compile non template nodes')
> TypeError: Can't compile non template nodes
> {code}
> *Reason*
> It's because: _class NoteBook(DataSet)_ expects _parameters_ to be _strings_, while papermill _execute_notebook_ function requires _parameters_ to be _dicts_. When we pass dict to _NoteBook.parameters_, then jinja templating fails when dict is passed to _env.from_string(value).render(**context)_ call. 
> *Solution:*
> Handling of nested structs in DataSet class
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)