[jira] [Created] (AIRFLOW-5774) PapermillOperator fails to render parameters

["Mariusz Górski (Jira)" <ji...@apache.org>]

Mariusz Górski created AIRFLOW-5774:
---------------------------------------

             Summary: PapermillOperator fails to render parameters
                 Key: AIRFLOW-5774
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5774
             Project: Apache Airflow
          Issue Type: Bug
          Components: utils
    Affects Versions: 1.10.6
            Reporter: Mariusz Górski
            Assignee: Mariusz Górski


*Problem*

When running sample PapermillOperator described in docs, it fails with
{code:java}
//[2019-10-27 10:48:53,134] {{taskinstance.py:1047}} ERROR - Can't compile non template nodes
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 922, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python3.7/site-packages/airflow/operators/papermill_operator.py", line 71, in execute
    logging.info(inlet.as_dict())
  File "/usr/local/lib/python3.7/site-packages/airflow/lineage/datasets.py", line 85, in as_dict
    attributes[key] = env.from_string(value).render(**self.context)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 880, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 581, in compile
    defer_init=defer_init)
  File "/usr/local/lib/python3.7/site-packages/jinja2/environment.py", line 543, in _generate
    optimized=self.optimized)
  File "/usr/local/lib/python3.7/site-packages/jinja2/compiler.py", line 78, in generate
    raise TypeError('Can\'t compile non template nodes')
TypeError: Can't compile non template nodes
{code}
*Reason*

It's because: `class NoteBook(DataSet)` expects `parameters` to be `strings`, while papermill `execute_notebook` function requires `parameters` to be `dicts`. When we pass dict to NoteBook.parameters, then jinja templating fails when dict is passed to `env.from_string(value).render(**context)` call. 

*Solution:*

Handling of nested structs in DataSet class

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)