You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/11 02:28:07 UTC

[GitHub] [airflow] josh-fell opened a new issue, #24388: Unable to access operator attrs within Jinja context for mapped tasks

josh-fell opened a new issue, #24388:
URL: https://github.com/apache/airflow/issues/24388

   ### Apache Airflow version
   
   2.3.2 (latest released)
   
   ### What happened
   
   When attempting to generate mapped SQL tasks using a Jinja-templated query, an exception like the following is thrown:
   
   `jinja2.exceptions.UndefinedError: 'airflow.models.mappedoperator.MappedOperator object' has no attribute '<operator attribute>'`
   
   For example, when attempting to map `SQLValueCheckOperator` tasks with respect to `database` using a query of `SELECT COUNT(*) FROM {{ task.database }}.tbl;`:
   `jinja2.exceptions.UndefinedError: 'airflow.models.mappedoperator.MappedOperator object' has no attribute 'database'`
   
   Or, when using `SnowflakeOperator` and mapping via `parameters` of a query like `SELECT * FROM {{ task.parameters.tbl }};`:
   `jinja2.exceptions.UndefinedError: 'airflow.models.mappedoperator.MappedOperator object' has no attribute 'parameters'`
   
   ### What you think should happen instead
   
   When using Jinja-template SQL queries, the attribute that is being using for the mapping should be accessible via `{{ task.<operator attribute>. }}`. Executing the same SQL query with classic, non-mapped tasks allows for this operator attr access from the `task` context object. Ideally, the same interface should apply for both non-mapped and mapped tasks.
   
   ### How to reproduce
   
   Consider the following DAG:
   ```python
   from pendulum import datetime
   
   from airflow.decorators import dag
   from airflow.operators.sql import SQLValueCheckOperator
   from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
   
   
   CORE_SQL = "SELECT COUNT(*) FROM {{ task.database }}.tbl;"
   SNOWFLAKE_SQL = """SELECT * FROM {{ task.parameters.tbl }};"""
   
   
   @dag(dag_id="map-city", start_date=datetime(2022, 6, 7), schedule_interval=None)
   def map_city():
           classic_sql_value_check = SQLValueCheckOperator(
           task_id="classic_sql_value_check",
           conn_id="snowflake",
           sql=CORE_SQL,
           database="dev",
           pass_value=20000,
       )
   
       mapped_value_check = SQLValueCheckOperator.partial(
           task_id="check_row_count",
           conn_id="snowflake",
           sql=CORE_SQL,
           pass_value=20000,
       ).expand(database=["dev", "production"])
   
       classic_snowflake_task = SnowflakeOperator(
           task_id="classic_snowflake_task",
           snowflake_conn_id="snowflake",
           sql=SNOWFLAKE_SQL,
           parameters={"tbl": "foo"},
       )
   
       mapped_snowflake_task = SnowflakeOperator.partial(
           task_id="mapped_snowflake_task", snowflake_conn_id="snowflake", sql=SNOWFLAKE_SQL
       ).expand(
           parameters=[
               {"tbl": "foo"},
               {"tbl": "bar"},
           ]
       )
   
   
   _ = map_city()
   ```
   
   **`SQLValueCheckOperator` tasks**
   The logs for the "classic_sql_value_check", non-mapped task show the query executing as expected:
    `[2022-06-11, 02:01:03 UTC] {sql.py:204} INFO - Executing SQL check: SELECT COUNT(*) FROM dev.tbl;`
   while the mapped "check_row_count" task fails with the following exception:
   ```bash
   [2022-06-11, 02:01:03 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'map-city', 'check_row_count', 'manual__2022-06-11T02:01:01.831761+00:00', '--job-id', '350', '--raw', '--subdir', 'DAGS_FOLDER/map_city.py', '--cfg-path', '/tmp/tmpm5bg9mt5', '--map-index', '0', '--error-file', '/tmp/tmp2kbilt2l']
   [2022-06-11, 02:01:03 UTC] {standard_task_runner.py:80} INFO - Job 350: Subtask check_row_count
   [2022-06-11, 02:01:03 UTC] {task_command.py:370} INFO - Running <TaskInstance: map-city.check_row_count manual__2022-06-11T02:01:01.831761+00:00 map_index=0 [running]> on host 569596df5be5
   [2022-06-11, 02:01:03 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1451, in _run_raw_task
       self._execute_task_with_callbacks(context, test_mode)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1555, in _execute_task_with_callbacks
       task_orig = self.render_templates(context=context)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2212, in render_templates
       rendered_task = self.task.render_template_fields(context)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line 726, in render_template_fields
       self._do_render_template_fields(
     File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 68, in wrapper
       return func(*args, **kwargs)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 344, in _do_render_template_fields
       rendered_content = self.render_template(
     File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 391, in render_template
       return render_template_to_string(template, context)
     File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 296, in render_template_to_string
       return render_template(template, context, native=False)
     File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 291, in render_template
       return "".join(nodes)
     File "<template>", line 13, in root
     File "/usr/local/lib/python3.9/site-packages/jinja2/runtime.py", line 903, in _fail_with_undefined_error
       raise self._undefined_exception(self._undefined_message)
   jinja2.exceptions.UndefinedError: 'airflow.models.mappedoperator.MappedOperator object' has no attribute 'database'
   ```
   
   **`SnowflakeOperator` tasks**
   Similarly, the "classic_snowflake_task" non-mapped task is able to execute the SQL query as expected:
   `[2022-06-11, 02:01:04 UTC] {snowflake.py:324} INFO - Running statement: SELECT * FROM foo;, parameters: {'tbl': 'foo'}`
   while the mapped "mapped_snowflake_task task fails to execute the query:
   ```bash
   [2022-06-11, 02:01:03 UTC] {standard_task_runner.py:79} INFO - Running: ['airflow', 'tasks', 'run', 'map-city', 'mapped_snowflake_task', 'manual__2022-06-11T02:01:01.831761+00:00', '--job-id', '347', '--raw', '--subdir', 'DAGS_FOLDER/map_city.py', '--cfg-path', '/tmp/tmp6kmqs5ew', '--map-index', '0', '--error-file', '/tmp/tmpkufg9xqx']
   [2022-06-11, 02:01:03 UTC] {standard_task_runner.py:80} INFO - Job 347: Subtask mapped_snowflake_task
   [2022-06-11, 02:01:03 UTC] {task_command.py:370} INFO - Running <TaskInstance: map-city.mapped_snowflake_task manual__2022-06-11T02:01:01.831761+00:00 map_index=0 [running]> on host 569596df5be5
   [2022-06-11, 02:01:03 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1451, in _run_raw_task
       self._execute_task_with_callbacks(context, test_mode)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1555, in _execute_task_with_callbacks
       task_orig = self.render_templates(context=context)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2212, in render_templates
       rendered_task = self.task.render_template_fields(context)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line 726, in render_template_fields
       self._do_render_template_fields(
     File "/usr/local/lib/python3.9/site-packages/airflow/utils/session.py", line 68, in wrapper
       return func(*args, **kwargs)
     File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 344, in _do_render_template_fields
       rendered_content = self.render_template(
     File "/usr/local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 391, in render_template
       return render_template_to_string(template, context)
     File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 296, in render_template_to_string
       return render_template(template, context, native=False)
     File "/usr/local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 291, in render_template
       return "".join(nodes)
     File "<template>", line 13, in root
     File "/usr/local/lib/python3.9/site-packages/jinja2/sandbox.py", line 326, in getattr
       value = getattr(obj, attribute)
     File "/usr/local/lib/python3.9/site-packages/jinja2/runtime.py", line 910, in __getattr__
       return self._fail_with_undefined_error()
     File "/usr/local/lib/python3.9/site-packages/jinja2/runtime.py", line 903, in _fail_with_undefined_error
       raise self._undefined_exception(self._undefined_message)
   jinja2.exceptions.UndefinedError: 'airflow.models.mappedoperator.MappedOperator object' has no attribute 'parameters'
   ```
   
   ### Operating System
   
   Debian GNU/Linux 10 (buster)
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-snowflake==2.7.0
   
   ### Deployment
   
   Astronomer
   
   ### Deployment details
   
   Astronomer Runtime 5.0.3
   
   ### Anything else
   
   Even though using the `{{ task.<operator attr> }}` method does not work for mapped tasks, there is a workaround. Given the `SnowflakeOperator` example from above attempting to execute the query: `SELECT * FROM {{ task.parameters.tbl }};`, users can modify the query to `SELECT * FROM {{ task.mapped_kwargs.parameters[ti.map_index].tbl }};` for successful execution.
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on issue #24388: Unable to access operator attrs within Jinja context for mapped tasks

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #24388:
URL: https://github.com/apache/airflow/issues/24388#issuecomment-1258055457

   The `mapped_kwargs` attribute is intended to be private and is subject to breakages. The equivalent in 2.4 is `task.expand_input.value`.
   
   The fix has been included in #26100. Once that’s merged (to main and released in 2.5), I plan to write up a patch to backport the behaviour to 2.4, so `task` points to the _unmapped_ operator instead to allow users to simply access the attribute directly. If this means a lot to you, feel free to do the backporting yourself—the relevant part is adding `context_update_for_unmapped` and calling it at the correct place.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr closed issue #24388: Unable to access operator attrs within Jinja context for mapped tasks

Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #24388: Unable to access operator attrs within Jinja context for mapped tasks
URL: https://github.com/apache/airflow/issues/24388


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] uranusjr commented on issue #24388: Unable to access operator attrs within Jinja context for mapped tasks

Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #24388:
URL: https://github.com/apache/airflow/issues/24388#issuecomment-1259111512

   Unfortunately there’s currently no way to access the _resolved_ value of a mapped operator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mrn-aglic commented on issue #24388: Unable to access operator attrs within Jinja context for mapped tasks

Posted by GitBox <gi...@apache.org>.
mrn-aglic commented on issue #24388:
URL: https://github.com/apache/airflow/issues/24388#issuecomment-1259108844

   Thanks for the response. I tried to do it a couple of ways, including this:
   `task.expand_input.value.parameters.batch_start`
   
   But I get the following error:
   ```
   
   [2022-09-27, 07:36:38 UTC] {crypto.py:84} WARNING - empty cryptography key - values will not be stored encrypted.
   [2022-09-27, 07:36:38 UTC] {base.py:71} INFO - Using connection ID 'src' for task execution.
   [2022-09-27, 07:36:38 UTC] {sql.py:315} INFO - Running statement: SELECT * FROM match_scores OFFSET {{ task_instance.xcom_pull(task_ids='get_batch_starts', dag_id='task_mapping_classic_operators', key='batch_start') }} LIMIT 80, parameters: {'batch_start': '80', 'batch_end': '160'}
   [2022-09-27, 07:36:38 UTC] {taskinstance.py:1851} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/postgres/operators/postgres.py", line 92, in execute
       self.hook.run(self.sql, self.autocommit, parameters=self.parameters)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/common/sql/hooks/sql.py", line 295, in run
       self._run_command(cur, sql_statement, parameters)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/providers/common/sql/hooks/sql.py", line 318, in _run_command
       cur.execute(sql_statement, parameters)
   psycopg2.errors.SyntaxError: syntax error at or near "{"
   LINE 1: SELECT * FROM match_scores OFFSET {{ task_instance.xcom_pull...
                                             ^
   
   [2022-09-27, 07:36:38 UTC] {taskinstance.py:1401} INFO - Marking task as FAILED. dag_id=task_mapping_classic_operators, task_id=batch_select, map_index=1, execution_date=20220927T040000, start_date=20220927T073637, end_date=20220927T073638
   [2022-09-27, 07:36:38 UTC] {standard_task_runner.py:102} ERROR - Failed to execute job 171 for task batch_select (syntax error at or near "{"
   LINE 1: SELECT * FROM match_scores OFFSET {{ task_instance.xcom_pull...
                                             ^
   ```
   
   Here is the full operator definition:
   ```
    batch_select_op = PostgresOperator.partial(
           task_id="batch_select",
           postgres_conn_id="src",
           sql=f"SELECT * FROM match_scores OFFSET {{{{ task.expand_input.value.parameters.batch_start }}}} LIMIT {BATCH_SIZE}"
       ).expand(parameters=get_batch_starts.output)
   ```
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on issue #24388: Unable to access operator attrs within Jinja context for mapped tasks

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #24388:
URL: https://github.com/apache/airflow/issues/24388#issuecomment-1259275084

   > Unfortunately there’s currently no way to access the _resolved_ value of a mapped operator.
   
   No way _currently_. We'll add it in 2.4.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ashb commented on issue #24388: Unable to access operator attrs within Jinja context for mapped tasks

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #24388:
URL: https://github.com/apache/airflow/issues/24388#issuecomment-1259274969

   > Unfortunately there’s currently no way to access the _resolved_ value of a mapped operator.
   
   No way _currently_. We'll add it in 2.4.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] mrn-aglic commented on issue #24388: Unable to access operator attrs within Jinja context for mapped tasks

Posted by GitBox <gi...@apache.org>.
mrn-aglic commented on issue #24388:
URL: https://github.com/apache/airflow/issues/24388#issuecomment-1257885521

   The proposed workaround: `SELECT * FROM {{ task.mapped_kwargs.parameters[ti.map_index].tbl }};`
   Doesn't seem to be working on **airflow 2.4.0**. 
   I get the following exception:
   
   
   ```
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 410, in _do_render_template_fields
       rendered_content = self.render_template(
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 474, in render_template
       return render_template_to_string(template, context)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
       return render_template(template, cast(MutableMapping[str, Any], context), native=False)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 283, in render_template
       return "".join(nodes)
     File "<template>", line 14, in root
     File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/sandbox.py", line 326, in getattr
       value = getattr(obj, attribute)
     File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/runtime.py", line 859, in __getattr__
       return self._fail_with_undefined_error()
     File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/runtime.py", line 852, in _fail_with_undefined_error
       raise self._undefined_exception(self._undefined_message)
   jinja2.exceptions.UndefinedError: '***.models.mappedoperator.MappedOperator object' has no attribute 'mapped_kwargs'
   [2022-09-26, 11:23:52 UTC] {taskinstance.py:1851} ERROR - Task failed with exception
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1457, in _run_raw_task
       self._execute_task_with_callbacks(context, test_mode)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1576, in _execute_task_with_callbacks
       task_orig = self.render_templates(context=context)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2193, in render_templates
       rendered_task = self.task.render_template_fields(context)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line 764, in render_template_fields
       self._do_render_template_fields(
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 410, in _do_render_template_fields
       rendered_content = self.render_template(
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/abstractoperator.py", line 474, in render_template
       return render_template_to_string(template, context)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 288, in render_template_to_string
       return render_template(template, cast(MutableMapping[str, Any], context), native=False)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/helpers.py", line 283, in render_template
       return "".join(nodes)
     File "<template>", line 14, in root
     File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/sandbox.py", line 326, in getattr
       value = getattr(obj, attribute)
     File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/runtime.py", line 859, in __getattr__
       return self._fail_with_undefined_error()
     File "/home/airflow/.local/lib/python3.9/site-packages/jinja2/runtime.py", line 852, in _fail_with_undefined_error
       raise self._undefined_exception(self._undefined_message)
   jinja2.exceptions.UndefinedError: '***.models.mappedoperator.MappedOperator object' has no attribute 'mapped_kwargs'
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org