You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/17 15:40:27 UTC
[GitHub] [airflow] MatrixManAtYrService opened a new issue #22340: Expanding operators inside of task groups causes
MatrixManAtYrService opened a new issue #22340:
URL: https://github.com/apache/airflow/issues/22340
### Apache Airflow version
main (development)
### What happened
Given this DAG:
```python3
foo_var = {"VAR1": "FOO"}
bar_var = {"VAR1": "BAR"}
hi_cmd = 'echo "hello $VAR1"'
bye_cmd = 'echo "goodbye $VAR1"'
@task
def envs():
return [foo_var, bar_var]
@task
def cmds():
return [hi_cmd, bye_cmd]
with DAG(dag_id="mapped_bash", start_date=datetime(1970, 1, 1)) as dag:
with TaskGroup(group_id="dynamic"):
dynamic = BashOperator.partial(task_id="bash").expand(
env=envs(), bash_command=cmds()
)
```
I ran `airflow dags test mapped_bash $(date +%Y-%m-%dT%H:%M:%SZ)`
Got this output:
```
[2022-03-17 09:21:24,590] {taskinstance.py:1451} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=mapped_bash
AIRFLOW_CTX_TASK_ID=dynamic.bash
AIRFLOW_CTX_EXECUTION_DATE=2022-03-17T09:21:12+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill__2022-03-17T09:21:12+00:00
[2022-03-17 09:21:24,590] {subprocess.py:62} INFO - Tmp dir root location:
/var/folders/5m/nvs9yfcs6mlfm_63gnk6__3r0000gn/T
[2022-03-17 09:21:24,591] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'echo "hello $VAR1"']
[2022-03-17 09:21:24,597] {subprocess.py:85} INFO - Output:
[2022-03-17 09:21:24,601] {subprocess.py:92} INFO - hello FOO
[2022-03-17 09:21:24,601] {subprocess.py:96} INFO - Command exited with return code 0
[2022-03-17 09:21:24,616] {taskinstance.py:1277} INFO - Marking task as SUCCESS. dag_id=mapped_bash, task_id=dynamic.bash, execution_date=20220317T092112, start_date=20220317T152114, end_date=20220317T152124
[2022-03-17 09:21:24,638] {taskinstance.py:1752} WARNING - We expected to get frame set in local storage but it was not. Please report this as an issue with full logs at https://github.com/apache/airflow/issues/new
Traceback (most recent call last):
File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 1335, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 1437, in _execute_task_with_callbacks
self.render_templates(context=context)
File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 2091, in render_templates
task = self.task.render_template_fields(context)
File "/Users/matt/src/airflow/airflow/models/mappedoperator.py", line 602, in render_template_fields
unmapped_task = self.unmap()
File "/Users/matt/src/airflow/airflow/models/mappedoperator.py", line 454, in unmap
dag._remove_task(self.task_id)
File "/Users/matt/src/airflow/airflow/models/dag.py", line 2188, in _remove_task
task = self.task_dict.pop(task_id)
KeyError: 'dynamic.bash'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/matt/src/airflow/airflow/models/taskinstance.py", line 1750, in get_truncated_error_traceback
execution_frame = _TASK_EXECUTION_FRAME_LOCAL_STORAGE.frame
AttributeError: '_thread._local' object has no attribute 'frame'
```
### What you think should happen instead
No error
### How to reproduce
Trigger the dag above
### Operating System
Mac OS 11.6
### Versions of Apache Airflow Providers
n/a
### Deployment
Virtualenv installation
### Deployment details
```
❯ cd ~/src/airflow
❯ git rev-parse --short HEAD
df6058c86
❯ airflow info
Apache Airflow
version | 2.3.0.dev0
executor | SequentialExecutor
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
sql_alchemy_conn | sqlite:////Users/matt/2022/03/16/airflow.db
dags_folder | /Users/matt/2022/03/16/dags
plugins_folder | /Users/matt/2022/03/16/plugins
base_log_folder | /Users/matt/2022/03/16/logs
remote_base_log_folder |
System info
OS | Mac OS
architecture | x86_64
uname | uname_result(system='Darwin', node='LIGO', release='20.6.0', version='Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64', machine='x86_64')
locale | ('en_US', 'UTF-8')
python_version | 3.9.10 (main, Jan 15 2022, 11:48:00) [Clang 13.0.0 (clang-1300.0.29.3)]
python_location | /Users/matt/src/qa-scenario-dags/venv/bin/python3.9
```
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #22340: Expanding operators inside of task groups causes KeyError
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #22340:
URL: https://github.com/apache/airflow/issues/22340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org