You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/23 20:32:36 UTC
[GitHub] [airflow] jsmodic opened a new issue #13293: Rule UndefinedJinjaVariablesRule is unusably slow with many DAGs
jsmodic opened a new issue #13293:
URL: https://github.com/apache/airflow/issues/13293
**Apache Airflow version**:
Airflow 1.10.14
Checker 1.1.0
**What happened**:
Running "airflow check_upgrade" will basically then work for at least 20+ hours on UndefinedJinjaVariablesRule at 100% CPU (that is until I killed it). I have a folder of 54 DAGs, most of them are fairly complicated.
Stack trace when I killed it:
```
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/checker.py", line 101, in <module>
__main__()
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/checker.py", line 97, in __main__
args.func(args)
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/checker.py", line 88, in run
all_problems = check_upgrade(formatter, rules)
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/checker.py", line 37, in check_upgrade
rule_status = RuleStatus.from_rule(rule)
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/problem.py", line 44, in from_rule
result = rule.check()
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 141, in check
dag_messages = self.iterate_over_dag_tasks(dag)
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 128, in iterate_over_dag_tasks
error_messages = self.iterate_over_template_fields(task)
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 114, in iterate_over_template_fields
task, content, context
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 96, in _render_task_content
renderend_content = task.render_template(content, context)
File "/home/ubuntu/.local/lib/python3.6/site-packages/airflow/models/baseoperator.py", line 755, in render_template
return jinja_env.from_string(content).render(**context)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/environment.py", line 941, in from_string
return cls.from_code(self, self.compile(source), globals, None)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/environment.py", line 629, in compile
source = self._generate(source, name, filename, defer_init=defer_init)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/environment.py", line 592, in _generate
optimized=self.optimized,
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/compiler.py", line 87, in generate
generator.visit(node)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/visitor.py", line 32, in visit
return f(node, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/compiler.py", line 774, in visit_Template
self.blockvisit(node.body, frame)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/compiler.py", line 381, in blockvisit
self.visit(node, frame)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/visitor.py", line 32, in visit
return f(node, *args, **kwargs)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/compiler.py", line 1429, in visit_Output
self.writeline("yield " + val)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/compiler.py", line 402, in writeline
self.write(x)
File "/home/ubuntu/.local/lib/python3.6/site-packages/jinja2/compiler.py", line 396, in write
self._new_lines = 0
```
I can just ignore that rule no problem and it finishes in a few seconds, but I imagine it's going to be quite confusing to anyone else that gets into this situation where it looks stuck without digging into the source.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13293: Rule UndefinedJinjaVariablesRule is unusably slow with many DAGs
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13293:
URL: https://github.com/apache/airflow/issues/13293#issuecomment-759528073
For now, users should just ignore this check until a fix comes out in **Airflow Upgrade Check - 1.2.0** :
```
airflow upgrade_check -i UndefinedJinjaVariablesRule
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13293: Rule UndefinedJinjaVariablesRule is unusably slow with many DAGs
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13293:
URL: https://github.com/apache/airflow/issues/13293
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org