You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/28 20:24:41 UTC
[GitHub] [airflow] madison-ookla opened a new issue #13350: UndefinedJinjaVariablesRule fails on non-standard python objects
madison-ookla opened a new issue #13350:
URL: https://github.com/apache/airflow/issues/13350
**Apache Airflow version**: 1.10.14
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**: X
- **OS** (e.g. from /etc/os-release): X
- **Kernel** (e.g. `uname -a`): X
- **Install tools**: X
- **Others**: X
**What happened**:
The `UndefinedJinjaVariablesRule` appears to expect non-standard python objects to have a `template_fields` attribute, when many (`pathlib.Path`, `None`) do not.
<details><summary>`Path` example</summary>
```python
========================================================================================================================================================== STATUS ==========================================================================================================================================================
Check for latest versions of apache-airflow and checker...........................................................................................................................................................................................................................................................SUCCESS
Remove airflow.AirflowMacroPlugin class...........................................................................................................................................................................................................................................................................SUCCESS
Chain between DAG and operator not allowed........................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_id is not unique..................................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_type is not nullable..............................................................................................................................................................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom operators.................................................................................................................................................................................................................................................SUCCESS
Fernet is enabled by default......................................................................................................................................................................................................................................................................................FAIL
GCP service account key deprecation...............................................................................................................................................................................................................................................................................SUCCESS
Unify hostname_callable option in core section....................................................................................................................................................................................................................................................................FAIL
Changes in import paths of hooks, operators, sensors and others...................................................................................................................................................................................................................................................SUCCESS
Users must delete deprecated configs for KubernetesExecutor.......................................................................................................................................................................................................................................................FAIL
Legacy UI is deprecated by default................................................................................................................................................................................................................................................................................SUCCESS
Logging configuration has been moved to new section...............................................................................................................................................................................................................................................................FAIL
Removal of Mesos Executor.........................................................................................................................................................................................................................................................................................SUCCESS
No additional argument allowed in BaseOperator....................................................................................................................................................................................................................................................................SUCCESS
Users must set a kubernetes.pod_template_file value...............................................................................................................................................................................................................................................................SKIPPED
SendGrid email uses old airflow.contrib module....................................................................................................................................................................................................................................................................SUCCESS
Changes in import path of remote task handlers....................................................................................................................................................................................................................................................................SUCCESS
Traceback (most recent call last):
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow", line 37, in <module>
args.func(args)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 88, in run
all_problems = check_upgrade(formatter, rules)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 37, in check_upgrade
rule_status = RuleStatus.from_rule(rule)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py", line 44, in from_rule
result = rule.check()
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 141, in check
dag_messages = self.iterate_over_dag_tasks(dag)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 128, in iterate_over_dag_tasks
error_messages = self.iterate_over_template_fields(task)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 117, in iterate_over_template_fields
self._check_rendered_content(rendered_content, set())
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 63, in _check_rendered_content
debug_error_messages.update(self._check_rendered_content(value))
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 69, in _check_rendered_content
return self._nested_check_rendered(rendered_content, seen_oids)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 75, in _nested_check_rendered
nested_template_fields = rendered_content.template_fields
AttributeError: 'PosixPath' object has no attribute 'template_fields'
```
</details>
<details><summary>`None` example</summary>
```python
========================================================================================================================================================== STATUS ==========================================================================================================================================================
Check for latest versions of apache-airflow and checker...........................................................................................................................................................................................................................................................SUCCESS
Remove airflow.AirflowMacroPlugin class...........................................................................................................................................................................................................................................................................SUCCESS
Chain between DAG and operator not allowed........................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_id is not unique..................................................................................................................................................................................................................................................................................SUCCESS
Connection.conn_type is not nullable..............................................................................................................................................................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom operators.................................................................................................................................................................................................................................................SUCCESS
Fernet is enabled by default......................................................................................................................................................................................................................................................................................FAIL
GCP service account key deprecation...............................................................................................................................................................................................................................................................................SUCCESS
Unify hostname_callable option in core section....................................................................................................................................................................................................................................................................FAIL
Changes in import paths of hooks, operators, sensors and others...................................................................................................................................................................................................................................................SUCCESS
Users must delete deprecated configs for KubernetesExecutor.......................................................................................................................................................................................................................................................FAIL
Legacy UI is deprecated by default................................................................................................................................................................................................................................................................................SUCCESS
Logging configuration has been moved to new section...............................................................................................................................................................................................................................................................FAIL
Removal of Mesos Executor.........................................................................................................................................................................................................................................................................................SUCCESS
No additional argument allowed in BaseOperator....................................................................................................................................................................................................................................................................SUCCESS
Users must set a kubernetes.pod_template_file value...............................................................................................................................................................................................................................................................SKIPPED
SendGrid email uses old airflow.contrib module....................................................................................................................................................................................................................................................................SUCCESS
Changes in import path of remote task handlers....................................................................................................................................................................................................................................................................SUCCESS
Error with <Task(PythonOperator): ts_manual_edits>: 'NoneType' object has no attribute 'template_fields'
Traceback (most recent call last):
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/bin/airflow", line 37, in <module>
args.func(args)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 88, in run
all_problems = check_upgrade(formatter, rules)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/checker.py", line 37, in check_upgrade
rule_status = RuleStatus.from_rule(rule)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/problem.py", line 44, in from_rule
result = rule.check()
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 146, in check
dag_messages = self.iterate_over_dag_tasks(dag)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 130, in iterate_over_dag_tasks
error_messages = self.iterate_over_template_fields(task)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 118, in iterate_over_template_fields
self._check_rendered_content(rendered_content, set())
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 64, in _check_rendered_content
debug_error_messages.update(self._check_rendered_content(value))
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 70, in _check_rendered_content
return self._nested_check_rendered(rendered_content, seen_oids)
File "/Users/madison/programs/anaconda3/envs/memphis-airflow/lib/python3.8/site-packages/airflow/upgrade/rules/undefined_jinja_varaibles.py", line 76, in _nested_check_rendered
nested_template_fields = rendered_content.template_fields
AttributeError: 'NoneType' object has no attribute 'template_fields'
```
</details>
**What you expected to happen**:
I expect these types to be handled appropriately by the upgrade rule and not assume that a `template_fields` attribute exists on them.
**How to reproduce it**:
Create a simple DAG with a `PythonOperator` task that has `op_kwargs: {"foo": None}` and run the upgrade check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13350: UndefinedJinjaVariablesRule fails on non-standard python objects
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13350:
URL: https://github.com/apache/airflow/issues/13350#issuecomment-752280813
This will be released in apache-airflow-upgrade-check **1.2.0**
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #13350: UndefinedJinjaVariablesRule fails on non-standard python objects
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #13350:
URL: https://github.com/apache/airflow/issues/13350#issuecomment-752278918
fixed by https://github.com/apache/airflow/pull/13373
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #13350: UndefinedJinjaVariablesRule fails on non-standard python objects
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #13350:
URL: https://github.com/apache/airflow/issues/13350
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org