You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/22 14:33:19 UTC
[GitHub] [airflow] WesleyBatista opened a new issue #14359: upgrade_check fails on ImportChangesRule
WesleyBatista opened a new issue #14359:
URL: https://github.com/apache/airflow/issues/14359
**Apache Airflow version**: 1.10.14
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
**Environment**:
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch)
- **Kernel** (e.g. `uname -a`): `Linux staging-airflow1.cve.one 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux`
- **Install tools**:
- **Others**:
**What happened**:
Running `airflow upgrade_check --config upgrade.yml` returns the following error:
```
airflow@staging-airflow1:~$ airflow upgrade_check --config upgrade.yml
Using config file: upgrade.yml
=========================================================================================== STATUS ==========================================================================================
Check for latest versions of apache-airflow and checker............................................................................................................................FAIL
Remove airflow.AirflowMacroPlugin class............................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom operators..................................................................................................................SUCCESS
Chain between DAG and operator not allowed.........................................................................................................................................SUCCESS
Connection.conn_type is not nullable...............................................................................................................................................SUCCESS
Custom Executors now require full path.............................................................................................................................................SUCCESS
Hooks that run DB functions must inherit from DBApiHook............................................................................................................................FAIL
Fernet is enabled by default.......................................................................................................................................................SUCCESS
GCP service account key deprecation................................................................................................................................................SUCCESS
Unify hostname_callable option in core section.....................................................................................................................................SUCCESS
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 37, in <module>
args.func(args)
File "/usr/local/lib/python3.6/dist-packages/airflow/upgrade/checker.py", line 118, in run
all_problems = check_upgrade(formatter, rules)
File "/usr/local/lib/python3.6/dist-packages/airflow/upgrade/checker.py", line 38, in check_upgrade
rule_status = RuleStatus.from_rule(rule)
File "/usr/local/lib/python3.6/dist-packages/airflow/upgrade/problem.py", line 48, in from_rule
messages = list(result)
File "/usr/local/lib/python3.6/dist-packages/airflow/upgrade/rules/import_changes.py", line 126, in _check_missing_providers
if current_airflow_version.major >= 2:
AttributeError: 'Version' object has no attribute 'major'
```
**What you expected to happen**: commands runs through and prints helpful messages
**How to reproduce it**: The minimum steps that should reproduce it are:
```
$ python3
Python 3.6.5 (default, May 9 2018, 10:19:36)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from packaging.version import Version
>>> current_airflow_version = Version(__import__("airflow").__version__)
>>> current_airflow_version.major >= 2
False
```
Unfortunately, as we can see it works fine.
**Anything else we need to know**: These are the versions we are using:
```
airflow@staging-airflow1:~$ pip3 freeze|grep airflow
airflow-plugins==0.4.0
apache-airflow==1.10.14
apache-airflow-upgrade-check==1.2.0
```
_`airflow-plugins` is an internal package that we maintain and I wouldn't think is related to the issue._
Adding `ImportChangesRule` to the list of `ignored_rules` _(as similarly suggested here #14262)_ makes the command run through, but then we might be oblivious to important recommendations related to changes in import paths of hooks, operators, sensors and others.
<details><summary>This is the output when we skip `ImportChangesRule` </summary>
```
airflow@staging-airflow1:~$ airflow upgrade_check --config upgrade.yml
Using config file: upgrade.yml
=========================================================================================== STATUS ==========================================================================================
Check for latest versions of apache-airflow and checker............................................................................................................................FAIL
Remove airflow.AirflowMacroPlugin class............................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom operators..................................................................................................................SUCCESS
Chain between DAG and operator not allowed.........................................................................................................................................SUCCESS
Connection.conn_type is not nullable...............................................................................................................................................SUCCESS
Custom Executors now require full path.............................................................................................................................................SUCCESS
Hooks that run DB functions must inherit from DBApiHook............................................................................................................................FAIL
Fernet is enabled by default.......................................................................................................................................................SUCCESS
GCP service account key deprecation................................................................................................................................................SUCCESS
Unify hostname_callable option in core section.....................................................................................................................................SUCCESS
Legacy UI is deprecated by default.................................................................................................................................................SUCCESS
Logging configuration has been moved to new section................................................................................................................................FAIL
Removal of Mesos Executor..........................................................................................................................................................SUCCESS
No additional argument allowed in BaseOperator.....................................................................................................................................SUCCESS
Users must set a kubernetes.pod_template_file value................................................................................................................................SKIPPED
SendGrid email uses old airflow.contrib module.....................................................................................................................................SUCCESS
Check Spark JDBC Operator default connection name..................................................................................................................................SUCCESS
Changes in import path of remote task handlers.....................................................................................................................................SUCCESS
Connection.conn_id is not unique...................................................................................................................................................SUCCESS
Found 6 problems.
====================================================================================== RECOMMENDATIONS ======================================================================================
Check for latest versions of apache-airflow and checker
-------------------------------------------------------
Check that the latest version of apache-airflow-upgrade-check is installed, and
that you are on the latest 1.10.x release of apache-airflow.
Problems:
1. Unable to go ask PyPI.org for latest release information: 'Version' object has no attribute 'is_devrelease'
Hooks that run DB functions must inherit from DBApiHook
-------------------------------------------------------
Hooks that run DB functions must inherit from DBApiHook instead of BaseHook
Problems:
1. Class <class 'airflow.hooks.http_hook.HttpHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
2. Class <class 'airflow.contrib.hooks.slack_webhook_hook.SlackWebhookHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
3. Class <redacted> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
Logging configuration has been moved to new section
---------------------------------------------------
The logging configurations have been moved from [core] to the new [logging] section.
Problems:
1. base_log_folder has been moved from [core] to a the new [logging] section.
Users must set a kubernetes.pod_template_file value
---------------------------------------------------
Skipped because this rule applies only to environment using KubernetesExecutor.
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #14359:
URL: https://github.com/apache/airflow/issues/14359#issuecomment-785410104
This is strange because it looks like you have different packaging versions?
In any case I think the easiest way to solve this is not to use `current_airflow_version.major` as this works only with `packaging >= 20.0`
https://github.com/pypa/packaging/blob/main/CHANGELOG.rst#200---2020-01-06
We don't have minimum requirement for [packaging](https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/setup.py#L635)
So probably changing https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/airflow/upgrade/rules/import_changes.py#L130
from:
`if current_airflow_version.major >= 2:`
to
`if current_airflow_version >= Version("2.0.0"):`
will solve this.
I'll PR shortly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14359:
URL: https://github.com/apache/airflow/issues/14359#issuecomment-785508548
Closed by https://github.com/apache/airflow/pull/14435
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] benbauer89 commented on issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
benbauer89 commented on issue #14359:
URL: https://github.com/apache/airflow/issues/14359#issuecomment-785702880
> I saw this issue some time ago but I don't remember what was the solution - maybe @benbauer89 remembers?
we had to ignore it - but yes we had the exact same error log (airflow 1.10.14 and python 3.6.*)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #14359:
URL: https://github.com/apache/airflow/issues/14359#issuecomment-785410104
This is strange because it looks like you have different packaging versions?
In any case I think the easiest way to solve this is not to use `current_airflow_version.major` as this works only with `packaging >= 20.0`
https://github.com/pypa/packaging/blob/main/CHANGELOG.rst#200---2020-01-06
We don't have minimum requirement for [packaging](https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/setup.py#L635)
So this just need to change https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/airflow/upgrade/rules/import_changes.py#L130
from:
`if current_airflow_version.major >= 2:`
to
`if current_airflow_version >= Version("2.0.0"):`
I'll PR shortly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #14359:
URL: https://github.com/apache/airflow/issues/14359#issuecomment-785410104
The easiest way to solve this is not to use `current_airflow_version.major` as this works only with `packaging >= 20.0`
https://github.com/pypa/packaging/blob/main/CHANGELOG.rst#200---2020-01-06
We don't have minimum requirement for [packaging](https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/setup.py#L635)
So this just need to change https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/airflow/upgrade/rules/import_changes.py#L130
from:
`if current_airflow_version.major >= 2:`
to
`if current_airflow_version >= Version("2.0.0"):`
I'll PR shortly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #14359:
URL: https://github.com/apache/airflow/issues/14359
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal edited a comment on issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
eladkal edited a comment on issue #14359:
URL: https://github.com/apache/airflow/issues/14359#issuecomment-785410104
This is strange because it looks like you don't use the same packaging version?
In any case I think the easiest way to solve this is not to use `current_airflow_version.major` as this works only with `packaging >= 20.0`
https://github.com/pypa/packaging/blob/main/CHANGELOG.rst#200---2020-01-06
We don't have minimum requirement for [packaging](https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/setup.py#L635)
So this just need to change https://github.com/apache/airflow/blob/e33313302173b6bd872523ced233acc26ff9d6fe/airflow/upgrade/rules/import_changes.py#L130
from:
`if current_airflow_version.major >= 2:`
to
`if current_airflow_version >= Version("2.0.0"):`
I'll PR shortly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #14359: upgrade_check fails on ImportChangesRule
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #14359:
URL: https://github.com/apache/airflow/issues/14359#issuecomment-785407287
I saw this issue some time ago but I don't remember what was the solution - maybe @benbauer89 remembers?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org