You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/12 14:13:10 UTC
[GitHub] [airflow] OliverPfau opened a new issue #13636: Import warnings from webserver
OliverPfau opened a new issue #13636:
URL: https://github.com/apache/airflow/issues/13636
Hi,
I uninstalled airflow 1.10.12 and installed airflow 2.0.0. When I start the webserver I see warning about failed imports. I haven't deployed any custom DAG and running with python 3.6.9. Can it be ignored? They are logged periodically.
```
[2021-01-12 15:11:40 +0100] [27648] [INFO] Booting worker with pid: 27648
[2021-01-12 15:11:42,386] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.dataprep.GoogleDataprepHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,386] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,386] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,387] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.compute_ssh.ComputeEngineSSHHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,387] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.bigquery.BigQueryHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,387] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.common.hooks.base_google.GoogleBaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common'
[2021-01-12 15:11:42,469] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.dataprep.GoogleDataprepHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,470] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,470] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,470] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.compute_ssh.ComputeEngineSSHHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,471] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.bigquery.BigQueryHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:42,471] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.common.hooks.base_google.GoogleBaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common'
[2021-01-12 15:11:43,949] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.dataprep.GoogleDataprepHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:43,950] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:43,950] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:43,950] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.compute_ssh.ComputeEngineSSHHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:43,951] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.bigquery.BigQueryHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:43,951] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.common.hooks.base_google.GoogleBaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common'
[2021-01-12 15:11:44,026] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.dataprep.GoogleDataprepHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:44,027] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:44,027] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.cloud_sql.CloudSQLDatabaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:44,027] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.compute_ssh.ComputeEngineSSHHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:44,028] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.cloud.hooks.bigquery.BigQueryHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.cloud'
[2021-01-12 15:11:44,028] {providers_manager.py:283} WARNING - Exception when importing 'airflow.providers.google.common.hooks.base_google.GoogleBaseHook' from 'apache-airflow-providers-google' package: No module named 'airflow.providers.google.common'
[2021-01-12 15:11:48 +0100] [11659] [INFO] Handling signal: ttou
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #13636:
URL: https://github.com/apache/airflow/issues/13636
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AdvaithD edited a comment on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
AdvaithD edited a comment on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-827179733
my guess is that it still persists, just ran into something similar, but also suspect my py env getting in the way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] OliverPfau commented on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
OliverPfau commented on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-759262366
I installed airflow 2.0.0 again in a new WSL ubuntu image and the warnings do not appear. I try to find out what the reason is my "real" environment. Probably the change of PYTHONPATH.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-758780708
My guess is that probably you installed airflow in `--editable` mode from sources (pip install -e ".") and this caused 'provider' packages to be installed as well as providers available from sources (or you have airflow sources in the path). That confuses providers manager when combined with PYTHONPATH pointing to airflow sources (adn might behave differently depending on the working directory you are in). .
There are several ways of solving the problem:
1) First of all this is bad idea if you want to run in production. I recommend to install airflow from package rather than from sources if possible.
2) If you sill decide to use --editable mode, you can uninstall all providers (after removing all provider packages (`pip freeze | grep 'apache-airlfow-providers' | xargs pip uninstall -y` should do that ). That should fix the problem.
3) When you install Airflow 2.0.0 you can set `INSTALL_PROVIDERS_FROM_SOURCES` to `true` to avoid installing provider packages in the first place
4) In Airflow 2.0.1 this will be fixed - #13439 , the providers will be automatically removed if you install airflow in editable mode.
I am closing this now, please let me know if my diagnosis if correct and if you managed to solve the problem.
If this is not the case, You might also have a problem that airflow is installed in a different namespace - please let me know more details of your installaton - how you installed airflow, which extras, whether you installed providers, do you have backport providers installed etc. I will reopen it in case we can find a reproducible way of getting there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] OliverPfau commented on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
OliverPfau commented on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-759256659
After uninstalling airflow 1.10.12 I installed airflow 2.0.0 with "pip install apache-airflow" in ubuntu 18.04 in WSL 2 without editable mode. I installed no extras or providers. I also dropped the database before the installation of airflow 2.0.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] AdvaithD commented on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
AdvaithD commented on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-827179733
my gues sis that it still persists, just ran into something similar, but also suspect my py env getting in the way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-758780708
My guess is that probably you installed airflow in `--editable` mode from sources (pip install -e ".") and this caused 'provider' packages to be installed as well as providers available from sources (or you have airflow sources in the path). That confuses providers manager when combined with PYTHONPATH pointing to airflow sources (adn might behave differently depending on the working directory you are in). .
There are several ways of solving the problem:
1) First of all this is bad idea if you want to run in production. I recommend to install airflow from package rather than from sources if possible.
2) If you sill decide to use --editable mode, you can uninstall all providers (after removing all provider packages (`pip freeze | grep 'apache-airlfow-providers' | xargs pip uninstall -y` should do that ). That should fix the problem.
3) When you install Airflow 2.0.0 you can set `INSTALL_PROVIDERS_FROM_SOURCES` to `true` to avoid installing provider packages in the first place
4) In Airflow 2.0.1 this will be fixed - #13439 , the providers will be automatically removed if you install airflow in editable mode.
I am closing this now, please let me know if my diagnosis if correct and if you managed to solve the problem.
If this is not the case, You might also have a problem that airflow is installed in a different namespace - please let me know more details of your installaton - how you installed airflow, which extras, whether you installed providers, do you have backport providers installed etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-758680571
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-759558319
I am looking into similar problem (this time with our prod image) and I will let you know if I find more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] OliverPfau commented on issue #13636: Import warnings from webserver
Posted by GitBox <gi...@apache.org>.
OliverPfau commented on issue #13636:
URL: https://github.com/apache/airflow/issues/13636#issuecomment-759329029
I got rid of the warning by installing airflow 2.0.0 from scratch. Probably I misunderstood the backport stuff. After the installation of airflow 2 I installed the backport providers which installs airflow 1.10.x then. But I am not 100% sure about this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org