You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/12 08:23:39 UTC
[GitHub] [airflow] agrubb86 opened a new issue #18901: Pandas Not Installed with Pip When Required for Providers Packages
agrubb86 opened a new issue #18901:
URL: https://github.com/apache/airflow/issues/18901
### Apache Airflow version
2.2.0 (latest released)
### Operating System
Ubuntu 20.04.2 LTS
### Versions of Apache Airflow Providers
apache-airflow-providers-apache-hive==2.0.2
### Deployment
Virtualenv installation
### Deployment details
Commands used to install:
```
python3.8 -m venv ./airflow-2.2.0
source airflow-2.2.0/bin/activate
pip install --upgrade pip
export AIRFLOW_VERSION=2.2.0
export PYTHON_VERSION="$(python --version | cut -d " " -f 2 | cut -d "." -f 1-2)"
export CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip install wheel
pip install --upgrade "apache-airflow[amazon,apache.druid,apache.hdfs,apache.hive,async,celery,http,jdbc,mysql,password,redis,ssh]==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"
```
### What happened
After installing, Airflow started throwing these errors:
```
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveCliHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveServer2Hook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveMetastoreHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveCliHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveServer2Hook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
WARNI [airflow.providers_manager] Exception when importing 'airflow.providers.apache.hive.hooks.hive.HiveMetastoreHook' from 'apache-airflow-providers-apache-hive' package: No module named 'pandas'
```
### What you expected to happen
I expected required packages (pandas) to be installed with the pip install command
### How to reproduce
Install using the commands in the deployment details.
### Anything else
This seems to be related to pandas being made optional in 2.2.0 but not accounting for providers packages still requiring it.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #18901: Pandas Not Installed with Pip When Required for Providers Packages
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18901:
URL: https://github.com/apache/airflow/issues/18901#issuecomment-940994759
Yep. The workaround is to use `[pandas]` extra that you should add to bring panda back. But yeah - we should likely add `pandas` as requirements for all those providers that use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #18901: Pandas Not Installed with Pip When Required for Providers Packages
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #18901:
URL: https://github.com/apache/airflow/issues/18901#issuecomment-940937938
Confirmed
```
$ find airflow/providers -name '*.py' | xargs grep -i 'import pandas'
airflow/providers/presto/hooks/presto.py: import pandas
airflow/providers/amazon/aws/transfers/mysql_to_s3.py:import pandas as pd
airflow/providers/trino/hooks/trino.py: import pandas
airflow/providers/exasol/hooks/exasol.py:import pandas as pd
airflow/providers/influxdb/hooks/influxdb.py:import pandas as pd
airflow/providers/salesforce/hooks/salesforce.py:import pandas as pd
airflow/providers/apache/hive/hooks/hive.py:import pandas
$ find airflow/providers -name 'provider.yaml' | xargs grep -i 'pandas' | wc -l
0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #18901: Pandas Not Installed with Pip When Required for Providers Packages
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #18901:
URL: https://github.com/apache/airflow/issues/18901
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #18901: Pandas Not Installed with Pip When Required for Providers Packages
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #18901:
URL: https://github.com/apache/airflow/issues/18901#issuecomment-940937938
Confirmed
```
find airflow/providers -name '*.py' | xargs grep -i 'import pandas'
airflow/providers/presto/hooks/presto.py: import pandas
airflow/providers/amazon/aws/transfers/mysql_to_s3.py:import pandas as pd
airflow/providers/trino/hooks/trino.py: import pandas
airflow/providers/exasol/hooks/exasol.py:import pandas as pd
airflow/providers/influxdb/hooks/influxdb.py:import pandas as pd
airflow/providers/salesforce/hooks/salesforce.py:import pandas as pd
airflow/providers/apache/hive/hooks/hive.py:import pandas
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #18901: Pandas Not Installed with Pip When Required for Providers Packages
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #18901:
URL: https://github.com/apache/airflow/issues/18901#issuecomment-940994759
Yep. The workaround is to use `[pandas]` extra that you should add to bring `pandas` back. But yeah - we should likely add `pandas` as requirements for all those providers that use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #18901: Pandas Not Installed with Pip When Required for Providers Packages
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #18901:
URL: https://github.com/apache/airflow/issues/18901#issuecomment-940937938
Confirmed
```
$ find airflow/providers -name '*.py' | xargs grep -i 'import pandas'
airflow/providers/presto/hooks/presto.py: import pandas
airflow/providers/amazon/aws/transfers/mysql_to_s3.py:import pandas as pd
airflow/providers/trino/hooks/trino.py: import pandas
airflow/providers/exasol/hooks/exasol.py:import pandas as pd
airflow/providers/influxdb/hooks/influxdb.py:import pandas as pd
airflow/providers/salesforce/hooks/salesforce.py:import pandas as pd
airflow/providers/apache/hive/hooks/hive.py:import pandas
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org