You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/02/12 18:51:00 UTC
[GitHub] [airflow] ccage-simp opened a new issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
ccage-simp opened a new issue #14211:
URL: https://github.com/apache/airflow/issues/14211
**Apache Airflow version**: 1.10.11
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A
**Environment**: Running command on macos.
**What happened**:
I am trying to build a production-like image for our environment using following command:
```
./breeze build-image --production-image --python 2.7 --additional-extras "aws, ssh, vertica" --install-airflow-version="1.10.11"
```
However, I get the following error which seems to be due to pandas==0.25.3 though pandas==0.24.2 I think is the last version that supports 2.7.
Thus it appears to be a problem with the constraints file.
```
ERROR: Could not find a version that satisfies the requirement pandas==0.25.3 (from -c https://raw.githubusercontent.com/apache/airflow/constraints-1.10.11/constraints-2.7.txt (line 206)) (from versions: 0.1, 0.2b0, 0.2b1, 0.2, 0.3.0b0, 0.3.0b2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0rc1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0rc1, 0.8.0rc2, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0rc1, 0.19.0, 0.19.1, 0.19.2, 0.20.0rc1, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0rc1, 0.21.0, 0.21.1, 0.22.0, 0.23.0rc2, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0rc1, 0.24.0, 0.24.1, 0.24.2)
ERROR: No matching distribution found for pandas==0.25.3 (from -c https://raw.githubusercontent.com/apache/airflow/constraints-1.10.11/constraints-2.7.txt (line 206))
```
**What you expected to happen**:
Successful build of image
**How to reproduce it**:
See breeze build-image command in description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778508323
Ah, embarrassed to say I realized that I didn't have latest of master branch so did not have the constraint option yet. Thank you!
Now it starts OK except for strange error about date option:
```
*************** PRODUCTION IMAGE *************************
Python version: 2.7
Additional extras : aws, datadog, hdfs, jdbc, ssh, vertica
Installs version of Airflow: 1.10.11 and skip mounting sources
Constraints location: ./constraints-2.7.txt
Build image
date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Sending build context to Docker daemon 35.99MB
...
```
but ends with :```
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: './constraints-2.7.txt'
```
However, I know `constraints-2.7.txt` file is in same directory because iterm is recognizing the path as valid.
Strange...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778425045
Probably a stupid question, but I haven't figure out how to do that. I was thinking the breeze code is pulling constraints from a URL and haven't noticed a command option to supply a constraints file to breeze command.
Thanks for the workaround idea. That would be great.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-787212557
Hi jarek -
I think I'm going to close this one. I noticed that the constraints-1-10 branch is already updated with the fixes I needed here (` pandas==0.24.2`, etc.). Only difference is I was requesting old 1.10.11 version.
It helped me to learn how to provide my own version of constraints file to breeze command. I think it's good enough for my needs. And now we are closer to upgrading to 1.10.14 with py3 and will then be ready for 2.x.
As for a PR for other updates to breeze scripts to allow using a local constraints file more easily, I do not have good generalized solution yet. If I do create that, I will try to submit my first AF pr.
Thanks for your help!
Chris
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778485044
just download it as a file and use it in `--constraint`. for example: `--constraint ./constraints-python3.6.txt`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778584810
> I am confused why I need this constraints file to install all these packages that we don't use in our production airflow instance. Maybe I should be starting with empty constraints? Or creating our own requirements.txt for packages we actually install in our prod airflow...
Constraints are not requirements. They are simply hint to PIP. They basicaly mean "if you are installing this package, then use this version". No more no less. So it's good to have constraint with ALL possible packages. But then you install only what you need.
> OK, got it working by referencing constraints file from path ./docker-context-files/constraints-2.7.txt which is accessible inside docker.
Yeah. That was exactly supposed to be like this.
> scripts/docker/install_airflow_from_latest_master.sh and scripts/docker/install_airflow.sh having different working path context and yet both try to use relative constraints path. I guess I'll have to modify the scripts for now.
Would be great if you contribute those changes back :). happy to review them :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778387667
Are you sure you want to use python 2.7 ;) ?
On a more serious note, I will take a look tomorrow and fix it manually.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778390984
Thank you for your response, Jarek.
Agree. This is an effort to test our python3 compatibility branch.
Yes, we need to get to python3 so that we can be ready to upgrade to AF 2.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-787422800
Yeah. I think we will only be fixing really critical issues for old 1.10.* and the fix will be to migrate to latest 1.10.* anyway. With the mitigation in place, I think it is as good as it can be.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778382641
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778560569
OK it seems like a little problem with
`scripts/docker/install_airflow_from_latest_master.sh` and `scripts/docker/install_airflow.sh` having different working path context and yet both try to use relative constraints path. I guess I'll have to modify the scripts for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778555198
I am confused why I need this constraints file to install all these packages that we don't use in our production airflow instance. Maybe I should be starting with empty constraints? Or creating our own requirements.txt for packages we actually install in our prod airflow...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778397727
For now as a workaround you can simply download the file locally and modify it manually and use it from there. If you find other problems with it - please let me know.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp edited a comment on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp edited a comment on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778560569
OK it seems like a little problem with
`scripts/docker/install_airflow_from_latest_master.sh` and `scripts/docker/install_airflow.sh` having different working path context and yet both try to use relative constraints path. I guess I'll have to modify the scripts for now.
i.e., in first case we are at root `/` and second case we are at `/opt/airflow/`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp edited a comment on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp edited a comment on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778425045
Probably a stupid question, but I haven't figured out how to do that. I was thinking the breeze code is pulling constraints from a URL and haven't noticed a command option to supply a constraints file to the breeze command.
Thanks for the workaround idea. That would be great.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp edited a comment on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp edited a comment on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778508323
Ah, embarrassed to say I realized that I didn't have latest of master branch so did not have the constraint option yet. Thank you!
Now it starts OK except for strange error about date option:
```
*************** PRODUCTION IMAGE *************************
Python version: 2.7
Additional extras : aws, datadog, hdfs, jdbc, ssh, vertica
Installs version of Airflow: 1.10.11 and skip mounting sources
Constraints location: ./constraints-2.7.txt
Build image
date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Sending build context to Docker daemon 35.99MB
...
```
but ends with :
```
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: './constraints-2.7.txt'
```
However, I know `constraints-2.7.txt` file is in same directory because iterm is recognizing the path as valid.
Strange...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp closed issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp closed issue #14211:
URL: https://github.com/apache/airflow/issues/14211
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp edited a comment on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp edited a comment on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778508323
Ah, embarrassed to say I realized that I didn't have latest of master branch so did not have the constraint option yet. Thank you!
Now it starts OK except for strange error about date option:
```
*************** PRODUCTION IMAGE *************************
Python version: 2.7
Additional extras : aws, datadog, hdfs, jdbc, ssh, vertica
Installs version of Airflow: 1.10.11 and skip mounting sources
Constraints location: ./constraints-2.7.txt
Build image
date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Sending build context to Docker daemon 35.99MB
...
```
but ends with :
```
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: './constraints-2.7.txt'
```
OK, got it working by referencing constraints file from path `./docker-context-files/constraints-2.7.txt` which is accessible inside docker.
- additional compatibility updates:
natsort==6.2.1
numpy==1.16.6
pandas-gbq==0.10.0
Pygments==2.5.2
SQLAlchemy-JSONField==0.8.0
and still testing...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp edited a comment on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp edited a comment on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778508323
Ah, embarrassed to say I realized that I didn't have latest of master branch so did not have the constraint option yet. Thank you!
Now it starts OK except for strange error about date option:
```
*************** PRODUCTION IMAGE *************************
Python version: 2.7
Additional extras : aws, datadog, hdfs, jdbc, ssh, vertica
Installs version of Airflow: 1.10.11 and skip mounting sources
Constraints location: ./constraints-2.7.txt
Build image
date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Sending build context to Docker daemon 35.99MB
...
```
but ends with :
```
ERROR: Could not open requirements file: [Errno 2] No such file or directory: './constraints-2.7.txt'
```
OK, got it working by referencing constraints file from path `./docker-context-files/constraints-2.7.txt` which is accessible inside docker.
- additional compatibility updates:
apispec==2.0.2
cachetools==3.1.1
humanize==1.0.0
natsort==6.2.1
numpy==1.16.6
pandas-gbq==0.10.0
Pygments==2.5.2
rsa==4.0
SQLAlchemy-JSONField==0.8.0
and still testing...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ccage-simp edited a comment on issue #14211: constraints-1.10.11/constraints-2.7.txt has pandas==0.25.3 but python 2.7 support ends at pandas==0.24.2
Posted by GitBox <gi...@apache.org>.
ccage-simp edited a comment on issue #14211:
URL: https://github.com/apache/airflow/issues/14211#issuecomment-778508323
Ah, embarrassed to say I realized that I didn't have latest of master branch so did not have the constraint option yet. Thank you!
Now it starts OK except for strange error about date option:
```
*************** PRODUCTION IMAGE *************************
Python version: 2.7
Additional extras : aws, datadog, hdfs, jdbc, ssh, vertica
Installs version of Airflow: 1.10.11 and skip mounting sources
Constraints location: ./constraints-2.7.txt
Build image
date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Sending build context to Docker daemon 35.99MB
...
```
but ends with :
```
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
ERROR: Could not open requirements file: [Errno 2] No such file or directory: './constraints-2.7.txt'
```
OK, got it working by referencing constraints file from path `./docker-context-files/constraints-2.7.txt` which is accessible inside docker.
- additional compatibility updates:
natsort==6.2.1
pandas-gbq==0.10.0
Pygments==2.5.2
SQLAlchemy-JSONField==0.8.0
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org