You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "prabodh1194 (via GitHub)" <gi...@apache.org> on 2023/08/07 10:47:39 UTC
[GitHub] [airflow] prabodh1194 opened a new issue, #33169: snowflake python connector with secure-local-storage is broken
prabodh1194 opened a new issue, #33169:
URL: https://github.com/apache/airflow/issues/33169
### Apache Airflow version
2.6.3
### What happened
Airflow on python 3.8 defines [constraints](https://raw.githubusercontent.com/apache/airflow/constraints-2.6.3/constraints-3.8.txt) following constraints:
- `keyring==24.2.0`
- `snowflake-connector-python==3.0.4`
However, these constraints are conflicting because `snowflake-connector-python @ v3.0.4` [defines keyring dependency](https://github.com/snowflakedb/snowflake-connector-python/blob/v3.0.4/setup.cfg#L98) on a higher version as follows:
```
secure-local-storage =
keyring!=16.1.0,<24.0.0
```
### What you think should happen instead
_No response_
### How to reproduce
Install airflow v2.6.3 on python 3.8 and then try to `pip install snowflake-connector-python[secure-local-storage]`
### Operating System
MacOs 13.4.1
### Versions of Apache Airflow Providers
```
apache-airflow-providers-snowflake==4.2.0
```
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #33169: snowflake python connector with secure-local-storage is broken
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33169:
URL: https://github.com/apache/airflow/issues/33169#issuecomment-1667643412
https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#upgrading-and-installing-dependencies-including-providers
See the Warning note
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] prabodh1194 commented on issue #33169: snowflake python connector with secure-local-storage is broken
Posted by "prabodh1194 (via GitHub)" <gi...@apache.org>.
prabodh1194 commented on issue #33169:
URL: https://github.com/apache/airflow/issues/33169#issuecomment-1667690421
Fair enough. Just because it never broke till airflow 2.6.1, I raised an issue requesting consistency. Also, it would be great if we could figure out what all versions require `keyring`. I'll more than happy to investigate this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #33169: snowflake python connector with secure-local-storage is broken
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #33169: snowflake python connector with secure-local-storage is broken
URL: https://github.com/apache/airflow/issues/33169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #33169: snowflake python connector with secure-local-storage is broken
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33169:
URL: https://github.com/apache/airflow/issues/33169#issuecomment-1667640627
> pip install snowflake-connector-python[secure-local-storage]
How is this related to airflow constraints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #33169: snowflake python connector with secure-local-storage is broken
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #33169:
URL: https://github.com/apache/airflow/issues/33169#issuecomment-1667628019
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #33169: snowflake python connector with secure-local-storage is broken
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #33169:
URL: https://github.com/apache/airflow/issues/33169#issuecomment-1667720954
This is a misunderstanding on how constraints work. When you read details on constraints, the main (and only reason for constrainst is to provide a reproducible build for airflow and all community providers released at the moment of airflow release). However that does NOT include when you are installing optional dependencies of some of the dependencies of airflow. If they are not included during installation of airflow, one of it's extras or one of it's providers constraints **might** be conflicting. There is no guarantees of any sort.
People also often make the mistake to install anything "new" with constraints on top of airflow. This is wrong. This is now clearly describe in the docuemntation (as Elad pointed out) and I am going to give a whole talk about it at the upcoming Airflow Summit - https://airflowsummit.org/sessions/2023/mastering-dependencies-the-airflow-way/ - recording will be published after that.
Constraints are there to provide reproducible installation of Airflow. Full stop. Constraints is something that you SHOULD use if you are installing airflow with constraints in a clear venv. And this where constraints role ends.
If you are adding anytning on top of it, constraints makes simply no sense. You should use the usual dependency mechanism. Using `snowflake-connector-python[secure-local-storage]` adds a different dependency specified in "secure-local-storage' extra of snowflake-connector-python. And as long it does not conflct with other **requirements** of other packages, this is fine to upgrade or downgrade the version. Ths is PRECISELY why we are using constraints - because the do not limit you in upgrading and downgrading other packages as your OTHER dependencies might need them.
> Also, it would be great if we could figure out what all versions require keyring. I'll more than happy to investigate this.
This is a very good question. But this is a question for `snowflake-connector-python` team, documentation and changelog. If you choose one of our dependencies (`snowflake-connector-python`) and decided to manually install an optional dependency of it (this is is what `[secure-local-storage]` extra does), then you should check whatever the documentation, changelog etc. of `snowflake-connector-python`. Airflow has 650 dependencies (including transitive ones). It would be entirely infisible to bring to airflow documentation or anyhow to our user's attentions every single variation of every single optional extra of every single dependency. This is likely a matrix of likely of a 10s of milions of entries including all combinations. This is why - when you are actually diverging and installing some "other" dependencies than the ones that default airflow installation covers, it's up to the user to make sure that all the dependencies are ok.
So - look at snowflake-connector-python documentation and see what they say. If the documentation is lacking - look at the package content and requirements they have. We cannot do much here in Airflow about all those optonal things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org