You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/06 13:43:53 UTC
[GitHub] [airflow] aru-trackunit opened a new issue, #28766: Cannot create connection without defining host using CLI
aru-trackunit opened a new issue, #28766:
URL: https://github.com/apache/airflow/issues/28766
### Apache Airflow version
2.5.0
### What happened
In order to send logs to s3 bucket after finishing the task, I added a connection to airflow using cli.
```airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
Then I got a logging warning saying:
[2023-01-06T13:28:39.585+0000] {logging_mixin.py:137} WARNING - <string>:8 DeprecationWarning: Host s3 specified in the connection is not used. Please, set it on extra['endpoint_url'] instead
Instead I was trying to remove the host from the `conn-uri` I provided but every attempt to create a connection failed (list of my attempts below):
```airflow connections add connection_id_1 --conn-uri aws://?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
```airflow connections add connection_id_1 --conn-uri aws:///?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
### What you think should happen instead
I believe there are 2 options:
1. Allow to create connection without defining host
2. Remove the warning log
### How to reproduce
Create an S3 connection using CLI:
```airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
### Operating System
Linux - official airflow image from docker hub apache/airflow:slim-2.5.0
### Versions of Apache Airflow Providers
```
apache-airflow-providers-cncf-kubernetes | 5.0.0 | Kubernetes
apache-airflow-providers-common-sql | 1.3.1 | Common SQL Provider
apache-airflow-providers-databricks | 4.0.0 | Databricks
apache-airflow-providers-ftp | 3.2.0 | File Transfer Protocol (FTP)
apache-airflow-providers-hashicorp | 3.2.0 | Hashicorp including Hashicorp Vault
apache-airflow-providers-http | 4.1.0 | Hypertext Transfer Protocol (HTTP)
apache-airflow-providers-imap | 3.1.0 | Internet Message Access Protocol (IMAP)
apache-airflow-providers-postgres | 5.3.1 | PostgreSQL
apache-airflow-providers-sqlite | 3.3.1 | SQLite
```
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
This log message is printed every second minute so it is pretty annoying.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28766: Cannot create connection without defining host using CLI
URL: https://github.com/apache/airflow/issues/28766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1374615537
>How to reproduce
Create an S3 connection using CLI:
>
>`airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com`
This is incorrect connection URL, please use [snippet from documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#snippet-to-create-connection-and-convert-to-uri) for generate correct one, you also could check [AWS Connection reference](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#configuring-the-connection) for available values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis closed issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
Taragolis closed issue #28766: Cannot create connection without defining host using CLI
URL: https://github.com/apache/airflow/issues/28766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aru-trackunit commented on issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
aru-trackunit commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375424421
Thank you! Workaround really works, even if you assume that this ticket has very little impact and low priority it's worth mentioning the workaround in the docs or even in the code :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375967447
> Thank you! Workaround really works, even if you assume that this ticket has very little impact and low priority it's worth mentioning the workaround in the docs or even in the code :)
Feel free to add a PR @aru-trackunit . This is an easy way to become a contributor - now that you know how it works, you are likely the best person in the world to know where to add it (and where people like you would look for it). Changing in code - is easy, documentation even easier - just click "Suggest a change on this page" and PR will be opened where you will be able to add the docs - without leaving GitHub UI - so adding a DOC PR is as easy as adding a comment here.
This is how Airflow contributions work - similarly to > 2300 contributors - most of which like you who added a doc they missed or change they felt were missing you can become and Airflow contributor.
Can we count on this please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] euq8 commented on issue #28766: Cannot create connection without defining host using CLI
Posted by "euq8 (via GitHub)" <gi...@apache.org>.
euq8 commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1558778376
@aru-trackunit for `apache-airflow[amazon]==2.6.0` you can add airflow connections for aws s3 using CLI like -
```
airflow connections add "s3_connection" \
--conn-type aws \
--conn-login "${AWS_ACCESS_KEY_ID}" \
--conn-password "${AWS_SECRET_ACCESS_KEY}" \
--conn-extra "{\"region_name\": \"${AWS_REGION}\", \"bucket_name\": \"${AWS_S3_BUCKET_NAME}\"}"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aru-trackunit commented on issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
aru-trackunit commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1377223758
If this is preferred way of solving this issue then sure, let me do it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] aru-trackunit commented on issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
aru-trackunit commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375297770
Thank you for reply @Taragolis . I followed the documentation you sent and generated the `conn_uri` with `get_uri ` method
```
conn=Connection(conn_id="test-s3",conn_type="aws", extra={"endpoint_url":"https://s3.eu-west-1.amazonaws.com", "region_name"="eu-west-1"})
```
The outcome of `conn.get_uri()` method was
`’aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com®ion_name=eu-west-1'`
After following your documentation here https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/logging/s3-task-handler.html
it took me to the connection I wrote above and when I try to execute
```airflow connections add connection_id_1 --conn-uri aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com®ion_name=eu-west-1``` then it’s not accepted by airflow cli method.
However when I add s3 (or whatever string to the uri that corresponds host) then it becomes parseable, however during runtime I get warnings.
The code below works but gives warnings as described above.
```
airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com
```
However I am able to add this connection described above via UI
<img width="1261" alt="Screenshot 2023-01-09 at 09 58 53" src="https://user-images.githubusercontent.com/93520526/211272234-9bae07f7-14d0-43e5-9b6e-55d05059c048.png">
The behaviour seems inconsistent between two ways of inserting credentials.
Maybe I missed something in your reply, but can you elaborate more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #28766: Cannot create connection without defining host using CLI
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375346374
@aru-trackunit Ahhh... I see. I've just check and able to reproduce your problem.
URI `aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com®ion_name=eu-west-1` is valid for Connection.
```python
Python 3.9.10 (main, Feb 25 2022, 16:54:01)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.
PyDev console: using IPython 8.7.0
Python 3.9.10 (main, Feb 25 2022, 16:54:01)
[Clang 13.0.0 (clang-1300.0.29.30)] on darwin
In[2]: from airflow.models.connection import Connection
In[3]: conn = Connection(uri="aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com®ion_name=eu-west-1")
In[4]: conn.host
Out[4]: ''
In[5]: conn.extra_dejson
Out[5]:
{'endpoint_url': 'https://s3.eu-west-1.amazonaws.com',
'region_name': 'eu-west-1'}
```
But unfortunetly it is not valid for cli. So there is actually exists inconsistency between cli `airflow connections add` and `airflow.models.connection.Connection`. Connection URI created by `Connection` could be invalid for cli in case if all fields `host`, `login`, `password`, `port` are empty.
https://github.com/apache/airflow/blob/71306b31f1842ee2b1eb1cc2980b90f0fb6b11dc/airflow/cli/commands/connection_command.py#L133-L138
There is no problem with Amazon Provider as well as Deprecation Warning, however problem exists in core.
As workaround you could add `@` instead of empty `host` for now
```console
❯ airflow connections add connection_id_1 --conn-uri "aws://@/?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com®ion_name=eu-west-1"
Successfully added `conn_id`=connection_id_1 : aws://@/?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com®ion_name=eu-west-1
❯ airflow connections get connection_id_3
id | conn_id | conn_type | description | host | schema | login | password | port | is_encrypted | is_extra_encrypted | extra_dejson | get_uri
===+=================+===========+=============+======+========+=======+==========+======+==============+====================+==========================+===========================
60 | connection_id_1 | aws | None | | | | None | None | False | False | {'endpoint_url': | aws:///?endpoint_url=https
| | | | | | | | | | | 'https://s3.eu-west-1.am | %3A%2F%2Fs3.eu-west-1.amaz
| | | | | | | | | | | azonaws.com', | onaws.com®ion_name=eu-w
| | | | | | | | | | | 'region_name': | est-1
| | | | | | | | | | | 'eu-west-1'} |
```
Reopen Issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org