You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2023/01/06 13:43:53 UTC

[GitHub] [airflow] aru-trackunit opened a new issue, #28766: Cannot create connection without defining host using CLI

aru-trackunit opened a new issue, #28766:
URL: https://github.com/apache/airflow/issues/28766

   ### Apache Airflow version
   
   2.5.0
   
   ### What happened
   
   In order to send logs to s3 bucket after finishing the task, I added a connection to airflow using cli.
   
   ```airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
   
   Then I got a logging warning saying:
   [2023-01-06T13:28:39.585+0000] {logging_mixin.py:137} WARNING - <string>:8 DeprecationWarning: Host s3 specified in the connection is not used. Please, set it on extra['endpoint_url'] instead
   
   Instead I was trying to remove the host from the `conn-uri` I provided but every attempt to create a connection failed (list of my attempts below):
   
   ```airflow connections add connection_id_1 --conn-uri aws://?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
   ```airflow connections add connection_id_1 --conn-uri aws:///?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
   
   ### What you think should happen instead
   
   I believe there are 2 options:
   
   1. Allow to create connection without defining host
   2. Remove the warning log
   
   ### How to reproduce
   
   Create an S3 connection using CLI:
   
   ```airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com```
   
   ### Operating System
   
   Linux -  official airflow image from docker hub apache/airflow:slim-2.5.0
   
   ### Versions of Apache Airflow Providers
   
   ```
   apache-airflow-providers-cncf-kubernetes | 5.0.0 | Kubernetes
   apache-airflow-providers-common-sql | 1.3.1 | Common SQL Provider
   apache-airflow-providers-databricks | 4.0.0 | Databricks
   apache-airflow-providers-ftp | 3.2.0 | File Transfer Protocol (FTP)
   apache-airflow-providers-hashicorp | 3.2.0 | Hashicorp including Hashicorp Vault
   apache-airflow-providers-http | 4.1.0 | Hypertext Transfer Protocol (HTTP)
   apache-airflow-providers-imap | 3.1.0 | Internet Message Access Protocol (IMAP)
   apache-airflow-providers-postgres | 5.3.1 | PostgreSQL
   apache-airflow-providers-sqlite | 3.3.1 | SQLite
   ```
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   This log message is printed every second minute so it is pretty annoying.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #28766: Cannot create connection without defining host using CLI
URL: https://github.com/apache/airflow/issues/28766


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1374615537

   >How to reproduce
   Create an S3 connection using CLI:
   >
   >`airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com`
   
   This is incorrect connection URL, please use [snippet from documentation](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#snippet-to-create-connection-and-convert-to-uri) for generate correct one, you also could check [AWS Connection reference](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#configuring-the-connection) for available values.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis closed issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
Taragolis closed issue #28766: Cannot create connection without defining host using CLI
URL: https://github.com/apache/airflow/issues/28766


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] aru-trackunit commented on issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
aru-trackunit commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375424421

   Thank you! Workaround really works, even if you assume that this ticket has very little impact and low priority it's worth mentioning the workaround in the docs or even in the code :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375967447

   > Thank you! Workaround really works, even if you assume that this ticket has very little impact and low priority it's worth mentioning the workaround in the docs or even in the code :)
   
   Feel free to add a PR @aru-trackunit . This is an easy way to become a contributor - now that you know how it works, you are likely the best person in the world to know where to add it (and where people like you would look for it). Changing in code - is easy, documentation even easier - just click "Suggest a change on this page" and PR will be opened where you will be able to add the docs - without leaving GitHub UI - so adding a DOC PR is as easy as adding a comment here.
   
   This is how Airflow contributions work - similarly to > 2300 contributors - most of which like you who added a doc they missed or change they felt were missing you can become and Airflow contributor.
   
   Can we count on this please?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] euq8 commented on issue #28766: Cannot create connection without defining host using CLI

Posted by "euq8 (via GitHub)" <gi...@apache.org>.
euq8 commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1558778376

   @aru-trackunit for `apache-airflow[amazon]==2.6.0` you can add airflow connections for aws s3 using CLI like -
   ```
   airflow connections add "s3_connection" \
       --conn-type aws \
       --conn-login "${AWS_ACCESS_KEY_ID}" \
       --conn-password "${AWS_SECRET_ACCESS_KEY}" \
       --conn-extra "{\"region_name\": \"${AWS_REGION}\", \"bucket_name\": \"${AWS_S3_BUCKET_NAME}\"}"
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] aru-trackunit commented on issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
aru-trackunit commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1377223758

   If this is preferred way of solving this issue then sure, let me do it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] aru-trackunit commented on issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
aru-trackunit commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375297770

   Thank you for reply @Taragolis . I followed the documentation you sent and generated the `conn_uri` with `get_uri ` method
   ```
   conn=Connection(conn_id="test-s3",conn_type="aws", extra={"endpoint_url":"https://s3.eu-west-1.amazonaws.com", "region_name"="eu-west-1"})
   ```
   The outcome of `conn.get_uri()` method was
   `’aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1'`
   
   After following your documentation here https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/logging/s3-task-handler.html
   it took me to the connection I wrote above and when I try to execute 
   ```airflow connections add connection_id_1 --conn-uri aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1``` then it’s not accepted by airflow cli method.
   
   However when I add s3 (or whatever string to the uri that corresponds host) then it becomes parseable, however during runtime I get warnings.
   
   The code below works but gives warnings as described above.
   ```
   airflow connections add connection_id_1 --conn-uri aws://s3/?region_name=eu-west-1&endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com
   ```
   
   However I am able to add this connection described above via UI
   <img width="1261" alt="Screenshot 2023-01-09 at 09 58 53" src="https://user-images.githubusercontent.com/93520526/211272234-9bae07f7-14d0-43e5-9b6e-55d05059c048.png">
   The behaviour seems inconsistent between two ways of inserting credentials.
   
   Maybe I missed something in your reply, but can you elaborate more?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on issue #28766: Cannot create connection without defining host using CLI

Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #28766:
URL: https://github.com/apache/airflow/issues/28766#issuecomment-1375346374

   @aru-trackunit Ahhh... I see. I've just check and able to reproduce your problem.
   
   
   URI `aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1` is valid for Connection.
   
   ```python
   
   Python 3.9.10 (main, Feb 25 2022, 16:54:01) 
   Type 'copyright', 'credits' or 'license' for more information
   IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.
   PyDev console: using IPython 8.7.0
   Python 3.9.10 (main, Feb 25 2022, 16:54:01) 
   [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
   In[2]: from airflow.models.connection import Connection
   In[3]: conn = Connection(uri="aws:///?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1")
   In[4]: conn.host
   Out[4]: ''
   In[5]: conn.extra_dejson
   Out[5]: 
   {'endpoint_url': 'https://s3.eu-west-1.amazonaws.com',
    'region_name': 'eu-west-1'}
   ```
   
   But unfortunetly it is not valid for cli. So there is actually exists inconsistency between cli `airflow connections add` and `airflow.models.connection.Connection`. Connection URI created by `Connection` could be invalid for cli in case if all fields `host`, `login`, `password`, `port` are empty.
   https://github.com/apache/airflow/blob/71306b31f1842ee2b1eb1cc2980b90f0fb6b11dc/airflow/cli/commands/connection_command.py#L133-L138
   
   There is no problem with Amazon Provider as well as Deprecation Warning, however problem exists in core.
   
   As workaround you could add `@` instead of empty `host` for now
   
   ```console
   ❯ airflow connections add connection_id_1 --conn-uri "aws://@/?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1"
   
   Successfully added `conn_id`=connection_id_1 : aws://@/?endpoint_url=https%3A%2F%2Fs3.eu-west-1.amazonaws.com&region_name=eu-west-1
   
   ❯ airflow connections get connection_id_3
   id | conn_id         | conn_type | description | host | schema | login | password | port | is_encrypted | is_extra_encrypted | extra_dejson             | get_uri                   
   ===+=================+===========+=============+======+========+=======+==========+======+==============+====================+==========================+===========================
   60 | connection_id_1 | aws       | None        |      |        |       | None     | None | False        | False              | {'endpoint_url':         | aws:///?endpoint_url=https
      |                 |           |             |      |        |       |          |      |              |                    | 'https://s3.eu-west-1.am | %3A%2F%2Fs3.eu-west-1.amaz
      |                 |           |             |      |        |       |          |      |              |                    | azonaws.com',            | onaws.com&region_name=eu-w
      |                 |           |             |      |        |       |          |      |              |                    | 'region_name':           | est-1                     
      |                 |           |             |      |        |       |          |      |              |                    | 'eu-west-1'}             |                           
   ``` 
   
   Reopen Issue
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org