You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2026/02/20 10:32:12 UTC

[PR] Add support to create connections using uri in SDK [airflow]

amoghrajesh opened a new pull request, #62211:
URL: https://github.com/apache/airflow/pull/62211

    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull request.
   Write a good git commit message following this guide: http://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the documentation.
   
   Feel free to ping (in general) for the review if you do not see reaction for a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
   -->
   
   - [ ] No
   
   <!--
   Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   ## Why?
   
   Happened to notice in https://github.com/apache/airflow/blob/main/providers/apache/livy/tests/unit/apache/livy/hooks/test_livy.py#L91 for example that it was allowed to create a connection in airflow 2.x with `uri` field. Although we have support for it now using `from_uri` this could mean that some DAGs from legacy base could fail due to this.
   
   
   So I am intending to support constructing Connection from a URI now. 
   
   I had to use `__attrs_post_init__` (to avoid `attrs.define(init=False)` and a custom `__init__` to keep the class simple) to parse uri when provided and populate other fields: conn_type, host,
   login, password, port, schema, and extra. This is because: uri is init-only and excluded from serialization (to_dict, from_json) matching the behaviour of models.connection.
   
   
   
   ---
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add support to create connections using uri in SDK [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #62211:
URL: https://github.com/apache/airflow/pull/62211#discussion_r2844855879


##########
task-sdk/src/airflow/sdk/definitions/connection.py:
##########
@@ -119,9 +120,26 @@ class Connection:
     password: str | None = None
     port: int | None = None
     extra: str | None = None
+    uri: str | None = None
 
     EXTRA_KEY = "__extra__"
 
+    _URI_FORBIDDEN_FIELDS = ("conn_type", "host", "login", "password", "schema", "port", "extra")
+
+    def __attrs_post_init__(self) -> None:
+        if self.uri is not None:
+            if any(getattr(self, f) for f in self._URI_FORBIDDEN_FIELDS):
+                raise AirflowException(
+                    "You must create an object using the URI or individual values "
+                    "(conn_type, host, login, password, schema, port or extra). "
+                    "You can't mix these two ways to create this object."
+                )
+            conn_from_uri = self.from_uri(self.uri, conn_id=self.conn_id)
+            for attr in attrs.fields(type(conn_from_uri)):
+                if attr.name != "uri":
+                    object.__setattr__(self, attr.name, getattr(conn_from_uri, attr.name))
+            object.__setattr__(self, "uri", None)

Review Comment:
   Actually thats a very good idea. Let me try it out. 
   
   I went ahead with `__attrs_post_init__` cos I couldn't find any better alternative at the time and was hesitant with the approach too



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add support to create connections using uri in SDK [airflow]

Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #62211:
URL: https://github.com/apache/airflow/pull/62211#discussion_r2845046645


##########
task-sdk/src/airflow/sdk/definitions/connection.py:
##########
@@ -119,9 +120,26 @@ class Connection:
     password: str | None = None
     port: int | None = None
     extra: str | None = None
+    uri: str | None = None
 
     EXTRA_KEY = "__extra__"
 
+    _URI_FORBIDDEN_FIELDS = ("conn_type", "host", "login", "password", "schema", "port", "extra")
+
+    def __attrs_post_init__(self) -> None:
+        if self.uri is not None:
+            if any(getattr(self, f) for f in self._URI_FORBIDDEN_FIELDS):
+                raise AirflowException(
+                    "You must create an object using the URI or individual values "
+                    "(conn_type, host, login, password, schema, port or extra). "
+                    "You can't mix these two ways to create this object."
+                )
+            conn_from_uri = self.from_uri(self.uri, conn_id=self.conn_id)
+            for attr in attrs.fields(type(conn_from_uri)):
+                if attr.name != "uri":
+                    object.__setattr__(self, attr.name, getattr(conn_from_uri, attr.name))
+            object.__setattr__(self, "uri", None)

Review Comment:
   @uranusjr I tried it out and seems there is a problem there. (trying to use `__new__` without an `__init__`
   
   This is what seems to happen:
   
   1. Lets say i did this: `obj = Connection(conn_id="test", uri="postgres://host")`
   2. It would call `__new__` first and eventually it would return a `Connection` class leading to call `__init__` since a new object of Connection is returned
   3. In short I saw this `If __new__() returns an instance of cls, 
   then __init__(instance, ...) will be called with the same arguments.`
   
   
   So essentially this happens:
   ```python
   from airflow.sdk.definitions.connection import Connection
   Connection(conn_id="test_conn", uri="postgres://user:pass@host:5432/db")
   Out[4]: Connection(conn_id='test_conn', conn_type=None, description=None, host=None, schema=None, login=None, password=None, port=None, extra=None, uri='postgres://user:pass@host:5432/db')
   
   ```
   
   `__new__` called from_uri() which returned a Connection with parsed values and called `__init__` again which reset everything. Any suggestions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add support to create connections using uri in SDK [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #62211:
URL: https://github.com/apache/airflow/pull/62211#discussion_r2844788571


##########
task-sdk/src/airflow/sdk/definitions/connection.py:
##########
@@ -119,9 +120,26 @@ class Connection:
     password: str | None = None
     port: int | None = None
     extra: str | None = None
+    uri: str | None = None
 
     EXTRA_KEY = "__extra__"
 
+    _URI_FORBIDDEN_FIELDS = ("conn_type", "host", "login", "password", "schema", "port", "extra")
+
+    def __attrs_post_init__(self) -> None:
+        if self.uri is not None:
+            if any(getattr(self, f) for f in self._URI_FORBIDDEN_FIELDS):
+                raise AirflowException(
+                    "You must create an object using the URI or individual values "
+                    "(conn_type, host, login, password, schema, port or extra). "
+                    "You can't mix these two ways to create this object."
+                )
+            conn_from_uri = self.from_uri(self.uri, conn_id=self.conn_id)
+            for attr in attrs.fields(type(conn_from_uri)):
+                if attr.name != "uri":
+                    object.__setattr__(self, attr.name, getattr(conn_from_uri, attr.name))
+            object.__setattr__(self, "uri", None)

Review Comment:
   I’d prefer using a custom `__init__` with `@overload` instead of `__attrs_post_init__`. That’d make the interface cleaner to the user.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add support to create connections using uri in SDK [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #62211:
URL: https://github.com/apache/airflow/pull/62211#discussion_r2844766474


##########
task-sdk/src/airflow/sdk/definitions/connection.py:
##########
@@ -119,9 +120,26 @@ class Connection:
     password: str | None = None
     port: int | None = None
     extra: str | None = None
+    uri: str | None = None
 
     EXTRA_KEY = "__extra__"
 
+    _URI_FORBIDDEN_FIELDS = ("conn_type", "host", "login", "password", "schema", "port", "extra")
+
+    def __attrs_post_init__(self) -> None:
+        if self.uri is not None:
+            if any(getattr(self, f) for f in self._URI_FORBIDDEN_FIELDS):
+                raise AirflowException(
+                    "You must create an object using the URI or individual values "
+                    "(conn_type, host, login, password, schema, port or extra). "
+                    "You can't mix these two ways to create this object."
+                )
+            conn_from_uri = self.from_uri(self.uri, conn_id=self.conn_id)
+            for attr in attrs.fields(type(conn_from_uri)):
+                if attr.name != "uri":
+                    object.__setattr__(self, attr.name, getattr(conn_from_uri, attr.name))
+            object.__setattr__(self, "uri", None)

Review Comment:
   Does `self.uri = None` not work here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Add support to create connections using uri in SDK [airflow]

Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on code in PR #62211:
URL: https://github.com/apache/airflow/pull/62211#discussion_r2844807340


##########
task-sdk/src/airflow/sdk/definitions/connection.py:
##########
@@ -119,9 +120,26 @@ class Connection:
     password: str | None = None
     port: int | None = None
     extra: str | None = None
+    uri: str | None = None
 
     EXTRA_KEY = "__extra__"
 
+    _URI_FORBIDDEN_FIELDS = ("conn_type", "host", "login", "password", "schema", "port", "extra")
+
+    def __attrs_post_init__(self) -> None:
+        if self.uri is not None:
+            if any(getattr(self, f) for f in self._URI_FORBIDDEN_FIELDS):
+                raise AirflowException(
+                    "You must create an object using the URI or individual values "
+                    "(conn_type, host, login, password, schema, port or extra). "
+                    "You can't mix these two ways to create this object."
+                )
+            conn_from_uri = self.from_uri(self.uri, conn_id=self.conn_id)
+            for attr in attrs.fields(type(conn_from_uri)):
+                if attr.name != "uri":
+                    object.__setattr__(self, attr.name, getattr(conn_from_uri, attr.name))
+            object.__setattr__(self, "uri", None)

Review Comment:
   Actually, maybe we can use `__new__`?
   
   ```python
   # Add @overload signatures...
   
   def __new__(
       cls,
       conn_id: str,
       conn_type: str | None = None,
       description: str | None = None,
       host: str | None = None,
       schema: str | None = None,
       login: str | None = None,
       password: str | None = None,
       port: int | None = None,
       extra: str | None = None,
       *,
       uri: str | None = None,
   ):
       if uri is not None:
           if any((conn_type ...)):
               raise TypeError("uri cannot be used with...")
           return cls.from_uri(uri=uri, conn_id=conn_id)
       return cls(conn_id, conn_type, ...)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org