You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dabla (via GitHub)" <gi...@apache.org> on 2024/07/05 07:35:25 UTC
[PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
dabla opened a new pull request, #40615:
URL: https://github.com/apache/airflow/pull/40615
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: [40609](https://github.com/apache/airflow/issues/40609)
related: [40609](https://github.com/apache/airflow/issues/40609)
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
This PR will fix unnecessary invocations of the get_connection method from the placeholder property.
To achieve this I changed it to cached_property, so on the next invocation the cached result is returned.
I also fixed the warning message within the placeholder property and of course added unit test for both changes.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #40615:
URL: https://github.com/apache/airflow/pull/40615
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666639771
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
Will try to remove that ;)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666612349
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
my comment that I just accidentally noticed that few lines below (just scroll down) below your change the same snippet uses `cast` - which likely might be removed as part of this change - because apparently mypy does not complain any more here (Apparently someone added cast in the past because it did complain for some reason).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #40615:
URL: https://github.com/apache/airflow/pull/40615#issuecomment-2210893494
AH. ... one small static check to fix!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666792851
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
Nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #40615:
URL: https://github.com/apache/airflow/pull/40615#issuecomment-2210466230
Some tests to fix, but it looks good :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666583046
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
I don't understand what you mean by casting, here I just keep the result of that getattr so I can also print it in the warning message if needed. I think this getattr thing is a generic thing as some hooks have different attribute names to define the connection id. Maybe something to cleanup when going to Airflow 3.0. and generalize it across all hooks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666647928
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
> ```python
> def get_conn(self):
> """Return a connection object."""
> db = self.get_connection(getattr(self, cast(str, self.conn_name_attr))) # <--- PROBABLY WE CAN REMOVE THIS CAST
> return self.connector.connect(host=db.host, port=db.port, username=db.login, schema=db.schema)
> ```
I even went a bit further, I replaced all duplicated code regarding this into a property to keep code DRY, I think there were 4 or 5 duplications in the DbApiHook regarding this while one was string casted like you mentioned and the others werent:
```
@property
def conn_id(self) -> str:
return getattr(self, self.conn_name_attr)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666613418
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
```python
def get_conn(self):
"""Return a connection object."""
db = self.get_connection(getattr(self, cast(str, self.conn_name_attr))) # <--- PROBABLY WE CAN REMOVE THIS CAST
return self.connector.connect(host=db.host, port=db.port, username=db.login, schema=db.schema)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666528174
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
One small thing that I noticed - is that for some reason few lines below the `conn_name_attr` is casted to str. I guess it was from some past where mypy has been complaining - maybe we can remove the cast there as well - since it seems it does not complain any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on PR #40615:
URL: https://github.com/apache/airflow/pull/40615#issuecomment-2210780200
> Some tests to fix, but it looks good :)
Some compatibility tests are failing but I don't understand why
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666647928
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
> ```python
> def get_conn(self):
> """Return a connection object."""
> db = self.get_connection(getattr(self, cast(str, self.conn_name_attr))) # <--- PROBABLY WE CAN REMOVE THIS CAST
> return self.connector.connect(host=db.host, port=db.port, username=db.login, schema=db.schema)
> ```
I even went a bit further, I replaced all duplicated code regarding this into a property to keep code DRY, I think there were 4 or 5 duplications in the DbApiHook regarding this while one was string casted like you mentioned and the others werent:
```
def get_conn_id(self) -> str:
return getattr(self, self.conn_name_attr)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666612349
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
my comment that I just accidentally noticed tha tfew lines below (just scroll down) below your change the same snippet uses `cast` - which likely might be removed as part of this change - because apparently mypy does not complain any more here (Apparently someone added cast in the past because it did complain for some reason).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] FIX: DbApiHook.insert_rows unnecessarily restarting connections [airflow]
Posted by "dabla (via GitHub)" <gi...@apache.org>.
dabla commented on code in PR #40615:
URL: https://github.com/apache/airflow/pull/40615#discussion_r1666639243
##########
airflow/providers/common/sql/hooks/sql.py:
##########
@@ -181,19 +182,21 @@ def __init__(self, *args, schema: str | None = None, log_sql: bool = True, **kwa
"replace_statement_format", "REPLACE INTO {} {} VALUES ({})"
)
- @property
+ @cached_property
def placeholder(self):
- conn = self.get_connection(getattr(self, self.conn_name_attr))
+ conn_id = getattr(self, self.conn_name_attr)
Review Comment:
Ow @potiuk now I get it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org