You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/09 20:28:06 UTC

[GitHub] [airflow] josh-fell opened a new pull request #20180: Remove db call from `DatabricksHook.__init__()`

josh-fell opened a new pull request #20180:
URL: https://github.com/apache/airflow/pull/20180


   This PR aligns both the original PR to move the db call to the metadata database out of the `DatabricksHook.__init__()` method (#18339) and a refactoring PR for the same hook (#19835).
   
   > FYI @potiuk I did not add Mypy fixes to this file. The number of changes to handle the Mypy errors out-numbered the changes in scope here. I'll have those Mypy updates in a separate PR.
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#discussion_r766150280



##########
File path: airflow/providers/databricks/hooks/databricks.py
##########
@@ -303,6 +299,14 @@ def _do_api_call(self, endpoint_info, json):
         :rtype: dict
         """
         method, endpoint = endpoint_info
+
+        self.databricks_conn = self.get_connection(self.databricks_conn_id)
+
+        if 'host' in self.databricks_conn.extra_dejson:
+            self.host = self._parse_host(self.databricks_conn.extra_dejson['host'])
+        else:
+            self.host = self._parse_host(self.databricks_conn.host)

Review comment:
       ```suggestion
           if self.databricks_conn is None:
               self.databricks_conn = self.get_connection(self.databricks_conn_id)
   
               if 'host' in self.databricks_conn.extra_dejson:
                   self.host = self._parse_host(self.databricks_conn.extra_dejson['host'])
               else:
                   self.host = self._parse_host(self.databricks_conn.host)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] josh-fell commented on pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
josh-fell commented on pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#issuecomment-990262975


   > Nice. I wonder how many of those we will find in other operators..
   > 
   > BTW. for the multi-tenancy work we are planning to add testing the DB method calls and we are planning to run all the tests we have for all the hooks and operators through a test harness tthat will be catching basically any DB operations and flagging those that are "not expected".
   > 
   > I think that might be great opportunity to catch and fix all similar cases too. Would be as easy as to check if there is a db call in any of the `initts` of any of the operators instantiated during tests. Sounds pretty doable.
   
   After seeing this I poked around, there are a good number of instances where, at the very least, `get_connection()` is called in an operator's constructor. I can compile a list and create an issue for folks to tackle. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] josh-fell commented on a change in pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
josh-fell commented on a change in pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#discussion_r766160324



##########
File path: airflow/providers/databricks/hooks/databricks.py
##########
@@ -303,6 +299,14 @@ def _do_api_call(self, endpoint_info, json):
         :rtype: dict
         """
         method, endpoint = endpoint_info
+
+        self.databricks_conn = self.get_connection(self.databricks_conn_id)
+
+        if 'host' in self.databricks_conn.extra_dejson:
+            self.host = self._parse_host(self.databricks_conn.extra_dejson['host'])
+        else:
+            self.host = self._parse_host(self.databricks_conn.host)

Review comment:
       Early access to a Mypy fix. LGTM




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] josh-fell commented on pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
josh-fell commented on pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#issuecomment-990221816


   CC @kaxil 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#issuecomment-990247802


   The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] josh-fell edited a comment on pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
josh-fell edited a comment on pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#issuecomment-990262975


   > Nice. I wonder how many of those we will find in other operators..
   > 
   > BTW. for the multi-tenancy work we are planning to add testing the DB method calls and we are planning to run all the tests we have for all the hooks and operators through a test harness tthat will be catching basically any DB operations and flagging those that are "not expected".
   > 
   > I think that might be great opportunity to catch and fix all similar cases too. Would be as easy as to check if there is a db call in any of the `initts` of any of the operators instantiated during tests. Sounds pretty doable.
   
   After seeing this I poked around. There are a good number of instances where, at the very least, `get_connection()` is called in an operator's constructor as the "db call". Not sure if there are other types of db calls though. I can compile a list and create an issue for folks to tackle. WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#issuecomment-990358017


   > > After seeing this I poked around. There are a good number of instances where, at the very least, `get_connection()` is called in an operator's constructor as the "db call". Not sure if there are other types of db calls though. I can compile a list and create an issue for folks to tackle. WDYT?
   > 
   > Yep. I suspected just that. Until we have an auto-detection of such cases this is a bit of an "uphill battle" (if it slipped through before, it will slip through in the future as well), but yeah i think we could do a one time effort to cleanup it long before we automate it.
   > 
   > Or maybe you could automate that not by running but by what you've done via - essentially - static analysis (either our own or plugin to one of the existing checkers). That would be even better.
   > 
   > We could have done that with pylint plugin but since we all (including myself) hate it, we should figure another way.
   
   Yeah agree, we do instantiate all provider classes somewhere in our test no?? I can’t remember where but that might be a good place to check this 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb merged pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
ashb merged pull request #20180:
URL: https://github.com/apache/airflow/pull/20180


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on pull request #20180: Remove db call from `DatabricksHook.__init__()`

Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #20180:
URL: https://github.com/apache/airflow/pull/20180#issuecomment-990315081


   > After seeing this I poked around. There are a good number of instances where, at the very least, `get_connection()` is called in an operator's constructor as the "db call". Not sure if there are other types of db calls though. I can compile a list and create an issue for folks to tackle. WDYT?
   
   Yep. I suspected just that. Until we have an auto-detection of such cases this is a bit of an "uphill battle" (if it slipped through before, it will slip through in the future as well), but yeah i think we could do a one time effort to cleanup it long before we automate it.
   
   Or maybe you could automate that not by running but by what you've done via - essentially - static analysis (either our own or plugin to one of the existing checkers). That would be even better.
   
   We could have done that with pylint plugin but since we all (including myself) hate it, we should figure another way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org