[GitHub] [airflow] uranusjr commented on a change in pull request #18447: add RedshiftStatementHook, RedshiftOperator

[GitBox <gi...@apache.org>]

uranusjr commented on a change in pull request #18447:
URL: https://github.com/apache/airflow/pull/18447#discussion_r714521670



##########
File path: airflow/providers/amazon/aws/hooks/redshift_statement.py
##########
@@ -0,0 +1,144 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Interact with AWS Redshift, using the boto3 library."""
+
+from typing import Callable, Dict, Optional, Tuple, Union
+
+import redshift_connector
+from redshift_connector import Connection as RedshiftConnection
+
+from airflow.hooks.dbapi import DbApiHook
+
+
+class RedshiftStatementHook(DbApiHook):
+    """
+    Execute statements against Amazon Redshift, using redshift_connector
+
+    This hook requires the redshift_conn_id connection. This connection must
+    be initialized with the host, port, login, password. Additional connection
+    options can be passed to extra as a JSON string.
+
+    :param redshift_conn_id: reference to
+        :ref:`Amazon Redshift connection id<howto/connection:redshift>`
+    :type redshift_conn_id: str
+
+    .. note::
+        get_sqlalchemy_engine() and get_uri() depend on sqlalchemy-amazon-redshift
+    """
+
+    conn_name_attr = 'redshift_conn_id'
+    default_conn_name = 'redshift_default'
+    conn_type = 'redshift'
+    hook_name = 'Amazon Redshift'
+    supports_autocommit = True
+
+    @staticmethod
+    def get_ui_field_behavior() -> Dict:
+        """Returns custom field behavior"""
+        return {
+            "hidden_fields": [],
+            "relabeling": {'login': 'User', 'schema': 'Database'},
+        }
+
+    def __init__(self, *args, **kwargs) -> None:
+        super().__init__(*args, **kwargs)
+
+    def _get_conn_params(self) -> Tuple[Dict[str, Union[str, int]], Dict]:
+        """Helper method to retrieve connection args and kwargs"""
+        conn = self.get_connection(
+            self.redshift_conn_id  # type: ignore[attr-defined]  # pylint: disable=no-member
+        )
+
+        conn_params: Dict[str, Union[str, int]] = {
+            "user": conn.login or '',
+            "password": conn.password or '',
+            "host": conn.host or '',
+            "port": conn.port or 5439,
+            "database": conn.extra_dejson.pop('database', ''),
+        }
+
+        conn_kwargs: Dict = {}
+
+        for param, arg in conn.extra_dejson.items():
+            conn_kwargs[param] = arg
+
+        return conn_params, conn_kwargs
+
+    def get_uri(self) -> str:
+        """
+        Override DbApiHook get_uri method for get_sqlalchemy_engine()
+
+        .. note::
+            Value passed to connection extra parameter will be excluded
+            from returned uri but passed to get_sqlalchemy_engine()
+            by default
+        """
+        from sqlalchemy.engine.url import URL
+
+        conn_params, _ = self._get_conn_params()
+        conn_params['username'] = conn_params['user']
+        del conn_params['user']
+
+        return URL(drivername='redshift+redshift_connector', **conn_params).__to_string__(hide_password=False)
+
+    def get_sqlalchemy_engine(self, engine_kwargs=None):
+        """Overrides DbApiHook get_sqlalchemy_engine to pass redshift_connector specific kwargs"""
+        _, conn_kwargs = self._get_conn_params()
+        if engine_kwargs is None:
+            engine_kwargs = {}

Review comment:
       [Do not set function argument default to a mutable value](https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org