You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/26 07:17:55 UTC

[GitHub] [airflow] alexkruc opened a new pull request, #24663: Adding generic SqlToSlackOperator

alexkruc opened a new pull request, #24663:
URL: https://github.com/apache/airflow/pull/24663

   This PR is adding a generic `SqlToSlackOperator`, meaning everything that uses `DbApiHook` and implements `get_pandas_df` will now be able to send the results to Slack.
   As part of this PR, I've also deprecated the current `PrestoToSlackOperator` and `SnowflakeToSlackOperator` - they now inherit from `SqlToSlackOperator`.
    
   closes: #24243
   reference: existing `PrestoToSlackOperator` and `SnowflakeToSlackOperator`, and the generic `SqlToS3Operator`.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal merged pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal merged PR #24663:
URL: https://github.com/apache/airflow/pull/24663


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908533583


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   @potiuk need your advise on this one.
   
   The `hook_params` was added in https://github.com/apache/airflow/pull/18718 /  https://github.com/apache/airflow/pull/19849 which were released on `2.3.0` The Slack provider min version is 2.2.0 so I think this code line should have cause failures?
   
   2.2.0:
   https://github.com/apache/airflow/blob/2.2.0/airflow/models/connection.py#L292
   
   2.3.0:
   https://github.com/apache/airflow/blob/2.3.0/airflow/models/connection.py#L321
   
    I wonder how it make sense that the CI is green?
   
   ~@alexkruc in any case we don't need the `hook_params` here.. the goal is just to check if the hook has the needed function, the user params are not required for that.~
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r906800914


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param slack_conn_id: The connection id for Slack. Mutually exclusive with 'slack_webhook_token'
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'Extra' JSON field.
+         Mutually exclusive with 'slack_conn_id'.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        slack_conn_id: Optional[str] = None,
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,

Review Comment:
   If I'm reading this right you are mixing between kwargs for BaseOperator and also kwargs for the underlying hook (SowflakeHook, PrestoHook, etc..) I don't think this is right.
   We should have a dedicated dict to pass the hook params
   similar to what we have in `SqlSensor`
   
   https://github.com/apache/airflow/blob/37ab8fc76b4d977914ed54fc392b14cdb478d9d1/airflow/sensors/sql.py#L67-L68
   
   To give specific example:
   ```
   SqlToSlackOperator(
       ...,
       retries=3,
       hook_params= {"warehouse": "my_warehouse"}
   )
   ```
    
   Did I miss something?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1166507676

   @eladkal let's keep track here
   
   > The generic operator will handle this - if implemented right there will be no need for individual classes in each provider, again if you have comments on this lets please take them to #24663
   
   I would suggest more generic way, `SqlToSlackOperator` this generic only for DBApi.
   
   If we have something that I mentioned before:  base class in **slack-provider** `BasePandasDataFrameToSlackOperator` without implementation `get_pandas_df`
   
   It could help to build not only `SqlToSlackOperator` but also some user-classes like `AwsDataWranglerDynamoDBToSlack`, `LocalDataFrameToSlack`, etc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909318220


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   Sure - we can modify the pre-commit and ignore lines that for example contain a comment `# ignore 2.2 comp check`. It's a python code that we can modify in this PR. leaving it as an exercise for the reader :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1169271620

   Needs rebase :(


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1166511569

   yes we can, but this is a further enhancement. If alex wants to handle this as well he is more than welcome :) 
   personally I'm OK with leaving it to a followup PR. It's OK to make a change in a few smaller PR.. it makes the review easier


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909501505


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   Great! thanks for the comment :) I changed it here - https://github.com/apache/airflow/pull/24663/commits/efd07a59159b4be94988b174b99af93557fad8ba
   
   > BTW. Pre-comits are SUPER easy :D
   
   I agree! and besides being easy, they are also kinda fun 😄  The thing is, that as a user, and as a young contributor, I'm not often exposed to this, so it's nice learning something new about the Airflow eco-system ;) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1166519554

   > yes we can, but this is a further enhancement. If alex wants to handle this as well he is more than welcome :)
   personally I'm OK with leaving it to a followup PR. It's OK to make a change in a few smaller PR.. it makes the review easier
   
   Totally agree that it should be further enhancement, otherwise it easily took ages to implements SqlToSlackOperator, and deprecate in Snowflake and Presto.
   
   I will look on this PR more close on Monday. Probably I would suggest something that I've already found during SnowflakeToSlack enhancement.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r907324744


##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -15,21 +15,18 @@
 # specific language governing permissions and limitations
 # under the License.
 
-from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+import logging
+import warnings
+from typing import Iterable, Mapping, Optional, Sequence, Union
 
-from pandas import DataFrame
-from tabulate import tabulate
+from airflow.exceptions import AirflowNotFoundException
+from airflow.hooks.base import BaseHook
+from airflow.providers.slack.transfers.sql_to_slack import SqlToSlackOperator
 
-from airflow.exceptions import AirflowException
-from airflow.models import BaseOperator
-from airflow.providers.presto.hooks.presto import PrestoHook
-from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+log = logging.getLogger(__name__)

Review Comment:
   you can use `self.log.info`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1166511088

   Additional benefits you need to test implementation how actually works part with sending DataFrame in `slack-provider` (consumer), and how to produce this DataFrame in actual implementation, like in `SqlToSlackOperator`.
   
   If any bugs appear in slack part, we need need to fix it `slack-provider`, if on producer part, only need to fix on producer.
   Same work with new features.
   
   Again, just an idea and I only talk about benefits without negative part of this idea.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r906771101


##########
tests/providers/presto/transfers/test_presto_to_slack.py:
##########
@@ -41,37 +41,60 @@ def _construct_operator(**kwargs):
         operator = PrestoToSlackOperator(task_id=TEST_DAG_ID, **kwargs)
         return operator
 
-    @mock.patch('airflow.providers.presto.transfers.presto_to_slack.PrestoHook')
-    @mock.patch('airflow.providers.presto.transfers.presto_to_slack.SlackWebhookHook')
-    def test_hooks_and_rendering(self, mock_slack_hook_class, mock_presto_hook_class):
+    @mock.patch('airflow.providers.slack.transfers.sql_to_slack.SlackWebhookHook')
+    def test_hooks_and_rendering_with_slack_conn(self, mock_slack_hook_class):

Review Comment:
   I left the test as is, but changed the mocking location.
   now, the test is verifying that the message that is getting built using the operator is actually triggered from the generic `sql_to_slack.SlackWebhookHook`.
   All the other tests like rendering different df names and such are in `SqlToSlackOperator` tests.. So I think that the tests are pretty minimal as is, and we can delete them and you won't lose any "important" tests, or you will miss some logic..
   My guideline here was end to end test `PrestoToSlackOperator`, and make sure that the actual call is called from `SqlToSlackOperator`.. if we remove `PrestoToSlackOperator` eventually, I see no issue here, as e2e is not important anymore :)
   
   Or am I mistaken here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908533583


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   @potiuk need your advise on this one.
   
   The `hook_params` was added in https://github.com/apache/airflow/pull/18718 /  https://github.com/apache/airflow/pull/19849 which were released on `2.3.0` The Slack provider min version is 2.2.0 so I think this code line should have cause failures?
   
   2.2.0:
   https://github.com/apache/airflow/blob/2.2.0/airflow/models/connection.py#L292
   
   2.3.0:
   https://github.com/apache/airflow/blob/2.3.0/airflow/models/connection.py#L321
   
    I wonder how it make sense that the CI is green?
   
   @alexkruc in any case we don't need the `hook_params` here.. the goal is just to check if the hook has the needed function, the user params are not required for that.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909318220


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   Sure - we can modify the pre-commit and ignore lines that for example contains a comment `# ignore 2.2 comp check`. It's a python code that we can modify in this PR. leaving it as an exercise for the reader :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909361050


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   @potiuk @eladkal added the ignore lines for compatibility check fix, you can find it in this commit:
   https://github.com/apache/airflow/pull/24663/commits/f3a29fb07754e6edaea94ff3e7f81817933958a1
   It should run as part of this PR and we can check that it works e2e :) (although it worked locally)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1166482361

   Rebased to fix selective check problem from #24665 after it's merged.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r906801686


##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -66,14 +63,13 @@ def __init__(
         sql: str,
         slack_message: str,
         presto_conn_id: str = 'presto_default',
-        slack_conn_id: str = 'slack_default',
+        slack_conn_id: Optional[str] = None,

Review Comment:
   Umm, we can, but I'm not sure how should I tackle this..
   The thing is, that using both `slack_conn_id` and `slack_webhook_token`, although supported by the `SlackWbhookHook`, can lead to misunderstandings.
   I added a check for this in `SqlToSlackOperator`, that if both are provided, we go out with an AirflowException.
   If I leave this as is, with the default initialization of the `slack_conn_id`, if a user provides me a `slack_webhook_token`, it will fail (because both are provided - once by the user, and once by the initialization).
   
   To solve this, I can add a check in `SqlToSlackOperator` that checks if `slack_conn_id` is provided, and the connection actually exists, but again it may lead to a confusing flow, as if `slack_conn_id` is provided and exists, it will be the one used by the `SlackWebhookHook`..
   
   I found it to be easier to provide either one, but idk.. WDYT? Should I add a check whether the connection actually exists? or do you might have some other creative suggestion? :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r906922366


##########
docs/apache-airflow-providers-slack/operators/sql_to_slack.rst:
##########
@@ -0,0 +1,38 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _howto/operator:SqlToSlackOperator:
+
+SqlToSlackOperator
+========================
+
+Use the :class:`~airflow.providers.slack.transfers.sql_to_slack` to post messages to predefined Slack
+channels.

Review Comment:
   Oh... 
   
   Slack Incoming Webhook based on [Slack App](https://api.slack.com/messaging/webhooks) can't change channel,  in other word one webhook = one channel
   
   Also exists legacy [Incoming Webhook](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks?tab=more_info) based on Deprecated Slack Integration which can change channel.
   
   The problem was it's unclear what exactly intend to use SlackWebhookHook. Actually it could use both of them, but I don't think we need to focus on legacy features.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908533583


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   @potiuk need your advise on this one.
   
   The `hook_params` was added in https://github.com/apache/airflow/pull/18718 /  https://github.com/apache/airflow/pull/19849 which were released on `2.3.0` The Slack provider min version is 2.2.0 so I think this code line should have cause failures?
   
   2.2.0:
   https://github.com/apache/airflow/blob/2.2.0/airflow/models/connection.py#L292
   
   2.3.0:
   https://github.com/apache/airflow/blob/2.3.0/airflow/models/connection.py#L321
   
    I wonder how it make sense that the CI is green?
   
   @alexkruc in any case we don't need the `hook_params` here.. the goal is just to check if the hook has the needed function, the user params are not needed for that.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908604095


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   A fix to pre-commit to make sure we don't have this case is in https://github.com/apache/airflow/pull/24706
   Thanks @potiuk 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r907334012


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,166 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param slack_conn_id: The connection id for Slack. Mutually exclusive with 'slack_webhook_token'
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'Extra' JSON field.
+         Mutually exclusive with 'slack_conn_id'.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        slack_conn_id: Optional[str] = None,
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,

Review Comment:
   added `sql_hook_params` - thanks for the tip! :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r906798620


##########
tests/providers/presto/transfers/test_presto_to_slack.py:
##########
@@ -41,37 +41,60 @@ def _construct_operator(**kwargs):
         operator = PrestoToSlackOperator(task_id=TEST_DAG_ID, **kwargs)
         return operator
 
-    @mock.patch('airflow.providers.presto.transfers.presto_to_slack.PrestoHook')
-    @mock.patch('airflow.providers.presto.transfers.presto_to_slack.SlackWebhookHook')
-    def test_hooks_and_rendering(self, mock_slack_hook_class, mock_presto_hook_class):
+    @mock.patch('airflow.providers.slack.transfers.sql_to_slack.SlackWebhookHook')
+    def test_hooks_and_rendering_with_slack_conn(self, mock_slack_hook_class):

Review Comment:
   fair point!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r907335993


##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -66,14 +63,13 @@ def __init__(
         sql: str,
         slack_message: str,
         presto_conn_id: str = 'presto_default',
-        slack_conn_id: str = 'slack_default',
+        slack_conn_id: Optional[str] = None,

Review Comment:
   added a check for the actual connection existence in `PrestoToSlackOperator`, and then passing the proper values to the `SqlToslackOperator`..
   LMK if it's ok :) 



##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -15,21 +15,17 @@
 # specific language governing permissions and limitations
 # under the License.
 
+import warnings
 from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
 
-from pandas import DataFrame
-from tabulate import tabulate
-
 from airflow.exceptions import AirflowException
-from airflow.models import BaseOperator
-from airflow.providers.presto.hooks.presto import PrestoHook
-from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+from airflow.providers.slack.transfers.sql_to_slack import SqlToSlackOperator
 
 if TYPE_CHECKING:
-    from airflow.utils.context import Context
+    pass

Review Comment:
   removed :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909471597


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   Cool! That's what I meant. One small NIT: since you made a "generic" exclusion - let's make it `# ignore airflow compat check`, We have some 2.2, some 2.3 and some 2.4 checks already foreseen to be there for quite some time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909472178


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   BTW. Pre-comits are SUPER easy :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908891277


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import warnings
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+from airflow.providers_manager import ProvidersManager
+from airflow.utils.module_loading import import_string
+from airflow.version import version
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+def _backported_get_hook(connection, *, hook_params=None):
+    """Return hook based on conn_type
+     For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed
+     when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Git reference:
+     https://github.com/apache/airflow/blob/main/airflow/providers/slack/provider.yaml#L38
+    """
+    hook = ProvidersManager().hooks.get(connection.conn_type, None)
+
+    if hook is None:
+        raise AirflowException(f'Unknown hook type "{connection.conn_type}"')
+    try:
+        hook_class = import_string(hook.hook_class_name)
+    except ImportError:
+        warnings.warn(
+            "Could not import %s when discovering %s %s",
+            hook.hook_class_name,
+            hook.hook_name,
+            hook.package_name,
+        )
+        raise
+    if hook_params is None:
+        hook_params = {}
+    return hook_class(**{hook.connection_id_attribute_name: connection.conn_id}, **hook_params)
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: Optional[str] = None,
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        if version >= '2.3':
+            # "hook_params" were introduced to into "get_hook()" only in Airflow 2.3.
+            hook = conn.get_hook(hook_params=self.sql_hook_params)
+        else:
+            # For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed
+            # when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Git reference:
+            # https://github.com/apache/airflow/blob/main/airflow/providers/slack/provider.yaml#L38
+            hook = _backported_get_hook(conn)

Review Comment:
   your'e right :) fixed it :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r906798492


##########
docs/apache-airflow-providers-slack/operators/sql_to_slack.rst:
##########
@@ -0,0 +1,38 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+ ..   http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+
+.. _howto/operator:SqlToSlackOperator:
+
+SqlToSlackOperator
+========================
+
+Use the :class:`~airflow.providers.slack.transfers.sql_to_slack` to post messages to predefined Slack
+channels.

Review Comment:
   ```suggestion
   channel.
   ```
   We don't support sending to more than 1 channel do we?



##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -66,14 +63,13 @@ def __init__(
         sql: str,
         slack_message: str,
         presto_conn_id: str = 'presto_default',
-        slack_conn_id: str = 'slack_default',
+        slack_conn_id: Optional[str] = None,

Review Comment:
   this is a breaking change. Can we avoid it?
   I prefer to leave the defaults of the old operators as is. In the new operator we can set new defaults as we like



##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -15,21 +15,17 @@
 # specific language governing permissions and limitations
 # under the License.
 
+import warnings
 from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
 
-from pandas import DataFrame
-from tabulate import tabulate
-
 from airflow.exceptions import AirflowException
-from airflow.models import BaseOperator
-from airflow.providers.presto.hooks.presto import PrestoHook
-from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+from airflow.providers.slack.transfers.sql_to_slack import SqlToSlackOperator
 
 if TYPE_CHECKING:
-    from airflow.utils.context import Context
+    pass

Review Comment:
   can we just remove the `TYPE_CHECKING` then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1166495890

   @alexkruc please check test failures


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908591010


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   @alexkruc I think we have two options here.
   1. backport the `get_hook` function into the operator code (temporarily, we will remove it once the provider will be Airflow>=2.3 compatible)
   2. Leave the `PrestoToSlackOperator` `SnowflakeToSlackOperator` code as is with just a deprecation warning, in that solution we should also check Airflow version when using the operator as it works only with 2.3 (you can check `from airflow.version import version` )
   
   I think option 1 is prefered as otherwise it means only users of Airflow>=2.3 will be able to use this new operator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908591978


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   Yep. 1) is preferred. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909297395


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   @potiuk is there a way to make the new pre commit ignore a specific row?
   The pre-commit fails the PR but @alexkruc checks the user Airflow version and have different code to handle base on version.
   We can use only the backported version if there is no other option



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r907333643


##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -15,21 +15,18 @@
 # specific language governing permissions and limitations
 # under the License.
 
-from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+import logging
+import warnings
+from typing import Iterable, Mapping, Optional, Sequence, Union
 
-from pandas import DataFrame
-from tabulate import tabulate
+from airflow.exceptions import AirflowNotFoundException
+from airflow.hooks.base import BaseHook
+from airflow.providers.slack.transfers.sql_to_slack import SqlToSlackOperator
 
-from airflow.exceptions import AirflowException
-from airflow.models import BaseOperator
-from airflow.providers.presto.hooks.presto import PrestoHook
-from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+log = logging.getLogger(__name__)

Review Comment:
   thanks, changed :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r908882238


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import warnings
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+from airflow.providers_manager import ProvidersManager
+from airflow.utils.module_loading import import_string
+from airflow.version import version
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+def _backported_get_hook(connection, *, hook_params=None):
+    """Return hook based on conn_type
+     For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed
+     when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Git reference:
+     https://github.com/apache/airflow/blob/main/airflow/providers/slack/provider.yaml#L38

Review Comment:
   ```suggestion
        when "apache-airflow-providers-slack" will depend on Airflow >= 2.3.
   ```



##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import warnings
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+from airflow.providers_manager import ProvidersManager
+from airflow.utils.module_loading import import_string
+from airflow.version import version
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+def _backported_get_hook(connection, *, hook_params=None):
+    """Return hook based on conn_type
+     For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed
+     when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Git reference:
+     https://github.com/apache/airflow/blob/main/airflow/providers/slack/provider.yaml#L38
+    """
+    hook = ProvidersManager().hooks.get(connection.conn_type, None)
+
+    if hook is None:
+        raise AirflowException(f'Unknown hook type "{connection.conn_type}"')
+    try:
+        hook_class = import_string(hook.hook_class_name)
+    except ImportError:
+        warnings.warn(
+            "Could not import %s when discovering %s %s",
+            hook.hook_class_name,
+            hook.hook_name,
+            hook.package_name,
+        )
+        raise
+    if hook_params is None:
+        hook_params = {}
+    return hook_class(**{hook.connection_id_attribute_name: connection.conn_id}, **hook_params)
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: Optional[str] = None,
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        if version >= '2.3':
+            # "hook_params" were introduced to into "get_hook()" only in Airflow 2.3.
+            hook = conn.get_hook(hook_params=self.sql_hook_params)
+        else:
+            # For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed
+            # when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Git reference:
+            # https://github.com/apache/airflow/blob/main/airflow/providers/slack/provider.yaml#L38
+            hook = _backported_get_hook(conn)

Review Comment:
   Shouldn't we pass the `sql_hook_params` as well? this is the whole reason for backporing the function isn't it?



##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import warnings
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+from airflow.providers_manager import ProvidersManager
+from airflow.utils.module_loading import import_string
+from airflow.version import version
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+def _backported_get_hook(connection, *, hook_params=None):
+    """Return hook based on conn_type
+     For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed
+     when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Git reference:
+     https://github.com/apache/airflow/blob/main/airflow/providers/slack/provider.yaml#L38
+    """
+    hook = ProvidersManager().hooks.get(connection.conn_type, None)
+
+    if hook is None:
+        raise AirflowException(f'Unknown hook type "{connection.conn_type}"')
+    try:
+        hook_class = import_string(hook.hook_class_name)
+    except ImportError:
+        warnings.warn(
+            "Could not import %s when discovering %s %s",
+            hook.hook_class_name,
+            hook.hook_name,
+            hook.package_name,
+        )
+        raise
+    if hook_params is None:
+        hook_params = {}
+    return hook_class(**{hook.connection_id_attribute_name: connection.conn_id}, **hook_params)
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: Optional[str] = None,
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        if version >= '2.3':
+            # "hook_params" were introduced to into "get_hook()" only in Airflow 2.3.
+            hook = conn.get_hook(hook_params=self.sql_hook_params)
+        else:
+            # For supporting Airflow versions < 2.3, we backport "get_hook()" method. This should be removed
+            # when "apache-airflow-providers-slack" will depend on Airflow >= 2.3. Git reference:
+            # https://github.com/apache/airflow/blob/main/airflow/providers/slack/provider.yaml#L38

Review Comment:
   ```suggestion
               # when "apache-airflow-providers-slack" will depend on Airflow >= 2.3.
   ```
   That description is enough.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eladkal commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r906770142


##########
tests/providers/presto/transfers/test_presto_to_slack.py:
##########
@@ -41,37 +41,60 @@ def _construct_operator(**kwargs):
         operator = PrestoToSlackOperator(task_id=TEST_DAG_ID, **kwargs)
         return operator
 
-    @mock.patch('airflow.providers.presto.transfers.presto_to_slack.PrestoHook')
-    @mock.patch('airflow.providers.presto.transfers.presto_to_slack.SlackWebhookHook')
-    def test_hooks_and_rendering(self, mock_slack_hook_class, mock_presto_hook_class):
+    @mock.patch('airflow.providers.slack.transfers.sql_to_slack.SlackWebhookHook')
+    def test_hooks_and_rendering_with_slack_conn(self, mock_slack_hook_class):

Review Comment:
   This test file should be minimal now.. only the tests that we need to verify that `PrestoToSlackOperator` is successfully "converted" into the new operator. All the other tests logic should be in `test_sql_to_slack.py`
   The reasoning for this is that once we remove `PrestoToSlackOperator` we will also remove it's test file and by doing so we don't want to lose any tests that are important.



##########
tests/system/providers/slack/example_sql_to_slack.py:
##########
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example DAG using SqlToSlackOperator.
+"""
+
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.slack.transfers.sql_to_slack import SqlToSlackOperator
+
+PRESTO_TABLE = os.environ.get("PRESTO_TABLE", "test_table")
+SQL_CONN_ID = 'presto_default'
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = "example_sql_to_slack"
+
+with models.DAG(
+    dag_id=DAG_ID,
+    schedule_interval='@once',  # Override to match your needs
+    start_date=datetime(2022, 1, 1),
+    catchup=False,
+    tags=["example"],
+) as dag:
+    # [START howto_operator_sql_to_slack]
+    SqlToSlackOperator(
+        task_id="presto_to_slack",
+        sql_conn_id=SQL_CONN_ID,
+        sql=f"SELECT col FROM {PRESTO_TABLE}",

Review Comment:
   ```suggestion
           sql=f"SELECT col FROM {SQL_TABLE}",
   ```



##########
tests/providers/snowflake/transfers/test_snowflake_to_slack.py:
##########
@@ -20,6 +20,7 @@
 from airflow.models import DAG
 from airflow.providers.snowflake.transfers.snowflake_to_slack import SnowflakeToSlackOperator
 from airflow.utils import timezone
+from tests.test_utils.config import conf_vars
 from tests.test_utils.db import clear_db_runs
 

Review Comment:
   Same comment like on the presto test file



##########
tests/system/providers/slack/example_sql_to_slack.py:
##########
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Example DAG using SqlToSlackOperator.
+"""
+
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.providers.slack.transfers.sql_to_slack import SqlToSlackOperator
+
+PRESTO_TABLE = os.environ.get("PRESTO_TABLE", "test_table")

Review Comment:
   Lets not use Presto names as it's not longer Presto example :)
   ```suggestion
   SQL_TABLE = os.environ.get("SQL_TABLE", "test_table")
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #24663:
URL: https://github.com/apache/airflow/pull/24663#issuecomment-1166503912

   Just an idea: https://github.com/apache/airflow/pull/24660#issuecomment-1166502250 make sending to slack more generic 
   
   Might be some side effect and multiple inheritance required in some cases or not.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Taragolis commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r907100763


##########
airflow/providers/presto/transfers/presto_to_slack.py:
##########
@@ -47,11 +43,12 @@ class PrestoToSlackOperator(BaseOperator):
         You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
         SQL results
     :param presto_conn_id: destination presto connection
-    :param slack_conn_id: The connection id for Slack
+    :param slack_conn_id: The connection id for Slack. Mutually exclusive with 'slack_token'
     :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
     :param parameters: The parameters to pass to the SQL query
     :param slack_token: The token to use to authenticate to Slack. If this is not provided, the
-        'webhook_token' attribute needs to be specified in the 'Extra' JSON field against the slack_conn_id
+        'webhook_token' attribute needs to be specified in the 'Extra' JSON field against the slack_conn_id.py
+        Mutually exclusive with 'slack_conn_id'

Review Comment:
   `webhook_token` in Extra marked as Deprecated
   
   https://github.com/apache/airflow/blob/b678dc25f6604ddf603e42500e3e5583b6663df6/airflow/providers/slack/hooks/slack_webhook.py#L103-L111



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alexkruc commented on a diff in pull request #24663: Adding generic SqlToSlackOperator

Posted by GitBox <gi...@apache.org>.
alexkruc commented on code in PR #24663:
URL: https://github.com/apache/airflow/pull/24663#discussion_r909329154


##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -0,0 +1,162 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import TYPE_CHECKING, Iterable, Mapping, Optional, Sequence, Union
+
+from pandas import DataFrame
+from tabulate import tabulate
+
+from airflow.exceptions import AirflowException
+from airflow.hooks.base import BaseHook
+from airflow.hooks.dbapi import DbApiHook
+from airflow.models import BaseOperator
+from airflow.providers.slack.hooks.slack_webhook import SlackWebhookHook
+
+if TYPE_CHECKING:
+    from airflow.utils.context import Context
+
+
+class SqlToSlackOperator(BaseOperator):
+    """
+    Executes an SQL statement in a given SQL connection and sends the results to Slack. The results of the
+    query are rendered into the 'slack_message' parameter as a Pandas dataframe using a JINJA variable called
+    '{{ results_df }}'. The 'results_df' variable name can be changed by specifying a different
+    'results_df_name' parameter. The Tabulate library is added to the JINJA environment as a filter to
+    allow the dataframe to be rendered nicely. For example, set 'slack_message' to {{ results_df |
+    tabulate(tablefmt="pretty", headers="keys") }} to send the results to Slack as an ascii rendered table.
+
+    .. seealso::
+        For more information on how to use this operator, take a look at the guide:
+        :ref:`howto/operator:SqlToSlackOperator`
+
+    :param sql: The SQL statement to execute on Snowflake (templated)
+    :param slack_message: The templated Slack message to send with the data returned from Snowflake.
+        You can use the default JINJA variable {{ results_df }} to access the pandas dataframe containing the
+        SQL results
+    :param sql_conn_id: Reference to
+        :ref:`Snowflake connection id<howto/connection:snowflake>`
+    :param sql_hook_params: Extra config params to be passed to the underlying hook.
+           Should match the desired hook constructor params.
+    :param slack_conn_id: The connection id for Slack.
+    :param slack_webhook_token: The token to use to authenticate to Slack. If this is not provided, the
+        'slack_conn_id' attribute needs to be specified in the 'password' field.
+    :param slack_channel: The channel to send message. Override default from Slack connection.
+    :param results_df_name: The name of the JINJA template's dataframe variable, default is 'results_df'
+    :param parameters: The parameters to pass to the SQL query
+    """
+
+    template_fields: Sequence[str] = ('sql', 'slack_message')
+    template_ext: Sequence[str] = ('.sql', '.jinja', '.j2')
+    template_fields_renderers = {"sql": "sql", "slack_message": "jinja"}
+    times_rendered = 0
+
+    def __init__(
+        self,
+        *,
+        sql: str,
+        sql_conn_id: str,
+        sql_hook_params: Optional[dict] = None,
+        slack_conn_id: str = 'slack_default',
+        slack_webhook_token: Optional[str] = None,
+        slack_channel: Optional[str] = None,
+        slack_message: str,
+        results_df_name: str = 'results_df',
+        parameters: Optional[Union[Iterable, Mapping]] = None,
+        **kwargs,
+    ) -> None:
+
+        super().__init__(**kwargs)
+
+        self.sql_conn_id = sql_conn_id
+        self.sql_hook_params = sql_hook_params
+        self.sql = sql
+        self.parameters = parameters
+        self.slack_conn_id = slack_conn_id
+        self.slack_webhook_token = slack_webhook_token
+        self.slack_channel = slack_channel
+        self.slack_message = slack_message
+        self.results_df_name = results_df_name
+        self.kwargs = kwargs
+
+        if not self.slack_conn_id and not self.slack_webhook_token:
+            raise AirflowException(
+                "SqlToSlackOperator requires either a `slack_conn_id` or a `slack_webhook_token` argument"
+            )
+
+    def _get_hook(self) -> DbApiHook:
+        self.log.debug("Get connection for %s", self.sql_conn_id)
+        conn = BaseHook.get_connection(self.sql_conn_id)
+        hook = conn.get_hook(hook_params=self.sql_hook_params)

Review Comment:
   Reader as in myself? ;)
   I'll take a look, will love to get familiar a bit with the pre-commit flow :)
   I'll update here if I'll get stuck or something.
   
   I just thought that checking for versions and matching will be easier to read for future operator maintainers, and also it's easier to deprecate later (just remove the if block), I think that using the generic methods (as in `conn.get_hook(hook_params=....)`) as much as possible is the better way here :) 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org