You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/21 14:45:40 UTC
[GitHub] [airflow] Taragolis opened a new pull request, #25852: Refactor Slack API Hook and add new Connection type
Taragolis opened a new pull request, #25852:
URL: https://github.com/apache/airflow/pull/25852
### Refactor Slack API hook
1. Retrieve token and create client outside of `__init__` method
2. Create new connection type. Right now only Slack Webhook connection exists.
3. Add test method to connection.
4. Customise Slack API Connection UI.
5. Show deprecation warning if user provide `token` in Hook argument. IMHO, there is not a good idea provide directly token better to use Connection.
6. Keep compatible with generic connection.
### Connection UI and Testing Connection



--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #25852: Refactor Slack API Hook and add new Connection type
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #25852:
URL: https://github.com/apache/airflow/pull/25852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #25852: Refactor Slack API Hook and add new Connection type
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #25852:
URL: https://github.com/apache/airflow/pull/25852#discussion_r950881054
##########
docs/apache-airflow-providers-slack/index.rst:
##########
@@ -42,9 +42,9 @@ Content
:maxdepth: 1
:caption: References
- Connection Types <connections/slack>
+ Connection Types <connections/index>
Python API <_api/airflow/providers/slack/index>
- Example DAGs <https://github.com/apache/airflow/tree/providers-slack/5.0.0/tests/system/providers/slack>
+ Example DAGs <https://github.com/apache/airflow/tree/providers-slack/5.1.0/tests/system/providers/slack>
Review Comment:
https://github.com/apache/airflow/tree/providers-slack/5.0.0/tests/system/providers/slack return 404
However might be better change to main branch rather than specific tag?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #25852: Refactor Slack API Hook and add new Connection type
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #25852:
URL: https://github.com/apache/airflow/pull/25852#discussion_r951115925
##########
airflow/providers/slack/hooks/slack.py:
##########
@@ -45,28 +65,124 @@ class SlackHook(BaseHook):
# For more details check https://slack.dev/python-slack-sdk/web/index.html#messaging
slack_hook.client.chat_postMessage(channel="#random", text="Hello world!")
- :param token: Slack API token
:param slack_conn_id: :ref:`Slack connection id <howto/connection:slack>`
that has Slack API token in the password field.
- :param use_session: A boolean specifying if the client should take advantage of
- connection pooling. Default is True.
- :param base_url: A string representing the Slack API base URL. Default is
- ``https://www.slack.com/api/``
- :param timeout: The maximum number of seconds the client will wait
- to connect and receive a response from Slack. Default is 30 seconds.
+ :param timeout: The maximum number of seconds the client will wait to connect
+ and receive a response from Slack. If not set than default WebClient value will use.
+ :param base_url: A string representing the Slack API base URL.
+ If not set than default WebClient BASE_URL will use (``https://www.slack.com/api/``).
+ :param proxy: Proxy to make the Slack Incoming Webhook call.
+ :param retry_handlers: List of handlers to customize retry logic in WebClient.
+ :param token: (deprecated) Slack API Token.
"""
+ conn_name_attr = 'slack_conn_id'
+ default_conn_name = 'slack_api_default'
+ conn_type = 'slack'
+ hook_name = 'Slack API'
+
def __init__(
self,
token: Optional[str] = None,
slack_conn_id: Optional[str] = None,
- **client_args: Any,
+ base_url: Optional[str] = None,
+ timeout: Optional[int] = None,
+ proxy: Optional[str] = None,
+ retry_handlers: Optional[List["RetryHandler"]] = None,
+ **extra_client_args: Any,
) -> None:
+ if not token and not slack_conn_id:
+ raise AirflowException("Either `slack_conn_id` or `token` should be provided.")
+
super().__init__()
- self.token = self.__get_token(token, slack_conn_id)
- self.client = WebClient(self.token, **client_args)
+ self.slack_conn_id = slack_conn_id
+ if not self.slack_conn_id:
+ warnings.warn(
+ "You have not set parameter `slack_conn_id`. Currently `Slack API` connection id optional "
+ "but in a future release it will mandatory.",
+ FutureWarning,
+ stacklevel=2,
+ )
+ self.base_url = base_url
+ self.timeout = timeout
+ self.proxy = proxy
+ self.retry_handlers = retry_handlers
+ self.extra_client_args = extra_client_args
+ if self.extra_client_args.pop("use_session", None) is not None:
+ warnings.warn("`use_session` has no affect in slack_sdk.WebClient.", UserWarning, stacklevel=2)
+
+ self._token = token
+ if self._token:
+ warnings.warn(
+ "Provide token as hook argument deprecated by security reason and will be removed "
+ "in a future releases. Please specify token in `Slack API` connection.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+
+ @cached_property
+ def client(self) -> WebClient:
+ """Get the underlying slack_sdk.WebClient (cached)."""
+ return WebClient(**self._get_conn_params())
+
+ def get_conn(self) -> WebClient:
+ """Get the underlying slack_sdk.WebClient (cached)."""
+ return self.client
+
+ def _get_conn_params(self) -> Dict[str, Any]:
+ """Fetch connection params as a dict and merge it with hook parameters."""
+ conn = self.get_connection(self.slack_conn_id) if self.slack_conn_id else None
+ conn_params: Dict[str, Any] = {}
+
+ if self._token:
+ conn_params["token"] = self._token
+ elif conn:
+ if not conn.password:
+ raise AirflowNotFoundException(
+ f"Connection ID {self.slack_conn_id!r} does not contain password (Slack API Token)."
+ )
+ conn_params["token"] = conn.password
+
+ extra_config = ConnectionExtraConfig(
+ conn_type=self.conn_type,
+ conn_id=conn.conn_id if conn else None,
+ extra=conn.extra_dejson if conn else {},
+ )
+
+ # Merge Hook parameters with Connection config
+ conn_params.update(
+ {
+ "timeout": self.timeout or extra_config.getint("timeout", default=None),
+ "base_url": self.base_url or extra_config.get("base_url", default=None),
+ "proxy": self.proxy or extra_config.get("proxy", default=None),
+ "retry_handlers": (
+ self.retry_handlers or extra_config.getimports("retry_handlers", default=None)
+ ),
+ }
+ )
+
+ # Add additional client args
+ conn_params.update(self.extra_client_args)
+ if "logger" not in conn_params:
+ conn_params["logger"] = self.log
+
+ return {k: v for k, v in conn_params.items() if v is not None}
+
+ @cached_property
+ def token(self) -> str:
+ warnings.warn(
+ "`SlackHook.token` property deprecated and will be removed in a future releases.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
+ return self._get_conn_params()["token"]
def __get_token(self, token: Any, slack_conn_id: Any) -> str:
+ warnings.warn(
+ "`SlackHook.__get_token` method deprecated and will be removed in a future releases.",
+ DeprecationWarning,
+ stacklevel=2,
+ )
Review Comment:
Also might be better just remove this method.
It not used after changes and it is hardly possible that someone override it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #25852: Refactor Slack API Hook and add new Connection type
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #25852:
URL: https://github.com/apache/airflow/pull/25852#discussion_r951204676
##########
airflow/providers/slack/utils/__init__.py:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict, Optional
+
+from airflow.utils.module_loading import import_string
+
+try:
+ from airflow.utils.types import NOTSET
+except ImportError: # TODO: Remove when the provider has an Airflow 2.3+ requirement.
+
+ class ArgNotSet:
+ """Sentinel type for annotations, useful when None is not viable."""
+
+ NOTSET = ArgNotSet() # type: ignore[assignment]
+
+
+def prefixed_extra_field(field: str, conn_type: str) -> str:
+ """Get prefixed extra field name."""
+ return f"extra__{conn_type}__{field}"
+
+
+class ConnectionExtraConfig:
+ """Helper class for rom Connection Extra.
+
+ :param conn_type: Hook connection type.
+ :param conn_id: Connection ID uses for appropriate error messages.
+ :param extra: Connection extra dictionary.
+ """
Review Comment:
I create separate class for use in further steps
1. Refactor `airflow.providers.slack.hooks.slack_webhook.SlackWebhookHook` which mostly designed for use with legacy [Slack Integrations](https://taragolisworkspace.slack.com/apps/A0F7XDUAZ-incoming-webhooks?tab=more_info)
2. Create new hook and connection which will use `slack_sdk.webhook.WebhookClient` and create integration with Slack Incoming Webhook based on Slack API/App
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #25852: Refactor Slack API Hook and add new Connection type
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #25852:
URL: https://github.com/apache/airflow/pull/25852#discussion_r959612731
##########
docs/apache-airflow-providers-slack/index.rst:
##########
@@ -42,9 +42,9 @@ Content
:maxdepth: 1
:caption: References
- Connection Types <connections/slack>
+ Connection Types <connections/index>
Python API <_api/airflow/providers/slack/index>
- Example DAGs <https://github.com/apache/airflow/tree/providers-slack/5.0.0/tests/system/providers/slack>
+ Example DAGs <https://github.com/apache/airflow/tree/providers-slack/5.1.0/tests/system/providers/slack>
Review Comment:
This number is updated automatically when we release - this way every doc in the index refers to the latest released version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #25852: Refactor Slack API Hook and add new Connection type
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #25852:
URL: https://github.com/apache/airflow/pull/25852#discussion_r950880873
##########
airflow/providers/slack/hooks/slack.py:
##########
@@ -15,27 +15,47 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-"""Hook for Slack"""
-from typing import Any, Optional
+
+import warnings
+from typing import TYPE_CHECKING, Any, Dict, List, Optional
from slack_sdk import WebClient
-from slack_sdk.web.slack_response import SlackResponse
+from slack_sdk.errors import SlackApiError
-from airflow.exceptions import AirflowException
+from airflow.compat.functools import cached_property
+from airflow.exceptions import AirflowException, AirflowNotFoundException
from airflow.hooks.base import BaseHook
+from airflow.providers.slack.utils import ConnectionExtraConfig, prefixed_extra_field
+from airflow.settings import json
+
+if TYPE_CHECKING:
+ from slack_sdk.http_retry import RetryHandler
+ from slack_sdk.web.slack_response import SlackResponse
class SlackHook(BaseHook):
"""
- Creates a Slack connection to be used for calls.
+ Creates a Slack API Connection to be used for calls.
+
+ This class provide a thin wrapper around the ``slack_sdk.WebClient``.
+
+ .. seealso::
+ - :ref:`Slack API connection <howto/connection:slack>`
+ - https://api.slack.com/messaging
+ - https://slack.dev/python-slack-sdk/web/index.html
+
+ .. warning::
+ This hook intend to use `Slack API` connection
+ and might not work correctly with `Slack Webhook` and `HTTP` connections.
Takes both Slack API token directly and connection that has Slack API token. If both are
supplied, Slack API token will be used. Also exposes the rest of slack.WebClient args.
+
Examples:
- .. code-block:: python
+ .. code-block:: python
Review Comment:
This additional indent required to proper formatted block in documentation and PyCharm
**Without additional space**
https://airflow.apache.org/docs/apache-airflow-providers-slack/stable/_api/airflow/providers/slack/hooks/slack/index.html#airflow.providers.slack.hooks.slack.SlackHook
**With additional space**

Might be possible avoid this "hack"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #25852: Refactor Slack API Hook and add new Connection type
Posted by GitBox <gi...@apache.org>.
Taragolis commented on code in PR #25852:
URL: https://github.com/apache/airflow/pull/25852#discussion_r951204676
##########
airflow/providers/slack/utils/__init__.py:
##########
@@ -0,0 +1,98 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, Dict, Optional
+
+from airflow.utils.module_loading import import_string
+
+try:
+ from airflow.utils.types import NOTSET
+except ImportError: # TODO: Remove when the provider has an Airflow 2.3+ requirement.
+
+ class ArgNotSet:
+ """Sentinel type for annotations, useful when None is not viable."""
+
+ NOTSET = ArgNotSet() # type: ignore[assignment]
+
+
+def prefixed_extra_field(field: str, conn_type: str) -> str:
+ """Get prefixed extra field name."""
+ return f"extra__{conn_type}__{field}"
+
+
+class ConnectionExtraConfig:
+ """Helper class for rom Connection Extra.
+
+ :param conn_type: Hook connection type.
+ :param conn_id: Connection ID uses for appropriate error messages.
+ :param extra: Connection extra dictionary.
+ """
Review Comment:
I create separate class for use in further steps
1. Refactor `airflow.providers.slack.hooks.slack_webhook.SlackWebhookHook` which mostly designed for use with legacy [Slack Integrations](https://taragolisworkspace.slack.com/apps/A0F7XDUAZ-incoming-webhooks?tab=more_info)
2. Create new hook and connection which will use `slack_sdk.webhook.WebhookClient` and create integration with Slack Incoming Webhook based on [Slack API/App](https://api.slack.com/messaging/webhooks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org