You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "andyguwc (via GitHub)" <gi...@apache.org> on 2024/03/12 16:21:33 UTC
[PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
andyguwc opened a new pull request, #38079:
URL: https://github.com/apache/airflow/pull/38079
This PR adds a new param to the SqlToSlackApiFileOperator to optionally check for null output. It raises an exception instead of sending a slack with an empty file.
closes: https://github.com/apache/airflow/issues/37191
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis merged PR #38079:
URL: https://github.com/apache/airflow/pull/38079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "andyguwc (via GitHub)" <gi...@apache.org>.
andyguwc commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1523617708
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
good suggestions - I updated the param with the following possible values. This maps 1:1 with the task state. Should be straightforward enough for the end user to understand.
```
:param action_on_empty_df: Specifying how to handle an empty sql output df. Possible values:
- ``send``: (default) send the slack with an empty file.
- ``skip``: skip sending the slack message. Task state set to "skipped".
- ``error``: raise an error to fail the task. Task state set to "failed".
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1523006444
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
Naming of literal values could be different
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "andyguwc (via GitHub)" <gi...@apache.org>.
andyguwc commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1522216852
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
also @eladkal curious about your thoughts on the behavior of skipping sending the slack message instead of raising an Exception to fail the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "tianyou-gu (via GitHub)" <gi...@apache.org>.
tianyou-gu commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1521885687
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -33,6 +33,10 @@
from airflow.utils.context import Context
+class SqlToSlackNullOutputException(AirflowException):
+ """An exception that indicates the sql output is empty and hence should not be sent with slack message."""
Review Comment:
I think so, it's not necessarily a parameter input. But also, based on comment below, won't need this Exception, just skipping instead of failing is the desired behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1521796637
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -58,6 +62,7 @@ class SqlToSlackApiFileOperator(BaseSqlToSlackOperator):
:param slack_base_url: A string representing the Slack API base URL. Optional
:param slack_method_version: The version of the Slack SDK Client method to be used, either "v1" or "v2".
:param df_kwargs: Keyword arguments forwarded to ``pandas.DataFrame.to_{format}()`` method.
+ :param allow_null: Keyword argument to allow null sql output which implies allowing sending an empty file.
Review Comment:
```suggestion
:param send_file_if_empty: True to allow send file if sql statemant return no reaults. Otherwise False
```
I think it's easier to understand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1523099668
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
`action_on_empty_df: Literal["send", "skip", "error", "ignore"] = "send"`
That mixes between what what to do with the empty file and what state to set the task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1521824259
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -33,6 +33,10 @@
from airflow.utils.context import Context
+class SqlToSlackNullOutputException(AirflowException):
+ """An exception that indicates the sql output is empty and hence should not be sent with slack message."""
Review Comment:
Do we need separate exception for that? Or we just could use ValueError instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "dirrao (via GitHub)" <gi...@apache.org>.
dirrao commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1521779196
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
Raising exception cause the task to fail. Instead of that, can we skip sending the slack message and keep the task status success?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1525534431
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +143,15 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if df_result.empty:
+ if self.action_on_empty_df == "skip":
+ raise AirflowSkipException("Sql output df is empty. Skipping.")
+ elif self.action_on_empty_df == "error":
+ raise ValueError("Sql output df must be non-empty. Failing.")
+ elif self.action_on_empty_df == "send":
+ pass
+ else:
Review Comment:
```suggestion
elif self.action_on_empty_df != "send":
```
##########
tests/providers/slack/transfers/test_sql_to_slack.py:
##########
@@ -157,6 +157,73 @@ def test_unsupported_format(self, filename):
with pytest.raises(ValueError):
op.execute(mock.MagicMock())
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_sending_empty_file_by_default(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ op.execute(mock.MagicMock)
+ mock_slack_hook_cls.assert_called_once()
+
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_skip_sending_file(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ "action_on_empty_df": "skip",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ with pytest.raises(AirflowSkipException):
+ op.execute(mock.MagicMock())
+ mock_slack_hook_cls.assert_not_called()
Review Comment:
```suggestion
op.execute(mock.MagicMock())
mock_slack_hook_cls.assert_not_called()
```
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +143,15 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if df_result.empty:
+ if self.action_on_empty_df == "skip":
+ raise AirflowSkipException("Sql output df is empty. Skipping.")
+ elif self.action_on_empty_df == "error":
+ raise ValueError("Sql output df must be non-empty. Failing.")
Review Comment:
```suggestion
raise ValueError("SQL output df must be non-empty. Failing.")
```
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +143,15 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if df_result.empty:
+ if self.action_on_empty_df == "skip":
+ raise AirflowSkipException("Sql output df is empty. Skipping.")
Review Comment:
```suggestion
raise AirflowSkipException("SQL output df is empty. Skipping.")
```
##########
tests/providers/slack/transfers/test_sql_to_slack.py:
##########
@@ -157,6 +157,73 @@ def test_unsupported_format(self, filename):
with pytest.raises(ValueError):
op.execute(mock.MagicMock())
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_sending_empty_file_by_default(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ op.execute(mock.MagicMock)
+ mock_slack_hook_cls.assert_called_once()
+
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_skip_sending_file(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ "action_on_empty_df": "skip",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ with pytest.raises(AirflowSkipException):
+ op.execute(mock.MagicMock())
+ mock_slack_hook_cls.assert_not_called()
+
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_raise_error(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ "action_on_empty_df": "error",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ with pytest.raises(ValueError):
Review Comment:
```suggestion
with pytest.raises(ValueError, match="output df must be non-empty\. Failing"):
```
##########
tests/providers/slack/transfers/test_sql_to_slack.py:
##########
@@ -157,6 +157,73 @@ def test_unsupported_format(self, filename):
with pytest.raises(ValueError):
op.execute(mock.MagicMock())
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_sending_empty_file_by_default(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ op.execute(mock.MagicMock)
+ mock_slack_hook_cls.assert_called_once()
+
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_skip_sending_file(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ "action_on_empty_df": "skip",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ with pytest.raises(AirflowSkipException):
+ op.execute(mock.MagicMock())
+ mock_slack_hook_cls.assert_not_called()
+
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.SlackHook")
+ @mock.patch("airflow.providers.slack.transfers.sql_to_slack.BaseSqlToSlackOperator._get_query_results")
+ def test_null_output_raise_error(self, mock_get_query_results, mock_slack_hook_cls):
+ op_kwargs = {
+ **self.default_op_kwargs,
+ "slack_conn_id": "expected-test-slack-conn-id",
+ "slack_filename": "test_filename.csv",
+ "slack_channels": ["#random"],
+ "slack_initial_comment": "test_comment",
+ "slack_title": "test_title",
+ "action_on_empty_df": "error",
+ }
+ op = SqlToSlackApiFileOperator(task_id="test_send_file", **op_kwargs)
+
+ # Mock empty query results
+ mock_df = mock.MagicMock()
+ mock_df.configure_mock(**{"empty.return_value": True})
+ mock_get_query_results.return_value = mock_df
+
+ with pytest.raises(ValueError):
+ op.execute(mock.MagicMock())
+ mock_slack_hook_cls.assert_not_called()
Review Comment:
```suggestion
mock_slack_hook_cls.assert_not_called()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1521796637
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -58,6 +62,7 @@ class SqlToSlackApiFileOperator(BaseSqlToSlackOperator):
:param slack_base_url: A string representing the Slack API base URL. Optional
:param slack_method_version: The version of the Slack SDK Client method to be used, either "v1" or "v2".
:param df_kwargs: Keyword arguments forwarded to ``pandas.DataFrame.to_{format}()`` method.
+ :param allow_null: Keyword argument to allow null sql output which implies allowing sending an empty file.
Review Comment:
```suggestion
:param send_file_if_empty: Keyword argument to allow null sql output which implies allowing sending an empty file.
```
I think it's easier to understand
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "andyguwc (via GitHub)" <gi...@apache.org>.
andyguwc commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1521958643
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
@dirrao updated, can you take another look? Thanks!
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
@dirrao updated, can you take another look? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "tianyou-gu (via GitHub)" <gi...@apache.org>.
tianyou-gu commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1521885687
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -33,6 +33,10 @@
from airflow.utils.context import Context
+class SqlToSlackNullOutputException(AirflowException):
+ """An exception that indicates the sql output is empty and hence should not be sent with slack message."""
Review Comment:
I think so, it's not necessarily a parameter input. But also, based on comment below, won't need this Exception, just skipping instead of failing is the desired behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1522580067
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
I think it makes sense but in this case probably we need another rename. Something like `set_skip_on_empty_result`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Update SqlToSlackApiFileOperator with new param to check empty output [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #38079:
URL: https://github.com/apache/airflow/pull/38079#discussion_r1523002335
##########
airflow/providers/slack/transfers/sql_to_slack.py:
##########
@@ -134,6 +141,8 @@ def execute(self, context: Context) -> None:
output_file_name = fp.name
output_file_format = output_file_format.upper()
df_result = self._get_query_results()
+ if not self.allow_null and df_result.empty:
+ raise SqlToSlackNullOutputException("Sql output is empty.")
Review Comment:
If we would like to have multiple options we might make it as enum or literal string because action non-binary.
I would prefer literal, it just a string which wouldn't have any problem with serde.
```python
from typing_extensions import Literal
...
action_on_empty_df: Literal["send", "skip", "error", "ignore"] = "send"
```
- `send` - send even empty
- `skip` - raise AirflowSkipException, and task will skiped
- `error` - raise an error
- `ignore` - do nothing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org