You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2022/06/17 00:58:17 UTC

[GitHub] [superset] zhaorui2022 opened a new pull request, #20419: fix(embedded) Retry when executing alert queries to avoid sending transient errors to users as alert failure notifications

zhaorui2022 opened a new pull request, #20419:
URL: https://github.com/apache/superset/pull/20419

   fix(embedded) Retry when executing alert queries to avoid sending transient errors to users as alert failure notifications
   
   
   ### SUMMARY
   Details documented at https://github.com/apache/superset/issues/20418
   
   When there are transient errors, alert owners will receive a notification saying the alert fails even though the underlying dataset is working properly. Adding retries to avoid users receiving redundant notifications.
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   N/A
   
   ### TESTING INSTRUCTIONS
   This only happens when there are transient errors, so there is no stable way to reproduce the issue. Added unit tests to make sure execute_query function is retried when it should.
   
   ### ADDITIONAL INFORMATION
   <!--- Check any relevant boxes with "x" -->
   <!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
   - [X] Has associated issue: #20418 
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
     - [ ] Migration is atomic, supports rollback & is backwards-compatible
     - [ ] Confirm DB migration upgrade and downgrade tested
     - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] bkyryliuk commented on a diff in pull request #20419: fix(embedded): Retry when executing alert queries to avoid sending transient errors to users as alert failure notifications

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on code in PR #20419:
URL: https://github.com/apache/superset/pull/20419#discussion_r907783151


##########
superset/reports/commands/alert.py:
##########
@@ -46,6 +47,10 @@
 # to avoid heavy loads done by a user mistake
 OPERATOR_FUNCTIONS = {">=": ge, ">": gt, "<=": le, "<": lt, "==": eq, "!=": ne}
 
+# Max tries to run queries to prevent false errors caused by transient errors
+# being returned to users
+EXECUTE_QUERY_MAX_TRIES = 2

Review Comment:
   keep it at 1 to keep the default behavior
   also let's move it to the config to let other to decide what retry policy they are looking for



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] bkyryliuk commented on a diff in pull request #20419: fix(embedded): Retry when executing alert queries to avoid sending transient errors to users as alert failure notifications

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on code in PR #20419:
URL: https://github.com/apache/superset/pull/20419#discussion_r910273070


##########
superset/reports/commands/alert.py:
##########
@@ -27,6 +27,7 @@
 
 from superset import app, jinja_context, security_manager
 from superset.commands.base import BaseCommand
+from superset.config import ALERT_REPORTS_QUERY_EXECUTION_MAX_TRIES

Review Comment:
   it should be accessed in a different way: https://github.com/apache/superset/blob/4f77824e550bc70d2f6365221b9078004d49b78c/superset/reports/commands/execute.py#L388



##########
tests/unit_tests/reports/commands/alert_tests.py:
##########
@@ -0,0 +1,117 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+
+import pandas as pd
+from pytest_mock import MockFixture
+
+from superset.config import ALERT_REPORTS_QUERY_EXECUTION_MAX_TRIES

Review Comment:
   let's overwrite it for the test to be set to 2, feel free to use a test config



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] bkyryliuk merged pull request #20419: fix(embedded): Retry when executing alert queries to avoid sending transient errors to users as alert failure notifications

Posted by GitBox <gi...@apache.org>.
bkyryliuk merged PR #20419:
URL: https://github.com/apache/superset/pull/20419


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] bkyryliuk commented on a diff in pull request #20419: fix(embedded): Retry when executing alert queries to avoid sending transient errors to users as alert failure notifications

Posted by GitBox <gi...@apache.org>.
bkyryliuk commented on code in PR #20419:
URL: https://github.com/apache/superset/pull/20419#discussion_r911259279


##########
tests/integration_tests/reports/alert_tests.py:
##########
@@ -0,0 +1,122 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# pylint: disable=invalid-name, unused-argument, import-outside-toplevel
+
+import pandas as pd
+from pytest_mock import MockFixture
+
+from superset import app
+
+
+def test_execute_query_succeeded_no_retry(
+    mocker: MockFixture, app_context: None
+) -> None:
+
+    from superset.reports.commands.alert import AlertCommand
+
+    execute_query_mock = mocker.patch(
+        "superset.reports.commands.alert.AlertCommand._execute_query",
+        side_effect=lambda: pd.DataFrame([{"sample_col": 0}]),
+    )
+
+    command = AlertCommand(report_schedule=mocker.Mock())
+
+    command.validate()
+
+    assert execute_query_mock.call_count == 1
+
+
+def test_execute_query_succeeded_with_retries(
+    mocker: MockFixture, app_context: None
+) -> None:
+    from superset.reports.commands.alert import AlertCommand, AlertQueryError
+
+    execute_query_mock = mocker.patch(
+        "superset.reports.commands.alert.AlertCommand._execute_query"
+    )
+
+    query_executed_count = 0
+    expected_max_retries = app.config["ALERT_REPORTS_QUERY_EXECUTION_MAX_TRIES"]
+
+    def _mocked_execute_query() -> pd.DataFrame:
+        nonlocal query_executed_count
+        query_executed_count += 1
+
+        if query_executed_count < expected_max_retries:
+            raise AlertQueryError()
+        else:
+            return pd.DataFrame([{"sample_col": 0}])
+
+    execute_query_mock.side_effect = _mocked_execute_query
+    execute_query_mock.__name__ = "mocked_execute_query"
+
+    command = AlertCommand(report_schedule=mocker.Mock())
+
+    command.validate()
+
+    assert execute_query_mock.call_count == expected_max_retries

Review Comment:
   do 
   ```assert execute_query_mock.call_count == 3```
   instead, it will make sure that you are using a right config



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org


[GitHub] [superset] codecov[bot] commented on pull request #20419: fix(embedded) Retry when executing alert queries to avoid sending transient errors to users as alert failure notifications

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #20419:
URL: https://github.com/apache/superset/pull/20419#issuecomment-1158360918

   # [Codecov](https://codecov.io/gh/apache/superset/pull/20419?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#20419](https://codecov.io/gh/apache/superset/pull/20419?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8bb45cc) into [master](https://codecov.io/gh/apache/superset/commit/467d8ef89ea11dcfc4cd67d88a514886254820d2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (467d8ef) will **decrease** coverage by `11.91%`.
   > The diff coverage is `100.00%`.
   
   ```diff
   @@             Coverage Diff             @@
   ##           master   #20419       +/-   ##
   ===========================================
   - Coverage   66.70%   54.79%   -11.92%     
   ===========================================
     Files        1739     1739               
     Lines       65135    65139        +4     
     Branches     6897     6897               
   ===========================================
   - Hits        43450    35693     -7757     
   - Misses      19932    27693     +7761     
     Partials     1753     1753               
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | hive | `?` | |
   | mysql | `?` | |
   | postgres | `?` | |
   | presto | `53.56% <66.66%> (+<0.01%)` | :arrow_up: |
   | python | `58.06% <100.00%> (-24.72%)` | :arrow_down: |
   | sqlite | `?` | |
   | unit | `50.70% <100.00%> (+0.55%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/superset/pull/20419?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [superset/reports/commands/alert.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvcmVwb3J0cy9jb21tYW5kcy9hbGVydC5weQ==) | `52.80% <100.00%> (-43.75%)` | :arrow_down: |
   | [superset/utils/dashboard\_import\_export.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvdXRpbHMvZGFzaGJvYXJkX2ltcG9ydF9leHBvcnQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [superset/key\_value/commands/upsert.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3Vwc2VydC5weQ==) | `0.00% <0.00%> (-89.59%)` | :arrow_down: |
   | [superset/key\_value/commands/update.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL3VwZGF0ZS5weQ==) | `0.00% <0.00%> (-89.37%)` | :arrow_down: |
   | [superset/key\_value/commands/delete.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZS5weQ==) | `0.00% <0.00%> (-85.30%)` | :arrow_down: |
   | [superset/db\_engines/hive.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGJfZW5naW5lcy9oaXZlLnB5) | `0.00% <0.00%> (-85.19%)` | :arrow_down: |
   | [superset/key\_value/commands/delete\_expired.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQva2V5X3ZhbHVlL2NvbW1hbmRzL2RlbGV0ZV9leHBpcmVkLnB5) | `0.00% <0.00%> (-80.77%)` | :arrow_down: |
   | [superset/dashboards/commands/importers/v0.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGFzaGJvYXJkcy9jb21tYW5kcy9pbXBvcnRlcnMvdjAucHk=) | `15.62% <0.00%> (-76.25%)` | :arrow_down: |
   | [superset/datasets/commands/update.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvdXBkYXRlLnB5) | `25.88% <0.00%> (-68.24%)` | :arrow_down: |
   | [superset/datasets/commands/create.py](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3VwZXJzZXQvZGF0YXNldHMvY29tbWFuZHMvY3JlYXRlLnB5) | `30.18% <0.00%> (-67.93%)` | :arrow_down: |
   | ... and [274 more](https://codecov.io/gh/apache/superset/pull/20419/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/superset/pull/20419?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/superset/pull/20419?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [467d8ef...8bb45cc](https://codecov.io/gh/apache/superset/pull/20419?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org