You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by "eschutho (via GitHub)" <gi...@apache.org> on 2026/08/01 15:16:33 UTC
[PR] fix(sqllab): roll back session before retrying get_query after a broken transaction [superset]
eschutho opened a new pull request, #42675:
URL: https://github.com/apache/superset/pull/42675
## Sentry issue
[SUPERSET-PYTHON-WDZ](https://preset-inc.sentry.io/issues/SUPERSET-PYTHON-WDZ) — `SqlLabException: Failed at getting query` — **30,500 occurrences, 24 users impacted**, first seen 2025-04-05, still firing daily in production.
## Root cause
`superset/sql_lab.py::get_query` is wrapped in a `backoff.on_exception(SqlLabException, interval=1, max_tries=5)` retry loop:
```python
def get_query(query_id: int) -> Query:
"""attempts to get the query and retry if it cannot"""
try:
return db.session.query(Query).filter_by(id=query_id).one()
except Exception as ex:
raise SqlLabException("Failed at getting query") from ex
```
The Sentry event's exception chain shows a SQLAlchemy `PendingRollbackError` ("This Session's transaction has been rolled back due to a previous exception during flush... first issue Session.rollback()"), itself chained under an `OperationalError`/`QueryCanceled` from a dropped connection or statement timeout. Once a session enters this state, SQLAlchemy refuses **any** further operation on it until `.rollback()` is called explicitly.
Because `get_query`'s `except` block never rolls back, every one of the 5 `backoff` retries reuses the same poisoned session and fails identically with the same `PendingRollbackError` — the retry loop is guaranteed to burn all 5 attempts and give up, with zero chance to recover from what may have been a transient connection blip.
**Verified with a standalone repro** (pinned `sqlalchemy==1.4.54`, in-memory sqlite): poisoning a session via a failed flush, then calling a `get_query`-shaped function under the same `backoff.on_exception` decorator — without a rollback, all 5 retries fail identically; adding `session.rollback()` in the except block lets the very next retry succeed.
## Fix
Add `db.session.rollback()` in `get_query`'s `except` block before re-raising `SqlLabException`, so each `backoff` retry starts from a clean session and has a real chance of succeeding instead of a guaranteed-failure loop.
## Tradeoffs
None — this is additive-only. The exception raised (`SqlLabException`) and logged (via `get_query_backoff_handler`/`get_query_giveup_handler`) is unchanged; only the session state between retries changes, giving the existing retry mechanism a real chance to work as originally designed. No change to `execute_query`/`execute_sql_statements`/`handle_query_error`, and no change to the `backoff` decorator's parameters.
## Testing
- Added `test_get_query_rolls_back_session_before_retrying` to `tests/unit_tests/sql_lab_test.py`, asserting `get_query` recovers via retry after a first failure, and that `db.session.rollback()` is called before the successful retry.
- `tests/unit_tests/sql_lab_test.py` — 12 passed.
- `ruff check` / `ruff format --check` (pinned `0.9.7`) — clean on both changed files.
## Shortcut
sc-115684
Fixes SUPERSET-PYTHON-WDZ
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): roll back session before retrying get_query after a broken transaction [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #42675:
URL: https://github.com/apache/superset/pull/42675#issuecomment-5152034006
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): roll back session before retrying get_query after a broken transaction [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #42675:
URL: https://github.com/apache/superset/pull/42675#issuecomment-5152050290
<!-- Bito Reply -->
The flagged issue is correct. If `db.session.rollback()` fails, it raises an exception that escapes the `try...except` block, bypassing the retry logic because the decorator only catches `SqlLabException`. To resolve this, wrap the rollback in its own `try...except` block to ensure the original exception is preserved and re-raised as a `SqlLabException`.
```python
except Exception as ex:
try:
db.session.rollback()
except Exception:
# Log rollback failure if necessary
pass
raise SqlLabException("Failed at getting query") from ex
```
**superset/sql_lab.py**
```
except Exception as ex:
try:
db.session.rollback()
except Exception:
# Log rollback failure if necessary
pass
raise SqlLabException("Failed at getting query") from ex
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): roll back session before retrying get_query after a broken transaction [superset]
Posted by "eschutho (via GitHub)" <gi...@apache.org>.
eschutho merged PR #42675:
URL: https://github.com/apache/superset/pull/42675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): roll back session before retrying get_query after a broken transaction [superset]
Posted by "bito-code-review[bot] (via GitHub)" <gi...@apache.org>.
bito-code-review[bot] commented on PR #42675:
URL: https://github.com/apache/superset/pull/42675#issuecomment-5160416845
AI Code Review is in progress (usually takes 3 to 15 minutes unless it's a very large PR).
<details>
<summary>
Bito Usage Guide
</summary>
**Commands**
Type the following command in the pull request comment and save the comment.
- `/review` - Manually triggers a full AI review.
- `/pause` - Pauses automatic reviews on this pull request.
- `/resume` - Resumes automatic reviews.
- `/resolve` - Marks all Bito-posted review comments as resolved.
- `/abort` - Cancels all in-progress reviews.
Refer to the <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/available-commands" target="_blank" rel="noopener noreferrer">documentation</a> for additional commands.
**Configuration**
This repository uses `Superset` You can customize the agent settings <a href="https://alpha.bito.ai/home/ai-agents/code-review-agent" target="_blank" rel="noopener noreferrer">here</a> or contact your Bito workspace admin at evan@preset.io.
**Documentation & Help**
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/getting-started/install-run-using-bito-cloud/create-or-customize-an-agent-instance" target="_blank" rel="noopener noreferrer">Customize agent settings</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/implementing-custom-code-review-rules" target="_blank" rel="noopener noreferrer">Review rules</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent" target="_blank" rel="noopener noreferrer">General documentation</a>
- <a href="https://docs.bito.ai/bito-dev-agents/ai-code-review-agent/faqs" target="_blank" rel="noopener noreferrer">FAQ</a>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): roll back session before retrying get_query after a broken transaction [superset]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #42675:
URL: https://github.com/apache/superset/pull/42675#issuecomment-5152048460
## [Codecov](https://app.codecov.io/gh/apache/superset/pull/42675?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.85%. Comparing base ([`22c305f`](https://app.codecov.io/gh/apache/superset/commit/22c305f75888e5e90ecdd7c71d4223514db38a39?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7d5e8ad`](https://app.codecov.io/gh/apache/superset/commit/7d5e8ad785c1fd1ca04919add6fdf567a3fb9a6e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/superset/pull/42675?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [superset/sql\_lab.py](https://app.codecov.io/gh/apache/superset/pull/42675?src=pr&el=tree&filepath=superset%2Fsql_lab.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3VwZXJzZXQvc3FsX2xhYi5weQ==) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/superset/pull/42675?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (22c305f) and HEAD (7d5e8ad). Click for more details.
>
> <details><summary>HEAD has 82 uploads less than BASE</summary>
>
>| Flag | BASE (22c305f) | HEAD (7d5e8ad) |
>|------|------|------|
>|python|72|1|
>|presto|12|1|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #42675 +/- ##
==========================================
- Coverage 65.43% 55.85% -9.59%
==========================================
Files 2810 2810
Lines 159460 159461 +1
Branches 36396 36396
==========================================
- Hits 104350 89061 -15289
- Misses 53068 69568 +16500
+ Partials 2042 832 -1210
```
| [Flag](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hive](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [mysql](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [postgres](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [presto](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.96% <0.00%> (-0.01%)` | :arrow_down: |
| [python](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.96% <0.00%> (-19.27%)` | :arrow_down: |
| [sqlite](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unit](https://app.codecov.io/gh/apache/superset/pull/42675/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/superset/pull/42675?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
Re: [PR] fix(sqllab): roll back session before retrying get_query after a broken transaction [superset]
Posted by "codeant-ai-for-open-source[bot] (via GitHub)" <gi...@apache.org>.
codeant-ai-for-open-source[bot] commented on code in PR #42675:
URL: https://github.com/apache/superset/pull/42675#discussion_r3695896146
##########
superset/sql_lab.py:
##########
@@ -161,6 +161,9 @@ def get_query(query_id: int) -> Query:
try:
return db.session.query(Query).filter_by(id=query_id).one()
except Exception as ex:
+ # roll back so a poisoned session (e.g. PendingRollbackError after a
+ # failed flush) doesn't fail every subsequent backoff retry identically
+ db.session.rollback()
Review Comment:
**Suggestion:** A failed database rollback can itself raise an exception, which would replace the original error and escape as a non-`SqlLabException`. Because the decorator retries only `SqlLabException`, this prevents retries precisely when the session or connection is too broken to roll back normally. Preserve the original wrapped exception and ensure rollback failures do not bypass the retry contract. [possible bug]
<details>
<summary><b>Severity Level:</b> Major ⚠️</summary>
```mdx
- ❌ Broken database connections can bypass `get_query()` retries.
- ⚠️ SQL Lab error handling receives an unexpected rollback exception.
- ⚠️ Transient query failures may fail instead of recovering.
```
</details>
[](https://app.codeant.ai/fix-in-ide?tool=cursor&prompt_id=5890de64c82c4e2c92c49a17e6fee651&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset) [](https://app.codeant.ai/fix-in-ide?tool=vscode-claude&prompt_id=5890de64c82c4e2c92c49a17e6fee651&service=github&base_url=https%3A%2F%2Fgithub.com&org=apache&repo=apache%2Fsuperset)
*(Use Cmd/Ctrl + Click for best experience)*
<details>
<summary><b>Prompt for AI Agent 🤖 </b></summary>
```mdx
This is a comment left during a code review.
**Path:** superset/sql_lab.py
**Line:** 166:166
**Comment:**
*Possible Bug: A failed database rollback can itself raise an exception, which would replace the original error and escape as a non-`SqlLabException`. Because the decorator retries only `SqlLabException`, this prevents retries precisely when the session or connection is too broken to roll back normally. Preserve the original wrapped exception and ensure rollback failures do not bypass the retry contract.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
```
</details>
<a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F42675&comment_hash=62d2268b5d6ed24b5251d6b6bef0556225783e97a9d21502466c16d68076de48&reaction=like'>👍</a> | <a href='https://app.codeant.ai/feedback?pr_url=https%3A%2F%2Fgithub.com%2Fapache%2Fsuperset%2Fpull%2F42675&comment_hash=62d2268b5d6ed24b5251d6b6bef0556225783e97a9d21502466c16d68076de48&reaction=dislike'>👎</a>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org